Package edu.berkeley.nlp.lm.values
Class CountValueContainer
java.lang.Object
edu.berkeley.nlp.lm.values.CountValueContainer
- All Implemented Interfaces:
CompressibleValueContainer<LongRef>,ValueContainer<LongRef>,Serializable
- See Also:
-
Field Summary
FieldsModifier and TypeFieldDescriptionprotected final intprotected final long[]protected final booleanprotected final int[]protected booleanprotected final VariableLengthBitCompressorprotected final intprotected final CustomWidthArray[]protected int -
Constructor Summary
ConstructorsConstructorDescriptionCountValueContainer(LongToIntHashMap countCounter, int valueRadix, boolean storePrefixes, long[] numNgramsForEachOrder) -
Method Summary
Modifier and TypeMethodDescriptionbooleanadd(int[] ngram, int startPos, int endPos, int ngramOrder, long offset, long prefixOffset, int word, LongRef val_, long suffixOffset, boolean ngramIsNew) Adds a new value at the specified offset.voidclearStorageAfterCompression(int ngramOrder) voidclearStorageForOrder(int ngramOrder) createFreshValues(long[] numNgramsForEachOrder_) Creates a fresh value container for copying purposes.voiddecompress(BitStream bits, int ngramOrder, boolean justConsume, LongRef outputVal) Reads and decompresses from the bit stream bits.getCompressed(long offset, int ngramOrder) Compresses the value at the given offset into a list of bits.final longgetCount(int ngramOrder, long index) protected longgetCountRank(long val) protected LongRefvoidgetFromOffset(long index, int ngramOrder, LongRef outputVal) Gets the value living at a particular offset.protected voidgetFromRank(long rank, LongRef outputVal) protected longgetRank(int ngramOrder, long offset) Creates a fresh value of object (useful for passing as an output parameter)longgetSuffixOffset(long index, int ngramOrder) protected intgetSuffixOffsetFromInternalVal(long internalVal) longintnumValueBits(int ngramOrder) voidDestructively sets internal storage from another object.voidInitializes a value container with the map that contains itvoidsetSizeAtLeast(long size, int ngramOrder) Sets internal storage for size for a particular n-gram orderbooleanvoidswap(long a, long b, int ngramOrder) Swaps values at offsets a and b.voidtrim()Final clean up of storage.voidtrimAfterNgram(int ngramOrder, long size) Clear storage after an n-gram order is completeprotected boolean
-
Field Details
-
valueRanks
-
storeSuffixIndexes
protected final boolean storeSuffixIndexes -
valueCoder
-
valueRadix
protected final int valueRadix -
valueWidth
protected int valueWidth -
defaultValRank
protected final int defaultValRank- See Also:
-
numNgramsForEachOrder
protected final long[] numNgramsForEachOrder -
suffixBitsForOrder
protected final int[] suffixBitsForOrder -
useMapValueArray
protected boolean useMapValueArray
-
-
Constructor Details
-
CountValueContainer
public CountValueContainer(LongToIntHashMap countCounter, int valueRadix, boolean storePrefixes, long[] numNgramsForEachOrder)
-
-
Method Details
-
createFreshValues
Description copied from interface:ValueContainerCreates a fresh value container for copying purposes.- Returns:
-
getFromOffset
Description copied from interface:ValueContainerGets the value living at a particular offset.- Parameters:
index-ngramOrder-
-
getFromRank
-
getCount
public final long getCount(int ngramOrder, long index) -
getDefaultVal
-
trimAfterNgram
public void trimAfterNgram(int ngramOrder, long size) Description copied from interface:ValueContainerClear storage after an n-gram order is complete- Specified by:
trimAfterNgramin interfaceValueContainer<LongRef>- Parameters:
ngramOrder-size-
-
getUnigramSum
public long getUnigramSum() -
getScratchValue
Description copied from interface:ValueContainerCreates a fresh value of object (useful for passing as an output parameter)- Returns:
-
setFromOtherValues
Description copied from interface:ValueContainerDestructively sets internal storage from another object.- Specified by:
setFromOtherValuesin interfaceValueContainer<LongRef>- Parameters:
o-
-
trim
public void trim()Description copied from interface:ValueContainerFinal clean up of storage.- Specified by:
trimin interfaceValueContainer<LongRef>
-
getCountRank
protected long getCountRank(long val) -
setMap
Description copied from interface:ValueContainerInitializes a value container with the map that contains it- Specified by:
setMapin interfaceValueContainer<V extends LongRepresentable<V>>
-
useValueStoringArray
protected boolean useValueStoringArray() -
swap
public void swap(long a, long b, int ngramOrder) Description copied from interface:CompressibleValueContainerSwaps values at offsets a and b.- Specified by:
swapin interfaceCompressibleValueContainer<V extends LongRepresentable<V>>- Parameters:
a-b-ngramOrder-
-
add
public boolean add(int[] ngram, int startPos, int endPos, int ngramOrder, long offset, long prefixOffset, int word, LongRef val_, long suffixOffset, boolean ngramIsNew) Description copied from interface:ValueContainerAdds a new value at the specified offset.- Specified by:
addin interfaceValueContainer<V extends LongRepresentable<V>>- Parameters:
ngramOrder- As always, ngramOrder is 0-based (0=unigram)offset-prefixOffset-word-val_-suffixOffset-- Returns:
- Whether or not the add was successful
-
getCompressed
Description copied from interface:CompressibleValueContainerCompresses the value at the given offset into a list of bits.- Specified by:
getCompressedin interfaceCompressibleValueContainer<V extends LongRepresentable<V>>- Parameters:
offset-ngramOrder-- Returns:
-
decompress
Description copied from interface:CompressibleValueContainerReads and decompresses from the bit stream bits.- Specified by:
decompressin interfaceCompressibleValueContainer<V extends LongRepresentable<V>>- Parameters:
bits-ngramOrder-justConsume- If true, nothing is returned, and the function simply consumes the appropriate number of bits from the BitStream.
-
setSizeAtLeast
public void setSizeAtLeast(long size, int ngramOrder) Description copied from interface:ValueContainerSets internal storage for size for a particular n-gram order- Specified by:
setSizeAtLeastin interfaceValueContainer<V extends LongRepresentable<V>>- Parameters:
size-ngramOrder-
-
getSuffixOffset
public long getSuffixOffset(long index, int ngramOrder) -
getSuffixOffsetFromInternalVal
protected int getSuffixOffsetFromInternalVal(long internalVal) - Parameters:
internalVal-- Returns:
-
getRank
protected long getRank(int ngramOrder, long offset) -
clearStorageAfterCompression
public void clearStorageAfterCompression(int ngramOrder) - Specified by:
clearStorageAfterCompressionin interfaceCompressibleValueContainer<V extends LongRepresentable<V>>
-
clearStorageForOrder
public void clearStorageForOrder(int ngramOrder) - Specified by:
clearStorageForOrderin interfaceValueContainer<V extends LongRepresentable<V>>
-
storeSuffixoffsets
public boolean storeSuffixoffsets()- Specified by:
storeSuffixoffsetsin interfaceValueContainer<V extends LongRepresentable<V>>
-
numValueBits
public int numValueBits(int ngramOrder) - Specified by:
numValueBitsin interfaceValueContainer<V extends LongRepresentable<V>>
-