Package edu.berkeley.nlp.lm.io
Class KneserNeyFileWritingLmReaderCallback<W>
java.lang.Object
edu.berkeley.nlp.lm.io.KneserNeyFileWritingLmReaderCallback<W>
- Type Parameters:
W-
- All Implemented Interfaces:
ArpaLmReaderCallback<ProbBackoffPair>,LmReaderCallback<ProbBackoffPair>,NgramOrderedLmReaderCallback<ProbBackoffPair>
public class KneserNeyFileWritingLmReaderCallback<W>
extends Object
implements ArpaLmReaderCallback<ProbBackoffPair>
Class for producing a Kneser-Ney language model in ARPA format from raw text.
- Author:
- adampauls
-
Constructor Summary
ConstructorsConstructorDescriptionKneserNeyFileWritingLmReaderCallback(File outputFile, WordIndexer<W> wordIndexer) KneserNeyFileWritingLmReaderCallback(PrintWriter out, WordIndexer<W> wordIndexer) -
Method Summary
Modifier and TypeMethodDescriptionvoidcall(int[] ngram, int startPos, int endPos, ProbBackoffPair value, String words) Called for each n-gramvoidcleanup()Called once all reading is done.voidhandleNgramOrderFinished(int order) Called when all n-grams of a given order are finishedvoidhandleNgramOrderStarted(int order) Called when n-grams of a given order are startedvoidinitWithLengths(List<Long> numNGrams) Called initially with a list of how many n-grams will appear for each order.
-
Constructor Details
-
KneserNeyFileWritingLmReaderCallback
-
KneserNeyFileWritingLmReaderCallback
-
-
Method Details
-
handleNgramOrderFinished
public void handleNgramOrderFinished(int order) Description copied from interface:NgramOrderedLmReaderCallbackCalled when all n-grams of a given order are finished- Specified by:
handleNgramOrderFinishedin interfaceNgramOrderedLmReaderCallback<W>- Parameters:
order-
-
handleNgramOrderStarted
public void handleNgramOrderStarted(int order) Description copied from interface:NgramOrderedLmReaderCallbackCalled when n-grams of a given order are started- Specified by:
handleNgramOrderStartedin interfaceNgramOrderedLmReaderCallback<W>- Parameters:
order-
-
call
Description copied from interface:LmReaderCallbackCalled for each n-gram- Specified by:
callin interfaceLmReaderCallback<W>- Parameters:
ngram- The integer representation of the words as given by the provided WordIndexervalue- The value of the n-gramwords- The string representation of the n-gram (space separated)
-
cleanup
public void cleanup()Description copied from interface:LmReaderCallbackCalled once all reading is done.- Specified by:
cleanupin interfaceLmReaderCallback<W>
-
initWithLengths
Description copied from interface:ArpaLmReaderCallbackCalled initially with a list of how many n-grams will appear for each order.- Specified by:
initWithLengthsin interfaceArpaLmReaderCallback<W>- Parameters:
numNGrams- maps n-gram orders to number of n-grams (i.e. numNGrams.get(0) is the number of unigrams)
-