public static class ArabicTokenizer.ArabicTokenizerFactory<T extends HasWord> extends Object implements TokenizerFactory<T>, Serializable
Modifier and Type | Field and Description |
---|---|
protected LexedTokenFactory<T> |
factory |
protected Properties |
lexerProperties |
Modifier and Type | Method and Description |
---|---|
Iterator<T> |
getIterator(Reader r)
Return an iterator over the contents read from r.
|
Tokenizer<T> |
getTokenizer(Reader r)
Get a tokenizer for this reader.
|
Tokenizer<T> |
getTokenizer(Reader r,
String extraOptions)
Get a tokenizer for this reader.
|
static TokenizerFactory<CoreLabel> |
newTokenizerFactory() |
void |
setOptions(String options)
options: A comma-separated list of options
|
protected final LexedTokenFactory<T extends HasWord> factory
protected Properties lexerProperties
public static TokenizerFactory<CoreLabel> newTokenizerFactory()
public Iterator<T> getIterator(Reader r)
IteratorFromReaderFactory
getIterator
in interface IteratorFromReaderFactory<T extends HasWord>
r
- Where to read objects frompublic Tokenizer<T> getTokenizer(Reader r)
TokenizerFactory
getTokenizer
in interface TokenizerFactory<T extends HasWord>
r
- A Reader (which is assumed to already by buffered, if appropriate)public void setOptions(String options)
setOptions
in interface TokenizerFactory<T extends HasWord>
options
- Options for how this tokenizer should behavepublic Tokenizer<T> getTokenizer(Reader r, String extraOptions)
TokenizerFactory
getTokenizer
in interface TokenizerFactory<T extends HasWord>
r
- A Reader (which is assumed to already by buffered, if appropriate)extraOptions
- Options for how this tokenizer should behave