public class ArabicTreebankTokenizer extends PennTreebankTokenizer
StreamTokenizer
.
This implementation is current as of the following LDC catalog numbers: LDC2008E61 (ATBp1v4), LDC2008E62 (ATBp2v3), and LDC2008E22 (ATBp3v3.1)
eolString, st
NEWLINE_TOKEN, nextToken
Constructor and Description |
---|
ArabicTreebankTokenizer(Reader r) |
Modifier and Type | Method and Description |
---|---|
String |
getNext()
Internally fetches the next token.
|
static void |
main(String[] args) |
isEol, setEolString
hasNext, next, peek, remove, tokenize
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
forEachRemaining
public ArabicTreebankTokenizer(Reader r)
public String getNext()
getNext
in class TokenizerAdapter
public static void main(String[] args) throws IOException
IOException