and after yet another indexes
correction we enter normal mode. This approach to JFlex matching won't work in
general, but I suppose it'll work for tokenizer, because I did't see any
backtracking in the code (zzCurrentPos never backtracks non-processed
characters).
It would be
actual contents: "a" 255 times, "a", "abc"
}
}
It seems like StandardTokenizer considers completely filled buffer as a
successfully extracted token (1), and also includes tail of too-long-token as a
separate token (2). Maybe (1) is disputable (I think it is bug), but I think
(2) is a bug.
Best regards,
Alexey Makeev
makeev...@mail.ru
