This is an automated email from the ASF dual-hosted git repository.
rzo1 pushed a change to branch OPENNLP-1220
in repository https://gitbox.apache.org/repos/asf/opennlp.git
from f0741075 OPENNLP-1220 - Address review feedback for BPE tokenization
add b38d07e2 OPENNLP-1220 - Address review feedback for BPE tokenization
No new revisions were added by this update.
Summary of changes:
.../main/java/opennlp/tools/tokenize/BPEModel.java | 4 +
.../tools/tokenize/BPETokenizerFactory.java | 28 +------
.../tools/tokenize/BPETokenizerTrainer.java | 2 +-
.../opennlp/tools/tokenize/BPEModelDeTest.java | 8 +-
.../tools/tokenize/BPETokenizerFactoryTest.java | 25 ++-----
.../tokenize/BPETokenizerRealisticDeTest.java | 22 +++---
.../tokenize/BPETokenizerRealisticEsTest.java | 85 +++++++++++++++-------
.../tokenize/BPETokenizerRealisticFrTest.java | 83 ++++++++++++++-------
.../opennlp/tools/tokenize/BPETokenizerTest.java | 2 +-
9 files changed, 145 insertions(+), 114 deletions(-)
