This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch OPENNLP-1850-2-tokenizer
in repository https://gitbox.apache.org/repos/asf/opennlp.git
omit 7a3c25ac7 OPENNLP-1850 Address review: fail-loud TermAnalyzer default;
harden WordBreakProperty
omit a75f272f9 OPENNLP-1850 Fail fast on null public-entry arguments
(review nits)
omit f70c1956a OPENNLP-1850 Clarify that Extended_Pictographic symbols are
kept as emoji
omit cc89abf52 OPENNLP-1850 Address tokenizer review comments
omit f48f50f1f OPENNLP-1850 Address Copilot review on the UAX #29 tokenizer
omit 59043dfea OPENNLP-1850 UAX #29 word tokenizer and the layered Term
model
omit 8f1d947dc OPENNLP-1850 Harden andThen insertion mapping docs/tests;
label rung index
omit 9f2622ed9 OPENNLP-1850 Document the NFC precondition of the German
umlaut fold
omit ac7d7d354 OPENNLP-1850 Apply low-priority review polish to the offset
model
omit b7cde2c50 OPENNLP-1850 Harden the offset-aware folds from review
feedback
omit 28846d479 OPENNLP-1850 Make the per-code-point substitution folds
offset-aware
omit 85b10b080 OPENNLP-1850 Offset-aware normalization pipeline
(buildAligned)
omit f2ce18340 OPENNLP-1850 Address review nits: generated
serialVersionUIDs, nested TextNormalizer.Builder
omit 688e50f3b OPENNLP-1850 Add edge-case tests for the aligned offset API
omit 9f23fe8cf OPENNLP-1850 Report the offending line on malformed
confusables data
omit 52e0c9061 OPENNLP-1850 Add Alignment offset model; move normalizer
engine to opennlp-api
omit 629a375a4 OPENNLP-1850 Address Copilot review on the normalization
foundation
omit 48120c7aa OPENNLP-1850 Unicode normalization foundation (CharClass
engine, rungs, Dimension)
add bc9c73ff8 OPENNLP-1850 Unicode normalization engine: CharClass, rungs,
Dimension, confusables (1a)
add 08de0d35c OPENNLP-1850 Offset/alignment layer: Alignment, AlignedText,
buildAligned, *Aligned (1b)
add 064d36345 OPENNLP-1850 UAX #29 word tokenizer and the layered Term
model
add 2ecf63796 OPENNLP-1850 Address Copilot review on the UAX #29 tokenizer
add 724a2544e OPENNLP-1850 Address tokenizer review comments
add d06489714 OPENNLP-1850 Clarify that Extended_Pictographic symbols are
kept as emoji
add 2da2949c8 OPENNLP-1850 Fail fast on null public-entry arguments
(review nits)
add ab038b41e OPENNLP-1850 Address review: fail-loud TermAnalyzer default;
harden WordBreakProperty
add 0bf7f6c03 OPENNLP-1850 Lazy, recoverable loading for WordBreakProperty
and ExtendedPictographic
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (7a3c25ac7)
\
N -- N -- N refs/heads/OPENNLP-1850-2-tokenizer (0bf7f6c03)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../tools/tokenize/uax29/ExtendedPictographic.java | 41 ++++++++---
.../tools/tokenize/uax29/WordBreakProperty.java | 84 +++++++++++++++-------
.../opennlp/tools/util/normalizer/Confusables.java | 25 ++++++-
3 files changed, 112 insertions(+), 38 deletions(-)
