[
https://issues.apache.org/jira/browse/TIKA-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111139#comment-18111139
]
ASF GitHub Bot commented on TIKA-4875:
--------------------------------------
Copilot commented on code in PR #3123:
URL: https://github.com/apache/tika/pull/3123#discussion_r3925846689
##########
tika-eval/tika-eval-app/src/main/java/org/apache/tika/eval/app/ExtractProfileRunner.java:
##########
@@ -124,8 +124,8 @@ private static String getJdbcConnectionString(String
dbPath) {
}
//default to h2
Path p = Paths.get(dbPath);
- return "jdbc:h2:file:" + p.toAbsolutePath();
-
+ // drop the 45s MVStore chunk retention (bloat + growing compaction
cost); CACHE_SIZE is KB
+ return "jdbc:h2:file:" + p.toAbsolutePath() +
";RETENTION_TIME=0;CACHE_SIZE=1048576";
Review Comment:
The default H2 JDBC URL hard-codes a very large page cache
(CACHE_SIZE=1048576 KB = 1 GiB). This can significantly increase memory usage
for users running tika-eval on constrained machines/CI. Consider making
RETENTION_TIME/CACHE_SIZE configurable (e.g., via system properties) rather
than always forcing these values.
##########
tika-eval/tika-eval-app/src/main/java/org/apache/tika/eval/app/ExtractComparerRunner.java:
##########
@@ -183,8 +183,8 @@ private static String getJdbcConnectionString(String
dbPath) {
}
//default to h2
Path p = Paths.get(dbPath);
- return "jdbc:h2:file:" + p.toAbsolutePath();
-
+ // drop the 45s MVStore chunk retention (bloat + growing compaction
cost); CACHE_SIZE is KB
+ return "jdbc:h2:file:" + p.toAbsolutePath() +
";RETENTION_TIME=0;CACHE_SIZE=1048576";
Review Comment:
The default H2 JDBC URL hard-codes a very large page cache
(CACHE_SIZE=1048576 KB = 1 GiB). This can significantly increase memory usage
for users running tika-eval on constrained machines/CI. Consider making
RETENTION_TIME/CACHE_SIZE configurable (e.g., via system properties) rather
than always forcing these values.
##########
tika-langdetect/tika-langdetect-charsoup/src/test/java/org/apache/tika/langdetect/charsoup/core/CharSoupFeatureExtractorTest.java:
##########
@@ -93,6 +98,87 @@ public void testNFCNormalization() {
assertArrayEquals(countsC, countsD);
}
+ @Test
+ public void testUrlMailStrippingMatchesGreedyRegexReference() {
+ // TIKA-4875: the scanners must stay byte-identical to the regexes
they replaced --
+ // the langdetect and junkdetect models were trained on this exact
preprocessing.
+ // Inputs are NFC-stable, so preprocessNoTruncate's NFC step is an
identity here.
+ Pattern greedyUrl =
Pattern.compile("https?://[-_.?&~;+=/#0-9A-Za-z]{10,10000}");
+ Pattern greedyMail =
Pattern.compile("[-_.0-9A-Za-z]{1,100}@[-_0-9A-Za-z]{1,100}[-_.0-9A-Za-z]{1,100}");
+
+ List<String> cases = new ArrayList<>();
+ // middle mail repeat must backtrack to feed the dot-class tail
+ cases.add("a@bb");
+ cases.add("a@b");
+ cases.add("[email protected]");
+ cases.add("[email protected]");
+ cases.add("a@" + "b".repeat(250));
+ cases.add("a".repeat(150) + "@x.y");
+ cases.add("a".repeat(200) + "@");
+ cases.add("local@local@local");
+ cases.add("[email protected] a@bb [email protected]");
+ cases.add("@@@@@@");
+ cases.add("a@a@a@a@a@a@");
+ // mail caps: head {1,100} then tail {1,100} over one long run
+ cases.add("a@" + "b".repeat(100) + "." + "c".repeat(150));
+ cases.add("a@" + "b".repeat(99) + "." + "c".repeat(99));
+ cases.add("a@" + "b".repeat(300));
+ // find() resumes after a match: leftover run chars are not a fresh
local part
+ cases.add("aa@bb cc@dd ee@ff");
+ cases.add("a@bb@cc@dd");
+ // URL length boundaries: min 10 after scheme, cap 10000
+ cases.add("http://"; + "a".repeat(9));
+ cases.add("http://"; + "a".repeat(10));
+ cases.add("https://"; + "a".repeat(10000));
+ cases.add("https://"; + "a".repeat(10001));
+ cases.add("https://"; + "a".repeat(10005) + "@bb");
+ cases.add("http://http://aaaaaaaaaa";);
+ cases.add("http://aaaahttp://bbbbbbbbbb";);
+ cases.add("hhttp://aaaaaaaaaaa";);
+ cases.add("http:/notaurl http//nope https:/x");
+ // the URL pass runs first; its replacement is a barrier for the mail
pass
+ cases.add("http://aaaaaaaaaa@bb";);
+ cases.add("a@http://aaaaaaaaaa";);
+ cases.add("[email protected]");
+ cases.add("[email protected]/http://foobarbazqux";);
+ // non-ASCII neighbors exercise the < 128 guards
+ cases.add("é@bb");
+ cases.add("aé@bb");
+ cases.add("a@büc.d");
+ cases.add("http://aéaaaaaaaaaa";);
+ cases.add("see http://example.com/a/b?q=1#f and mail
bob.smith@sub-domain_x.example.org.");
+
+ long seed = new Random().nextLong();
+ Random random = new Random(seed);
Review Comment:
This test uses a non-deterministic random seed (`new Random().nextLong()`),
which can make CI failures hard to reproduce (reruns likely generate different
cases). Use a fixed default seed (optionally overridable via a system property)
so any failure is reliably repeatable.
> Improve tika-eval performance
> -----------------------------
>
> Key: TIKA-4875
> URL: https://issues.apache.org/jira/browse/TIKA-4875
> Project: Tika
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Trivial
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
