This is an automated email from the ASF dual-hosted git repository.
nddipiazza pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tika.git
The following commit(s) were added to refs/heads/main by this push:
new 88220e158b TIKA-4735: Fix --content-only batch output and remove
-h/--help short option conflict (#2919)
88220e158b is described below
commit 88220e158b50fe5a13b35148160861db56c79b50
Author: Nicholas DiPiazza <[email protected]>
AuthorDate: Wed Jul 1 15:13:22 2026 +0000
TIKA-4735: Fix --content-only batch output and remove -h/--help short
option conflict (#2919)
* TIKA-4735: Fix --content-only batch output and remove -h/--help short
option conflict
- ParseHandler now writes the resolved ParseMode back into ParseContext so
EmitHandler (which reads ParseContext directly with no defaultParseMode
fallback) correctly invokes content-only emit when parseMode is set only
as a PipesConfig default and not explicitly in the per-request context.
- Removed the -h short option from --help in TikaAsyncCLI to eliminate the
conflict with the documented handler shorthand; --help still works as a
long option.
- Added reproducer tests:
AsyncCliParserTest: testTIKA4735_ContentOnlyHandlerMarkdown,
testTIKA4735_ContentOnlyDefaultHandler
TikaConfigAsyncWriterTest: testTIKA4735_ContentOnlyMarkdownConfig,
testTIKA4735_ContentOnlyDefaultMarkdownConfig
AsyncProcessorTest: testContentOnlyFromConfigDefault (integration test
that sends a request without ParseMode in the context and asserts the
output file is raw content, not a JSON metadata wrapper)
Co-authored-by: Copilot <[email protected]>
* TIKA-4735: Fix CONTENT_ONLY passback path in EmitHandler
When emitStrategy=DYNAMIC and the file is small, EmitHandler was
returning the result as a passback to AsyncProcessor. The parent-side
AsyncEmitter then calls emitter.emit(List<EmitData>) which serialises
the metadata list as JSON, bypassing the StreamEmitter path that
produces raw content output.
Fix: CONTENT_ONLY mode now always emits directly from the server side
(shouldEmit returns true for any strategy other than PASSBACK_ALL),
ensuring emitContentOnly() is always used and the output is raw text/
markdown rather than JSON.
Co-authored-by: Copilot <[email protected]>
* TIKA-4735: Fix EmitHandler CONTENT_ONLY to use
emitterManager.getSupported()
Use emitterManager.getSupported().contains(emitterId) instead of null
check to determine whether to force direct emission in CONTENT_ONLY mode.
This correctly handles the case where PipesClient is used directly without
a configured emitter (null emitterId), allowing passback to work normally
while still forcing direct emission (via emitContentOnly) when a real
StreamEmitter is registered.
Fixes CI failures in PipesClientTest.testContentOnlyMode and
testContentOnlyModeWithUserFilter.
Co-authored-by: Copilot <[email protected]>
* TIKA-4735: Replace noise tests with DYNAMIC-passback regression test
Remove tests that did not exercise the actual changed code:
- AsyncCliParserTest: testTIKA4735_ContentOnlyHandlerMarkdown,
testTIKA4735_ContentOnlyDefaultHandler (tested arg parsing; did not
cover the -h removal or any emit-path change)
- TikaConfigAsyncWriterTest: testTIKA4735_ContentOnlyMarkdownConfig,
testTIKA4735_ContentOnlyDefaultMarkdownConfig (tested unchanged config
generation; two tests for identical behaviour)
Add AsyncProcessorTest#testContentOnlyDynamicEmitStrategy: uses
emitStrategy=DYNAMIC with a 10 MB threshold so the small test file goes
through the passback path. Before the EmitHandler fix, the AsyncEmitter
would call emitter.emit(List<EmitData>) and produce JSON; after the fix
it calls emitContentOnly() and the output is raw text.
Co-authored-by: Copilot <[email protected]>
---------
Co-authored-by: Copilot <[email protected]>
---
.../org/apache/tika/async/cli/TikaAsyncCLI.java | 2 +-
.../apache/tika/async/cli/AsyncProcessorTest.java | 39 ++++++++++++++++
.../tika/async/cli/TikaConfigAsyncWriterTest.java | 1 -
.../configs/config-content-only-dynamic.json | 53 ++++++++++++++++++++++
.../apache/tika/pipes/core/server/EmitHandler.java | 10 +++-
.../tika/pipes/core/server/ParseHandler.java | 3 ++
6 files changed, 105 insertions(+), 3 deletions(-)
diff --git
a/tika-pipes/tika-async-cli/src/main/java/org/apache/tika/async/cli/TikaAsyncCLI.java
b/tika-pipes/tika-async-cli/src/main/java/org/apache/tika/async/cli/TikaAsyncCLI.java
index d9b96d359b..cd06371b20 100644
---
a/tika-pipes/tika-async-cli/src/main/java/org/apache/tika/async/cli/TikaAsyncCLI.java
+++
b/tika-pipes/tika-async-cli/src/main/java/org/apache/tika/async/cli/TikaAsyncCLI.java
@@ -69,7 +69,7 @@ public class TikaAsyncCLI {
options.addOption("o", "outputDir", true, "output directory");
options.addOption("n", "numClients", true, "number of forked clients");
options.addOption(null, "Xmx", true, "heap for the forked clients,
e.g. --Xmx 1g");
- options.addOption("h", "help", false, "this help message");
+ options.addOption(null, "help", false, "this help message");
options.addOption("T", "timeoutMs", true, "timeout for each parse in
milliseconds");
options.addOption(null, "handler", true, "handler type: t=text,
h=html, x=xml, m=markdown, b=body, i=ignore (default: m)");
options.addOption("p", "pluginsDir", true, "plugins directory");
diff --git
a/tika-pipes/tika-async-cli/src/test/java/org/apache/tika/async/cli/AsyncProcessorTest.java
b/tika-pipes/tika-async-cli/src/test/java/org/apache/tika/async/cli/AsyncProcessorTest.java
index 17fc05d2b7..4092c01029 100644
---
a/tika-pipes/tika-async-cli/src/test/java/org/apache/tika/async/cli/AsyncProcessorTest.java
+++
b/tika-pipes/tika-async-cli/src/test/java/org/apache/tika/async/cli/AsyncProcessorTest.java
@@ -201,6 +201,45 @@ public class AsyncProcessorTest extends TikaTest {
"content-only output must be raw content, not a JSON wrapper:
" + emitted);
}
+ @Test
+ public void testContentOnlyDynamicEmitStrategy() throws Exception {
+ // TIKA-4735: With emitStrategy=DYNAMIC and a file smaller than the
threshold,
+ // EmitHandler would previously let the file passback to AsyncEmitter,
which called
+ // emitter.emit(List<EmitData>) and serialised it as JSON rather than
raw content.
+ // The fix forces direct server-side emission (via emitContentOnly)
when an emitter
+ // is configured and parseMode=CONTENT_ONLY, regardless of emit
strategy.
+ Path contentOnlyConfig =
configDir.resolve("tika-config-content-only-dynamic.json");
+ Map<String, Object> replacements = new HashMap<>();
+ replacements.put("FETCHER_BASE_PATH", inputDir);
+ replacements.put("JSON_EMITTER_BASE_PATH", jsonOutputDir);
+ replacements.put("BYTES_EMITTER_BASE_PATH", bytesOutputDir);
+ replacements.put("PLUGIN_ROOTS", Paths.get("target/plugins"));
+
JsonConfigHelper.writeConfigFromResource("/configs/config-content-only-dynamic.json",
+ AsyncProcessorTest.class, replacements, contentOnlyConfig);
+
+ AsyncProcessor processor = AsyncProcessor.load(contentOnlyConfig);
+
+ FetchEmitTuple t = new FetchEmitTuple("co-dyn-1", new FetchKey("fsf",
"mock.xml"),
+ new EmitKey("fse-json", "emit-co-dyn"), new Metadata(), new
ParseContext(),
+ FetchEmitTuple.ON_PARSE_EXCEPTION.EMIT);
+ processor.offer(t, 1000);
+ for (int i = 0; i < 10; i++) {
+ processor.offer(PipesIterator.COMPLETED_SEMAPHORE, 1000);
+ }
+ while (processor.checkActive()) {
+ Thread.sleep(100);
+ }
+ processor.close();
+
+ String emitted =
Files.readString(jsonOutputDir.resolve("emit-co-dyn"));
+ assertContains("content", emitted);
+
assertFalse(emitted.contains(TikaCoreProperties.TIKA_CONTENT.getName()),
+ "TIKA-4735: DYNAMIC strategy must not produce JSON wrapper in
CONTENT_ONLY mode: " + emitted);
+ String trimmed = emitted.trim();
+ assertFalse(trimmed.startsWith("[") || trimmed.startsWith("{"),
+ "TIKA-4735: DYNAMIC strategy must produce raw content, not
JSON, in CONTENT_ONLY mode: " + emitted);
+ }
+
@Test
public void testStopsOnApplicationError() throws Exception {
AsyncProcessor processor =
AsyncProcessor.load(configDir.resolve("tika-config.json"));
diff --git
a/tika-pipes/tika-async-cli/src/test/java/org/apache/tika/async/cli/TikaConfigAsyncWriterTest.java
b/tika-pipes/tika-async-cli/src/test/java/org/apache/tika/async/cli/TikaConfigAsyncWriterTest.java
index c9b0f534bd..2be74747a2 100644
---
a/tika-pipes/tika-async-cli/src/test/java/org/apache/tika/async/cli/TikaConfigAsyncWriterTest.java
+++
b/tika-pipes/tika-async-cli/src/test/java/org/apache/tika/async/cli/TikaConfigAsyncWriterTest.java
@@ -48,5 +48,4 @@ public class TikaConfigAsyncWriterTest {
PipesConfig pipesConfig = PipesConfig.load(tikaJsonConfig);
assertEquals("-Xmx1g", pipesConfig.getForkedJvmArgs().get(0));
}
-
}
diff --git
a/tika-pipes/tika-async-cli/src/test/resources/configs/config-content-only-dynamic.json
b/tika-pipes/tika-async-cli/src/test/resources/configs/config-content-only-dynamic.json
new file mode 100644
index 0000000000..d7ef51200f
--- /dev/null
+++
b/tika-pipes/tika-async-cli/src/test/resources/configs/config-content-only-dynamic.json
@@ -0,0 +1,53 @@
+{
+ "fetchers": {
+ "fsf": {
+ "file-system-fetcher": {
+ "basePath": "FETCHER_BASE_PATH",
+ "extractFileSystemMetadata": false
+ }
+ }
+ },
+ "emitters": {
+ "fse-json": {
+ "file-system-emitter": {
+ "basePath": "JSON_EMITTER_BASE_PATH",
+ "fileExtension": "",
+ "onExists": "EXCEPTION"
+ }
+ },
+ "fse-bytes": {
+ "file-system-emitter": {
+ "basePath": "BYTES_EMITTER_BASE_PATH",
+ "fileExtension": "",
+ "onExists": "EXCEPTION"
+ }
+ }
+ },
+ "parse-context": {
+ "timeout-limits": {
+ "progressTimeoutMillis": 60000
+ }
+ },
+ "pipes": {
+ "parseMode": "CONTENT_ONLY",
+ "emitWithinMillis": 10000,
+ "queueSize": 10000,
+ "numEmitters": 1,
+ "emitIntermediateResults": false,
+ "shutdownClientAfterMillis": 300000,
+ "numClients": 2,
+ "maxFilesProcessedPerProcess": 10000,
+ "staleFetcherTimeoutSeconds": 600,
+ "staleFetcherDelaySeconds": 60,
+ "forkedJvmArgs": [
+ "-Xmx1g",
+ "-XX:+UseG1GC"
+ ],
+ "javaPath": "java",
+ "emitStrategy": {
+ "type": "DYNAMIC",
+ "thresholdBytes": 10000000
+ }
+ },
+ "plugin-roots": "PLUGIN_ROOTS"
+}
diff --git
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/EmitHandler.java
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/EmitHandler.java
index 78a21bfa23..94074e048a 100644
---
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/EmitHandler.java
+++
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/EmitHandler.java
@@ -83,7 +83,15 @@ class EmitHandler {
}
EmitDataImpl emitDataTuple = new
EmitDataImpl(t.getEmitKey().getEmitKey(), parseData.getMetadataList(), stack);
ParseMode parseMode = parseContext.get(ParseMode.class);
- if (shouldEmit(parseMode, parseData, emitDataTuple, parseContext))
{
+ String emitterId = emitKey.getEmitterId();
+ // For CONTENT_ONLY mode: force direct emission only when a real
emitter is configured,
+ // so the content is emitted as a raw byte stream via
emitContentOnly() instead of
+ // being returned as passback JSON to the parent AsyncEmitter.
+ boolean forceEmit = parseMode == ParseMode.CONTENT_ONLY
+ && emitterId != null
+ && emitterManager.getSupported().contains(emitterId);
+ boolean willEmit = forceEmit || shouldEmit(parseMode, parseData,
emitDataTuple, parseContext);
+ if (willEmit) {
return emit(t.getId(), emitKey, parseMode == ParseMode.UNPACK,
parseData, stack, parseContext);
} else {
diff --git
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/ParseHandler.java
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/ParseHandler.java
index cd02d99767..c52008d0d0 100644
---
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/ParseHandler.java
+++
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/ParseHandler.java
@@ -87,6 +87,9 @@ class ParseHandler {
List<Metadata> metadataList;
//this adds the EmbeddedDocumentByteStore to the parsecontext
ParseMode parseMode = getParseMode(parseContext);
+ // Ensure the resolved mode is visible to EmitHandler, which checks
parseContext directly
+ // and does not have access to the defaultParseMode fallback.
+ parseContext.set(ParseMode.class, parseMode);
ContentHandlerFactory contentHandlerFactory =
getContentHandlerFactory(parseContext);
if (parseMode == ParseMode.NO_PARSE) {
metadataList = detectOnly(fetchEmitTuple, stream, metadata,
parseContext);
