[
https://issues.apache.org/jira/browse/TIKA-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104077#comment-18104077
]
ASF GitHub Bot commented on TIKA-4793:
--------------------------------------
srujana-kuntumalla opened a new pull request, #3009:
URL: https://github.com/apache/tika/pull/3009
## Summary
Follow-on to the earlier TIKA-4793 configurable-payload-limit work. Four
issues were found after that PR:
- **OOM in forked JVM for large documents** (e.g. 500 MB text with uncapped
`maxIpcPayloadBytes`): `ServerProtocolIO.writeFinished()` called
`JsonPipesIpc.toBytes()` unconditionally, allocating a full Smile `byte[]` in
one shot. For large content this exhausted heap.
- **Heap-bound error on very large single files**: Same root cause —
Jackson's `ByteArrayBuilder` cannot produce a `byte[]` > 2 GB, so content that
serializes beyond that limit caused `OutOfMemoryError` inside `toBytes()`.
- **PDF "truncation" past ~80 M chars**: For Unicode-heavy PDFs, 80 M chars
× 3 bytes/char UTF-8 Smile exceeds the 100 MB default limit. Previously the
server wrote the oversized payload anyway, the client threw
`PayloadLimitExceededException`, the IPC stream desynced, and the connection
was torn down. Users saw "no content returned" and interpreted it as truncation.
- **7z / rar / iso DYNAMIC emit-strategy sizing**:
`EmitDataImpl.estimateSizeInBytes()` used `length × 2` (Java UTF-16 heap cost),
but the DYNAMIC threshold is in wire bytes (Smile UTF-8). ASCII-heavy archive
metadata (file paths, checksums, MIME types) encodes at ~1 byte/char, so
estimates were 2× too high and compressed-archive results were incorrectly
routed to direct-emit.
## Changes
**`ServerProtocolIO`** — three-layer guard in `writeFinished()`:
1. **Pre-check**: if `getEstimatedSizeBytes() > maxPayloadBytes`, skip
serialization entirely (prevents OOM before any allocation).
2. **OOM catch**: wraps `JsonPipesIpc.toBytes()` — caught `OutOfMemoryError`
frees the byte-builder segments; the tiny `PAYLOAD_LIMIT_EXCEEDED` response
then serializes cleanly.
3. **Post-check**: if serialized `bytes.length > maxPayloadBytes` (CJK-heavy
content where the 1 byte/char pre-estimate is optimistic), discard server-side
— client sees a clean `PAYLOAD_LIMIT_EXCEEDED`, no stream desync.
Constructor now takes `maxPayloadBytes` explicitly.
**`PipesServer` / `ConnectionHandler`** — pass
`pipesConfig.getMaxIpcPayloadBytes()` to `ServerProtocolIO` (one line each).
**`EmitDataImpl.estimateSizeInBytes()`** — change multiplier from `length ×
2` to `length` (1 byte/char ≈ Smile UTF-8 ASCII). The server-side post-check is
the safety net for content that does serialize larger than the estimate.
**`ServerProtocolIOTest`** (new) — unit tests for all three protection
layers plus a pin on the corrected estimate formula.
## Test plan
- [ ] `ServerProtocolIOTest` — 5 new tests covering pre-check, post-check,
OOM path, status-only pass-through, and estimate formula
- [ ] `PipesMessageTest` — 17 existing wire-protocol tests still pass
- [ ] No existing integration test behavior changes (all test documents are
well below the DYNAMIC strategy thresholds; UNPACK and CONTENT_ONLY modes
bypass the threshold entirely)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
> Make the Pipes IPC max payload size configurable (currently hard-coded to 100
> MB)
> ---------------------------------------------------------------------------------
>
> Key: TIKA-4793
> URL: https://issues.apache.org/jira/browse/TIKA-4793
> Project: Tika
> Issue Type: Improvement
> Components: tika-pipes
> Reporter: Srinivasarao Daruna
> Priority: Major
>
> PipesMessage.MAX_PAYLOAD_BYTES (tika-pipes-core) is a compile-time constant
> set to 100 MB:
> //
> tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/protocol/PipesMessage.java:44
> public static final int MAX_PAYLOAD_BYTES = 100 * 1024 * 1024;
> This cap is enforced on the read side of every IPC message in the
> PipesClient↔PipesServer socket protocol. It does not limit the file size
> being parsed (files are fetched server-side by a Fetcher); it limits the size
> of the serialized JSON payload — most critically the PipesResult (parsed
> metadata + extracted text) returned in FINISHED messages.
> Problems with the current implementation:
> 1. Hard-coded, not configurable. Users with very large documents that produce
> large parse results (and no MetadataWriteLimiterFactory configured) have no
> way to raise the cap short of forking the code. There is no corresponding
> field in PipesConfig.
> 2. No write-side guard. PipesMessage.write() applies no limit before writing.
> When the server serializes a PipesResult exceeding 100 MB and sends it, the
> client's PipesMessage.read() throws IOException("Payload length X exceeds
> maximum of 104857600 bytes"). This is caught by the catch-all Exception block
> in PipesClient.waitForServer() and surfaced to the caller as
> UNSPECIFIED_CRASH — a misleading status that provides no indication of the
> root cause.
> Proposed fix:
> 1. Add maxIpcPayloadBytes to PipesConfig with a default of 100 * 1024 * 1024,
> loaded from the "pipes" JSON config section (consistent with all other
> PipesConfig fields).
> 2. Thread the configured value through to both PipesMessage.read() and
> PipesMessage.write(), replacing the hard-coded constant.
> 3. Add a write-side guard in PipesMessage.write() so oversized results are
> caught server-side with a descriptive IOException rather than failing
> silently at the client with UNSPECIFIED_CRASH.
> Example config (proposed):
> {
> "pipes": {
> "maxIpcPayloadBytes": 209715200
> }
> }
> Note: Users hitting this limit should first consider configuring a
> MetadataWriteLimiterFactory to bound extracted-text size, which is the right
> long-term solution for very large documents. But the limit should still be
> configurable for cases where the full content is legitimately needed.
> Affected files:
> -
> tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/protocol/PipesMessage.java
> -
> tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/PipesConfig.java
> -
> tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/PipesClient.java
> -
> tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/PipesServer.java
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
