[
https://issues.apache.org/jira/browse/CAMEL-25574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-25574:
--------------------------------
Fix Version/s: 4.24.0
> camel-zip-deflater - after unmarshal().gzipDeflater() the stream-cached body
> cannot be read again from its start: a dead letter channel with
> useOriginalMessage gets only the end of the original body
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25574
> URL: https://issues.apache.org/jira/browse/CAMEL-25574
> Project: Camel
> Issue Type: Bug
> Reporter: shashank
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
>
> {{GzipDeflaterDataFormat.unmarshal}} (lines 57-72 of main df85d15a3dcb) gives
> the stream it gets to {{GzipCompressorInputStream}} of commons-compress. When
> the body of the message is an {{InputStream}}, that stream is the body itself
> ({{UnmarshalProcessor}}, lines 90-92), and with stream caching (on by
> default) the body is an in-memory stream cache: {{InputStreamCache}}, which
> extends {{ByteArrayInputStream}}, or {{ByteArrayInputStreamCache}}, which
> delegates {{mark}} and {{reset}} to a {{ByteArrayInputStream}}.
> The gzip reader uses a stream with {{markSupported()}} as it is, and calls
> {{mark}} before every buffer (8192 bytes) that it reads, to {{reset}} and
> step back when the compressed data ends inside the buffer. So after the
> unmarshal the mark of the stream cache is behind the 10 byte gzip header, or
> at the start of the last buffer of a larger body. {{reset()}} of a
> {{ByteArrayInputStream}} goes back to the last mark, so
> {{StreamCache.reset()}} ("Resets the StreamCache for a new stream
> consumption") no longer goes back to the start of the body.
> Camel resets the stream cache whenever the body is to be read again, and the
> original message that a unit of work keeps for {{useOriginalMessage}} holds
> the same stream cache object ({{DefaultUnitOfWork}}, lines 125-136). Effects,
> with the default configuration:
> * A dead letter channel with {{useOriginalMessage()}}, in a route that fails
> after {{unmarshal().gzipDeflater()}} (by the same mechanism
> {{onException(...).useOriginalMessage()}}; not run): the dead letter gets the
> original body without its first bytes. For a gzip body of 31 bytes it gets
> the last 21, for one of 20028 bytes the last 3634. It is not a gzip stream
> any more, and nothing is logged.
> * A route that keeps the compressed body (in a property, a variable or a
> header) and uses it after the unmarshal, for example to archive it, gets the
> same.
> Not affected: a body that is a {{byte[]}} or a file (a new stream is opened
> for each read), and a stream cache that is spooled to disk
> ({{FileInputStreamCache}} does not support {{mark}}, so the reader wraps it;
> read in the code, not run).
> h3. Reproduction
> New {{GzipDataFormatBodyReadAgainTest}} (3 tests, plain {{direct}} routes, no
> option set). With the {{GzipDeflaterDataFormat}} of main all 3 fail (run
> again with the final test class, 2026-10-12):
> * {{deadLetterChannelGetsTheWholeOriginalBody}}
> ({{errorHandler(deadLetterChannel("direct:dead").useOriginalMessage())}},
> {{unmarshal().gzipDeflater()}}, then an exception; the dead letter route
> converts the body to {{byte[]}}): {{array lengths differ, expected: <31> but
> was: <21>}}
> * {{deadLetterChannelGetsTheWholeOriginalBodyOfALargeMessage}}: {{array
> lengths differ, expected: <20028> but was: <3634>}}
> * {{theCompressedBodyKeptInAPropertyCanBeReadAfterTheUnmarshal}}
> ({{setProperty("compressed", body())}}, unmarshal,
> {{setBody(exchangeProperty("compressed"))}}): {{array lengths differ,
> expected: <31> but was: <21>}}
> h3. Proposed fix
> {{unmarshal}} wraps the stream in a {{BufferedInputStream}} before it gives
> it to the gzip reader, so the reader marks and resets a stream of its own and
> the body is only read. For a stream without mark support the reader already
> did the same wrapping itself; for the in-memory stream caches this adds one
> copy through a buffer of 8192 bytes.
> The unmarshalled result is the same, so there is no upgrade note.
> camel-zip-deflater with the fix: 16 tests, 0 failures.
> Not changed: the in-memory stream caches themselves. Any reader that calls
> {{mark}} in the middle of a stream-cached body moves the position that
> {{StreamCache.reset()}} goes back to. There is one more place in the main
> sources of Camel that moves the mark of such a stream, and the proposed fix
> does not change it: {{IOHelper.copy(InputStream, OutputStream, int, boolean,
> long)}} ({{IOHelper.java}}, lines 224-227) calls {{mark(0)}} and {{reset()}}
> on any {{ByteArrayInputStream}} (so on an {{InputStreamCache}}, not on a
> {{ByteArrayInputStreamCache}}) to get its size, which sets the mark to the
> position the stream has when the copy starts. For a stream at its start (the
> usual case, Camel resets the stream cache before each step) nothing changes;
> for a stream that was already partly read, a later {{reset()}} goes back to
> that position and not to the start. Its callers with a stream are
> {{DefaultHttpBinding.copyStream}}, {{UndertowConsumer}} ({{useStreaming}})
> and {{CachedOutputStream}} (a new stream); a case where one of them gets a
> partly read body was not run. Whether {{reset()}} of the stream caches should
> always go to the start is a question for a separate ticket.
> Related, and independent of this ticket: the read contract of the request
> body stream cache of camel-netty-http ({{NettyChannelBufferStreamCache}}) has
> its own ticket. With the fix proposed there that stream cache does not
> support {{mark}}, so the gzip reader wraps it by itself and neither fix needs
> the other.
> Found with a Lean 4 model of the stream cache (length, position, mark) and of
> the buffer loop of the gzip reader: the model shows that on main a
> {{reset()}} after the unmarshal never goes back before the end of the gzip
> header, for every size of body, so only the tail is returned (21 of 31 and
> 3634 of 20028 bytes in the two tested cases). The fix is modelled as not
> marking the body at all, so for the fix the model says nothing beyond that;
> the tests are the evidence for it. Then confirmed with the data format as
> above.
> Affected: main; camel-4.22.x, camel-4.18.x and camel-4.14.x use the same
> reader on the body (read, not run). Since 3.12.0 (CAMEL-16953).
> Duplicate check (2026-10-12): JIRA text "gzip useOriginalMessage" (1,
> unrelated), "gzipDeflater original message" (none),
> "GzipCompressorInputStream" (1: CAMEL-16953, the change that introduced the
> reader), "stream cache mark reset" (8, none about a reader moving the mark:
> CAMEL-5008, CAMEL-12078, CAMEL-16252 about azure-storage-blob needing mark
> support, CAMEL-25167, CAMEL-5940, CAMEL-24159, CAMEL-12955, CAMEL-12297),
> "gzip unmarshal stream cache truncated" (none); GitHub pull requests
> "gzipDeflater" (2 merged: #26619 YAML validation, #6864 docs),
> "GzipDeflaterDataFormat" (none), "zip-deflater" in the title (1 merged:
> #23642 deprecated methods), none open.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
