shashank created CAMEL-25574:
--------------------------------
Summary: camel-zip-deflater - after unmarshal().gzipDeflater() the
stream-cached body cannot be read again from its start: a dead letter channel
with useOriginalMessage gets only the end of the original body
Key: CAMEL-25574
URL: https://issues.apache.org/jira/browse/CAMEL-25574
Project: Camel
Issue Type: Bug
Reporter: shashank
Assignee: shashank
{{GzipDeflaterDataFormat.unmarshal}} (lines 57-72 of main df85d15a3dcb) gives
the stream it gets to {{GzipCompressorInputStream}} of commons-compress. When
the body of the message is an {{InputStream}}, that stream is the body itself
({{UnmarshalProcessor}}, lines 90-92), and with stream caching (on by default)
the body is an in-memory stream cache: {{InputStreamCache}}, which extends
{{ByteArrayInputStream}}, or {{ByteArrayInputStreamCache}}, which delegates
{{mark}} and {{reset}} to a {{ByteArrayInputStream}}.
The gzip reader uses a stream with {{markSupported()}} as it is, and calls
{{mark}} before every buffer (8192 bytes) that it reads, to {{reset}} and step
back when the compressed data ends inside the buffer. So after the unmarshal
the mark of the stream cache is behind the 10 byte gzip header, or at the start
of the last buffer of a larger body. {{reset()}} of a {{ByteArrayInputStream}}
goes back to the last mark, so {{StreamCache.reset()}} ("Resets the StreamCache
for a new stream consumption") no longer goes back to the start of the body.
Camel resets the stream cache whenever the body is to be read again, and the
original message that a unit of work keeps for {{useOriginalMessage}} holds the
same stream cache object ({{DefaultUnitOfWork}}, lines 125-136). Effects, with
the default configuration:
* A dead letter channel with {{useOriginalMessage()}}, in a route that fails
after {{unmarshal().gzipDeflater()}} (by the same mechanism
{{onException(...).useOriginalMessage()}}; not run): the dead letter gets the
original body without its first bytes. For a gzip body of 31 bytes it gets the
last 21, for one of 20028 bytes the last 3634. It is not a gzip stream any
more, and nothing is logged.
* A route that keeps the compressed body (in a property, a variable or a
header) and uses it after the unmarshal, for example to archive it, gets the
same.
Not affected: a body that is a {{byte[]}} or a file (a new stream is opened for
each read), and a stream cache that is spooled to disk
({{FileInputStreamCache}} does not support {{mark}}, so the reader wraps it;
read in the code, not run).
h3. Reproduction
New {{GzipDataFormatBodyReadAgainTest}} (3 tests, plain {{direct}} routes, no
option set). With the {{GzipDeflaterDataFormat}} of main all 3 fail (run again
with the final test class, 2026-10-12):
* {{deadLetterChannelGetsTheWholeOriginalBody}}
({{errorHandler(deadLetterChannel("direct:dead").useOriginalMessage())}},
{{unmarshal().gzipDeflater()}}, then an exception; the dead letter route
converts the body to {{byte[]}}): {{array lengths differ, expected: <31> but
was: <21>}}
* {{deadLetterChannelGetsTheWholeOriginalBodyOfALargeMessage}}: {{array lengths
differ, expected: <20028> but was: <3634>}}
* {{theCompressedBodyKeptInAPropertyCanBeReadAfterTheUnmarshal}}
({{setProperty("compressed", body())}}, unmarshal,
{{setBody(exchangeProperty("compressed"))}}): {{array lengths differ, expected:
<31> but was: <21>}}
h3. Proposed fix
{{unmarshal}} wraps the stream in a {{BufferedInputStream}} before it gives it
to the gzip reader, so the reader marks and resets a stream of its own and the
body is only read. For a stream without mark support the reader already did the
same wrapping itself; for the in-memory stream caches this adds one copy
through a buffer of 8192 bytes.
The unmarshalled result is the same, so there is no upgrade note.
camel-zip-deflater with the fix: 16 tests, 0 failures.
Not changed: the in-memory stream caches themselves. Any reader that calls
{{mark}} in the middle of a stream-cached body moves the position that
{{StreamCache.reset()}} goes back to. There is one more place in the main
sources of Camel that moves the mark of such a stream, and the proposed fix
does not change it: {{IOHelper.copy(InputStream, OutputStream, int, boolean,
long)}} ({{IOHelper.java}}, lines 224-227) calls {{mark(0)}} and {{reset()}} on
any {{ByteArrayInputStream}} (so on an {{InputStreamCache}}, not on a
{{ByteArrayInputStreamCache}}) to get its size, which sets the mark to the
position the stream has when the copy starts. For a stream at its start (the
usual case, Camel resets the stream cache before each step) nothing changes;
for a stream that was already partly read, a later {{reset()}} goes back to
that position and not to the start. Its callers with a stream are
{{DefaultHttpBinding.copyStream}}, {{UndertowConsumer}} ({{useStreaming}}) and
{{CachedOutputStream}} (a new stream); a case where one of them gets a partly
read body was not run. Whether {{reset()}} of the stream caches should always
go to the start is a question for a separate ticket.
Related, and independent of this ticket: the read contract of the request body
stream cache of camel-netty-http ({{NettyChannelBufferStreamCache}}) has its
own ticket. With the fix proposed there that stream cache does not support
{{mark}}, so the gzip reader wraps it by itself and neither fix needs the other.
Found with a Lean 4 model of the stream cache (length, position, mark) and of
the buffer loop of the gzip reader: the model shows that on main a {{reset()}}
after the unmarshal never goes back before the end of the gzip header, for
every size of body, so only the tail is returned (21 of 31 and 3634 of 20028
bytes in the two tested cases). The fix is modelled as not marking the body at
all, so for the fix the model says nothing beyond that; the tests are the
evidence for it. Then confirmed with the data format as above.
Affected: main; camel-4.22.x, camel-4.18.x and camel-4.14.x use the same reader
on the body (read, not run). Since 3.12.0 (CAMEL-16953).
Duplicate check (2026-10-12): JIRA text "gzip useOriginalMessage" (1,
unrelated), "gzipDeflater original message" (none), "GzipCompressorInputStream"
(1: CAMEL-16953, the change that introduced the reader), "stream cache mark
reset" (8, none about a reader moving the mark: CAMEL-5008, CAMEL-12078,
CAMEL-16252 about azure-storage-blob needing mark support, CAMEL-25167,
CAMEL-5940, CAMEL-24159, CAMEL-12955, CAMEL-12297), "gzip unmarshal stream
cache truncated" (none); GitHub pull requests "gzipDeflater" (2 merged: #26619
YAML validation, #6864 docs), "GzipDeflaterDataFormat" (none), "zip-deflater"
in the title (1 merged: #23642 deprecated methods), none open.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
