We are trying to upgrade our ActiveMQ dependency from 5.8.0 to
something newer. In our environment, we use an embedded broker with
useCompression set to true. After the upgrade to any version after
5.9.1, we see some unfortunate side effects on a test client which is
listening on a topic where ActiveMQBytesMessages are posted. The
following exception is thrown while running a test suite which
publishes around 100k ActiveMQBytesMessages on a non persistent topic.
It is very reproducible on about 95% of all runs, but only once or
maybe a few times per run:
11:04:21.075 [ActiveMQ Session Task-1] ERROR
o.a.activemq.ActiveMQMessageConsumer -
ID:hostname-43481-1453197809985-15:1:12:1 Exception while processing
message: ID:hostname-57271-1453197780850-6:2:1:3:95106
java.lang.RuntimeException: Failed to deserialize a message
(ActiveMQBytesMessage {commandId = 3702, responseRequired = false,
messageId = ID:office388-44548-1453286940386-5:1:1:3:1229,
originalDestination = null, originalTransactionId = null, producerId =
ID:office388-44548-1453286940386-5:1:1:3, destination =
topic://com.foo.bar, transactionId = null, expiration = 1453287566063,
timestamp = 1453286966063, arrival = 0, brokerInTime = 1453286966064,
brokerOutTime = 1453286966064, correlationId = null, replyTo = null,
persistent = false, type = null, priority = 4, groupID = null,
groupSequence = 0, targetConsumerId = null, compressed = true, userID
= null, content = org.apache.activemq.util.ByteSequence@7ec950bb,
marshalledProperties = org.apache.activemq.util.ByteSequence@3e90d35e,
dataStructure = null, redeliveryCounter = 0, size = 0, properties =
{mapped=1453286966034, received=1453286965865, source=3,
published=1453286959733, type=TEST, class=2884, party=TEST},
readOnlyProperties = true, readOnlyBody = true, droppable = false,
jmsXGroupFirstForConsumer = false} ActiveMQBytesMessage{ bytesOut =
null, dataOut = null, dataIn = null }).
at
com.foo.bar.PersistEntityMessageHandlerServiceListener.onMessage(PersistEntityMessageHandlerServiceListener.java:57)
~[foobar.jar:amq513-SNAPSHOT]
at
org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1393)
~[activemq-client.jar:5.13.0]
at
org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:131)
[activemq-client.jar:5.13.0]
at
org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:202)
[activemq-client.jar:5.13.0]
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
[activemq-client.jar:5.13.0]
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
[activemq-client.jar:5.13.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: javax.jms.JMSException: java.util.zip.DataFormatException:
incorrect header check
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
~[activemq-client.jar:5.13.0]
at
org.apache.activemq.command.ActiveMQBytesMessage.initializeReading(ActiveMQBytesMessage.java:884)
~[activemq-client.jar:5.13.0]
at
org.apache.activemq.command.ActiveMQBytesMessage.getBodyLength(ActiveMQBytesMessage.java:198)
~[activemq-client.jar:5.13.0]
at
com.foo.bar.JmsSerializerHelper.deserialize(JmsSerializerHelper.java:52)
~[foobar-client.jar:1761]
at
com.foo.bar.PersistEntityMessageHandlerServiceListener.onMessage(PersistEntityMessageHandlerServiceListener.java:53)
~[foobar.jar:amq513-SNAPSHOT]
... 8 common frames omitted
Caused by: java.io.IOException: java.util.zip.DataFormatException:
incorrect header check
at
org.apache.activemq.command.ActiveMQBytesMessage.decompress(ActiveMQBytesMessage.java:902)
~[activemq-client.jar:5.13.0]
at
org.apache.activemq.command.ActiveMQBytesMessage.initializeReading(ActiveMQBytesMessage.java:876)
~[activemq-client.jar:5.13.0]
... 11 common frames omitted
Caused by: java.util.zip.DataFormatException: incorrect header check
at java.util.zip.Inflater.inflateBytes(Native Method)
~[na:1.8.0_45]
at java.util.zip.Inflater.inflate(Inflater.java:259)
~[na:1.8.0_45]
at java.util.zip.Inflater.inflate(Inflater.java:280)
~[na:1.8.0_45]
at
org.apache.activemq.command.ActiveMQBytesMessage.decompress(ActiveMQBytesMessage.java:898)
~[activemq-client.jar:5.13.0]
... 12 common frames omitted
In one occasion, a different cause was shown:
Caused by: java.lang.NegativeArraySizeException: null
at
org.apache.activemq.command.ActiveMQBytesMessage.decompress(ActiveMQBytesMessage.java:897)
~[activemq-client.jar:5.13.0]
I have reproduced this with 5.10.0, 5.10.1, 5.10.2, 5.11.3 and 5.13.0.
It is not reproducible on 5.8.0 (our current version) and 5.9.1. The
stacktrace on 5.10.0 is slightly different:
java.lang.RuntimeException: Failed to deserialize a message
(ActiveMQBytesMessage {commandId = 3702, responseRequired = false,
messageId = ID:office388-44548-1453286940386-5:1:1:3:1229,
originalDestination = null, originalTransactionId = null, producerId =
ID:office388-44548-1453286940386-5:1:1:3, destination =
topic://com.foo.bar, transactionId = null, expiration = 1453287566063,
timestamp = 1453286966063, arrival = 0, brokerInTime = 1453286966064,
brokerOutTime = 1453286966064, correlationId = null, replyTo = null,
persistent = false, type = null, priority = 4, groupID = null,
groupSequence = 0, targetConsumerId = null, compressed = true, userID
= null, content = org.apache.activemq.util.ByteSequence@7ec950bb,
marshalledProperties = org.apache.activemq.util.ByteSequence@3e90d35e,
dataStructure = null, redeliveryCounter = 0, size = 0, properties =
{mapped=1453286966034, received=1453286965865, source=3,
published=1453286959733, type=TEST, class=2884, party=TEST},
readOnlyProperties = true, readOnlyBody = true, droppable = false,
jmsXGroupFirstForConsumer = false} ActiveMQBytesMessage{ bytesOut =
null, dataOut = null, dataIn = null })
at
com.foo.bar.PersistEntityMessageHandlerServiceListener.onMessage(PersistEntityMessageHandlerServiceListener.java:57)
~[classes/:na]
at
org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1390)
~[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:131)
[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:202)
[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
[activemq-client-5.10.0.jar:5.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_66]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_66]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]
Caused by: javax.jms.JMSException: java.lang.NegativeArraySizeException
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
~[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.command.ActiveMQBytesMessage.initializeReading(ActiveMQBytesMessage.java:890)
~[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.command.ActiveMQBytesMessage.getBodyLength(ActiveMQBytesMessage.java:199)
~[activemq-client-5.10.0.jar:5.10.0]
at
com.foo.ba.rJmsSerializerHelper.deserialize(JmsSerializerHelper.java:52)
~[foundations-rpc-client-70.1761.jar:70.1761]
at
com.foo.bar.PersistEntityMessageHandlerServiceListener.onMessage(PersistEntityMessageHandlerServiceListener.java:53)
~[classes/:na]
... 8 common frames omitted
Caused by: java.io.IOException: java.lang.NegativeArraySizeException
at
org.apache.activemq.command.ActiveMQBytesMessage.decompress(ActiveMQBytesMessage.java:908)
~[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.command.ActiveMQBytesMessage.initializeReading(ActiveMQBytesMessage.java:882)
~[activemq-client-5.10.0.jar:5.10.0]
... 11 common frames omitted
Caused by: java.lang.NegativeArraySizeException: null
at
org.apache.activemq.command.ActiveMQBytesMessage.decompress(ActiveMQBytesMessage.java:903)
~[activemq-client-5.10.0.jar:5.10.0]
... 12 common frames omitted
This points me towards
AMQ-4887<https://issues.apache.org/jira/browse/AMQ-4887> and
AMQ-5381<https://issues.apache.org/jira/browse/AMQ-5381> which
correlate with the versions when it starts failing and the code
changes that have been made in ActiveMQBytesMessage.
Halting the process with a debugger does not give me much extra
information, there is content in the Inflater buffer, but the native
method Inflater.inflateBytes just throws the exception. Both client
and server are running on the same machine, same JDK. Running with
1.8.0_66 has the same effect. What I did see was that the actual
compressed bytes at that point were different between what the server
sent and what the client received, which points to client side buffer
corruption. Both the compress on the server side as well as the
decompress methods were always called from the same thread, so there
does not seem to be a multi-threaded race condition at that location,
although it might be somewhere else in the stack.
Disabling compression on the server side will make the error go away.
I have not found any other flag that has effect on this yet.
I could not find any recent issues logged, is this something other
people also encountered? I'm currently trying to isolate the code and
create a test setup, but that's quite an undertaking so please bear
with me.
Thank you,
Claudio
________________________________
The information in this e-mail is intended only for the person or
entity to which it is addressed.
It may contain confidential and /or privileged material. If someone
other than the intended recipient should receive this e-mail, he / she
shall not be entitled to read, disseminate, disclose or duplicate it.
If you receive this e-mail unintentionally, please inform us
immediately by "reply" and then delete it from your system. Although
this information has been compiled with great care, neither IMC
Financial Markets & Asset Management nor any of its related entities
shall accept any responsibility for any errors, omissions or other
inaccuracies in this information or for the consequences thereof, nor
shall it be bound in any way by the contents of this e-mail or its
attachments. In the event of incomplete or incorrect transmission,
please return the e-mail to the sender and permanently delete this
message and any attachments.
Messages and attachments are scanned for all known viruses. Always
scan attachments before opening them.
