I’m using ActiveMQ, Broker Engine Version 5.15.6 on AWS. The instance
size is mq.t2.micro, in Single-instance broker deployment mode.
A Golang application that has been working for years started
misbehaving. The calling application is getting error messages:
time="2023-02-23 21:35:17.1885" level=error msg="Retrying send"
app=addef error="*Error{Condition: amqp:invalid-field, Description:
Broker: xxx-xxx-xxxx - Client: 6bcc2aa9-468c-4c38-a16e-8713e6e7833b
already connected from tcp://99.99.99.99:34834, Info:
map[invalid-field:container-id]}" id=6bcc2aa9-468c-4c38-a16e-8713e6e7833b
and:
time="2023-02-23 21:35:17.1887" level=*error*msg="failed to handle ct
deletion event" app=addef ctid=04f7519f-1125-4047-7047-c475d0ee87b2
*error*="failed to send ct changed message: **Error*{Condition:
amqp:invalid-field, Description: Broker: xxx-xxx-xxxx- Client:
6bcc2aa9-468c-4c38-a16e-8713e6e7833b already connected from tcp://
99.99.99.99::34834, Info: map[invalid-field:container-id]}"
request_id=42558916-d800-485e-9abb-a2ffd156a366
When I initially looked at the system, the home page showed Store
Percent Used to be 100%. I purged the dead-letter queue, and now Store
Percent Used shows 11%. After the purge, I rebooted the instance, but
the application is still failed.
I turned on Cloudwatch logging for AmazonMQ, and I can see the startup
messages. From looking at the AWS article on enabling logging, INFO is
the default log level and can’t be changed. Aside from startup, all I
see is a few entries:
Transport Connection to: tcp://10.100.16.40:33962 failed:
org.apache.activemq.transport.InactivityIOException: Channel was
inactive for too long |
org.apache.activemq.broker.TransportConnection.Transport |
AmqpInactivityMonitor Async Task:
java.util.concurrent.ThreadPoolExecutor$Worker@47ec0444[State = -1,
empty queue
I looked at the application code, and a message from the dead-letter
queue, and can say there is not a container-id field in them. I searched
around and found some references to invalid-field container-id, but they
seem to be internal references:
https://issues.apache.org/jira/browse/AMQ-5591
Does anyone have any ideas on how I can troubleshoot this, or areas I
can look into? Thanks for helping!
--
George Sexton
(303) 438 9585 x102
MH Software, Inc.
