I'm still facing an issue with somewhat confusing behavior regarding message expiration in the DLQ, maybe related to the memory issues I've been having. My aim is to have messages routed to DLQ expire and dropped in one hour. To achieve this, I've set an empty expiry-address and the appropriate expiry-delay. The problem is, most of the messages routed to DLQ end up in an in-delivery state - they are not expiring and I cannot remove them via JMX. Messagecount in the DLQ is slightly higher than the deliveringcount and attempting to remove all messages only removes a number of messages that is equal to the difference between deliveringcount and messagecount which is approximately a few thousand messages while the messagecount is tens of thousands and increasing as message delivery failures occur.
What could be the reason for this behavior and how could it be avoided? -----Original Message----- From: Ilkka Virolainen [mailto:ilkka.virolai...@bitwise.fi] Sent: 22. helmikuuta 2018 13:38 To: users@activemq.apache.org Subject: RE: Artemis 2.4.0 - Issues with memory leaks and JMS message redistribution To answer my own question in case anyone else is wondering about a similar issue, turns out the change in addressing is referred in ticket [1] and adding the multicastPrefix and anycastPrefix described in the ticket to my broker acceptors seems to have fixed my problem. If the issue regarding memory leaks persists I will try to provide a reproducible test case. Thank you for your help, Justin. Best regards, - Ilkka [1] https://issues.apache.org/jira/browse/ARTEMIS-1644 -----Original Message----- From: Ilkka Virolainen [mailto:ilkka.virolai...@bitwise.fi] Sent: 22. helmikuuta 2018 12:33 To: users@activemq.apache.org Subject: RE: Artemis 2.4.0 - Issues with memory leaks and JMS message redistribution Having removed the address configuration and having switched from 2.4.0 to yesterday's snapshot of 2.5.0 it seems like the redistribution of messages is now working, but there also seems to have been a change in addressing between the versions causing another problem related to jms.queue / jms.topic prefixing. While the NMS clients listen and artemis jms clients send to the same topics as described in the previous message, Artemis 2.5.0 prefixes the addresses with jms.topic. While the messages are being sent to e.g. A.B.f64dd592-a8fb-442e-826d-927834d566f4.C.D they are only received if I explicitly prefix the listening address with jms.topic, for example topic://jms.topic.A.B.*.C.D. Can this somehow be avoided in the broker configuration? Best regards -----Original Message----- From: Justin Bertram [mailto:jbert...@apache.org] Sent: 21. helmikuuta 2018 15:19 To: users@activemq.apache.org Subject: Re: Artemis 2.4.0 - Issues with memory leaks and JMS message redistribution Your first issue is probably a misconfiguration. Your cluster-connection is using an "address" value of '*' which I assume is supposed to mean "all addresses," but the "address" element doesn't support wildcards like this. Just leave it empty to match all addresses. See the documentation [1] for more details. Even after you fix that configuration issue you may run into issues. These may be fixed already via ARTEMIS-1523 and/or ARTEMIS-1680. If you have a reproducible test-case then you can verify using the head of the master branch. For the memory issue it would be helpful to have some heap dumps or something to actually see what's actually consuming the memory. Better yet would be a reproducible test-case. Do you have either? Justin [1] https://activemq.apache.org/artemis/docs/latest/clusters.html On Wed, Feb 21, 2018 at 5:39 AM, Ilkka Virolainen < ilkka.virolai...@bitwise.fi> wrote: > Hello, > > I am using Artemis 2.4.0 to broker messages through JMS queues/topics > between a set of clients. Some are Apache NMS 1.7.2 ActiveMQ clients > and others are using Artemis JMS client 1.5.4 included in Spring Boot 1.5.3. > Broker topology is a symmetric cluster of two live nodes with static > connectors, both nodes having been setup as replicating colocated > backup pairs with scale down. I have two quite frustrating issues at the > moment: > message redistribution not working correctly and a memory leak causing > eventual thread death. > > ISSUE #1 - Message redistribution / load balancing not working: > > Client 1 (NMS) connects to broker a and starts listening, artemis > creates the following address: > > (Broker a): > A.B.*.C.D > |-queues > |-multicast > |-f64dd592-a8fb-442e-826d-927834d566f4 > > Server 1 (artemis-jms-client) connects to broker b and sends a message > to > topic: A.B.f64dd592-a8fb-442e-826d-927834d566f4.C.D - this should be > routed to broker a since the corresponding queue has no consumers on > broker b (the queue does not exist). This however does not happen and > the client receives no messages. Broker b has some other clients > connected, causing similar (but not the same) queues having been created: > > (Broker b): > A.B.*.C.D > |-queues > |-multicast > |-1eb48079-7fd8-40e9-b822-bcc25695ced0 > |-9f295257-c352-4ae6-b74b-d5994f330485 > > > ISSUE #2: - Memory leak and eventual thread death > > Artemis broker has 4GB allocated heap space and global-max-size is set > up as half of that (being the default setting). Address-full-policy is > set to PAGE for all addresses and some individual addresses have small > max-size-bytes values set e.g. 104857600. As far as I know the paging > settings should limit memory usage but what happens is that at times > Artemis uses the whole heap space, encounters an out of memory error > and > dies: > > 05:39:29,510 WARN [org.eclipse.jetty.util.thread.QueuedThreadPool] : > java.lang.OutOfMemoryError: Java heap space > 05:39:16,646 WARN [io.netty.channel.ChannelInitializer] Failed to > initialize a channel. Closing: [id: ...]: java.lang.OutOfMemoryError: > Java heap space > 05:41:05,597 WARN [org.eclipse.jetty.util.thread.QueuedThreadPool] > Unexpected thread death: org.eclipse.jetty.util.thread. > QueuedThreadPool$2@5ffaba31 in qtp20111564{STARTED,8<=8<=200,i=2,q=0} > > Are these known issues in Artemis or misconfigurations in the brokers? > > The broker configurations are as follows. Broker b has an identical > configuration excluding that the cluster connector's connector-ref and > static-connector connector-ref refer to broker b and broker a respectively. > > Best regards, > > broker.xml (broker a): > > <?xml version='1.0'?> > <configuration xmlns="urn:activemq" xmlns:xsi="http://www.w3.org/ > 2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq > /schema/artemis-configuration.xsd"> > <core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/ > 2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq:core "> > <name>[broker-a-ip]</name> > <persistence-enabled>true</persistence-enabled> > > <journal-type>NIO</journal-type> > > <paging-directory>...</paging-directory> > <bindings-directory>...</bindings-directory> > <journal-directory>...</journal-directory> > <large-messages-directory>...</large-messages-directory> > > <journal-datasync>true</journal-datasync> > <journal-min-files>2</journal-min-files> > <journal-pool-files>-1</journal-pool-files> > <journal-buffer-timeout>788000</journal-buffer-timeout> > <disk-scan-period>5000</disk-scan-period> > > <max-disk-usage>97</max-disk-usage> > > <critical-analyzer>true</critical-analyzer> > <critical-analyzer-timeout>120000</critical-analyzer-timeout> > <critical-analyzer-check-period>60000</critical- > analyzer-check-period> > <critical-analyzer-policy>HALT</critical-analyzer-policy> > > <acceptors> > <acceptor name="invm-acceptor">vm://0</acceptor> > <acceptor name="artemis">tcp://0.0.0.0:61616</acceptor> > <acceptor name="ssl">tcp://0.0.0.0:61617?sslEnabled=true; > keyStorePath=...;keyStorePassword=...</acceptor> > </acceptors> > <connectors> > <connector name="invm-connector">vm://0</connector> > <connector name="netty-connector">tcp://[ > broker-a-ip]:61616</connector> > <connector name="broker-b-connector">[ > broker-b-ip]:61616</connector> > </connectors> > > <cluster-connections> > <cluster-connection name="cluster-name"> > <address>*</address> > <connector-ref>netty-connector</connector-ref> > <retry-interval>500</retry-interval> > <reconnect-attempts>5</reconnect-attempts> > <use-duplicate-detection>true</use-duplicate-detection> > <message-load-balancing>ON_DEMAND</message-load-balancing> > <max-hops>1</max-hops> > <static-connectors> > <connector-ref>broker-b-connector</connector-ref> > </static-connectors> > </cluster-connection> > </cluster-connections> > > <ha-policy> > <replication> > <colocated> > > <backup-request-retry-interval>5000</backup-request- > retry-interval> > <max-backups>3</max-backups> > <request-backup>true</request-backup> > <backup-port-offset>100</backup-port-offset> > <excludes> > <connector-ref>invm-connector</connector-ref> > <connector-ref>netty-connector</connector-ref> > </excludes> > <master> > <check-for-live-server>true</ > check-for-live-server> > </master> > <slave> > <restart-backup>false</restart-backup> > <scale-down /> > </slave> > </colocated> > </replication> > </ha-policy> > > <cluster-user>ARTEMIS.CLUSTER.ADMIN.USER</cluster-user> > <cluster-password>[the shared cluster > password]</cluster-password> > > <security-settings> > <security-setting match="#"> > <permission type="createDurableQueue" roles="amq, > other-role" /> > <permission type="deleteDurableQueue" roles="amq, > other-role" /> > <permission type="createNonDurableQueue" roles="amq, > other-role" /> > <permission type="createAddress" roles="amq, other-role" /> > <permission type="deleteNonDurableQueue" roles="amq, > other-role" /> > <permission type="deleteAddress" roles="amq, other-role" /> > <permission type="consume" roles="amq, other-role" /> > <permission type="browse" roles="amq, other-role" /> > <permission type="send" roles="amq, other-role" /> > <permission type="manage" roles="amq" /> > </security-setting> > <security-setting match="A.some.queue"> > <permission type="createNonDurableQueue" roles="amq, > other-role" /> > <permission type="deleteNonDurableQueue" roles="amq, > other-role" /> > <permission type="createDurableQueue" roles="amq, > other-role" /> > <permission type="deleteDurableQueue" roles="amq, > other-role" /> > <permission type="createAddress" roles="amq, other-role" /> > <permission type="deleteAddress" roles="amq, other-role" /> > <permission type="consume" roles="amq, other-role" /> > <permission type="browse" roles="amq, other-role" /> > <permission type="send" roles="amq, other-role" /> > </security-setting> > <security-setting match="A.some.other.queue"> > <permission type="createNonDurableQueue" roles="amq, > other-role" /> > <permission type="deleteNonDurableQueue" roles="amq, > other-role" /> > <permission type="createDurableQueue" roles="amq, > other-role" /> > <permission type="deleteDurableQueue" roles="amq, > other-role" /> > <permission type="createAddress" roles="amq, other-role" /> > <permission type="deleteAddress" roles="amq, other-role" /> > <permission type="consume" roles="amq, other-role" /> > <permission type="browse" roles="amq, other-role" /> > <permission type="send" roles="amq, other-role" /> > </security-setting> > ... > ... etc. > ... > </security-settings> > > <address-settings> > <address-setting match="activemq.management#"> > <dead-letter-address>DLQ</dead-letter-address> > <expiry-address>ExpiryQueue</expiry-address> > <redelivery-delay>0</redelivery-delay> > <max-size-bytes>-1</max-size-bytes> > > <message-counter-history-day-limit>10</message-counter- > history-day-limit> > <address-full-policy>PAGE</address-full-policy> > </address-setting> > <!--default for catch all --> > <address-setting match="#"> > <dead-letter-address>DLQ</dead-letter-address> > <expiry-address>ExpiryQueue</expiry-address> > <redelivery-delay>0</redelivery-delay> > <max-size-bytes>-1</max-size-bytes> > > <message-counter-history-day-limit>10</message-counter- > history-day-limit> > <address-full-policy>PAGE</address-full-policy> > <redistribution-delay>1000</redistribution-delay> > </address-setting> > <address-setting match="DLQ"> > <!-- 100 * 1024 * 1024 -> 100MB --> > <max-size-bytes>104857600</max-size-bytes> > <!-- 1000 * 60 * 60 -> 1h --> > <expiry-delay>3600000</expiry-delay> > <expiry-address /> > </address-setting> > <address-setting match="A.some.queue"> > <redelivery-delay-multiplier>1.0</redelivery-delay- > multiplier> > <redelivery-delay>0</redelivery-delay> > <max-redelivery-delay>10</max-redelivery-delay> > </address-setting> > <address-setting match="A.some.other.queue"> > <redelivery-delay-multiplier>1.0</redelivery-delay- > multiplier> > <redelivery-delay>0</redelivery-delay> > <max-redelivery-delay>10</max-redelivery-delay> > <max-delivery-attempts>1</max-delivery-attempts> > <max-size-bytes>104857600</max-size-bytes> > </address-setting> > ... > ... etc. > ... > </address-settings> > > <addresses> > <address name="DLQ"> > <anycast> > <queue name="DLQ" /> > </anycast> > </address> > <address name="ExpiryQueue"> > <anycast> > <queue name="ExpiryQueue" /> > </anycast> > </address> > <address name="A.some.queue"> > <anycast> > <queue name="A.some.queue"> > <durable>true</durable> > </queue> > </anycast> > </address> > <address name="A.some.other.queue"> > <anycast> > <queue name="A.some.other.queue"> > <durable>true</durable> > </queue> > </anycast> > </address> > ... > ... etc. > ... > </addresses> > </core> > </configuration> >
