The address setting is the garbage I was talking about. Upgrade to the latest broker and there will be a cleanup done at the load before it starts.
I highly recommend upgrade. On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram <jbert...@apache.org> wrote: > Analyzing thread dumps like this is pretty simple. I generally just scroll > through and look for long stack-traces with lots of calls from > org.apache.activemq.artemis. In your case every single thread dump has a > thread doing something like this: > > "main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20 nid=0x74fe runnable > [0x00007f9031675000] > java.lang.Thread.State: RUNNABLE > at > > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373) > at > > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192) > at > > org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113) > at > > org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108) > at > > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348) > at > > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168) > at > > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147) > at > > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219) > - locked <0x0000000080a8fce8> (a > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) > at > > org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466) > - locked <0x0000000080a8fce8> (a > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) > at > > org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111) > - locked <0x000000008098be80> (a > org.apache.activemq.artemis.integration.FileBroker) > at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73) > at > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148) > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95) > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129) > at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49) > > In every case it's the "main" thread which isn't surprising as that is the > thread responsible for starting the broker. Also, you can pretty clearly > see in the trace that this is the thread starting the broker, it's loading > the journals, & restoring stored configuration (either address settings or > security settings). I've seen high broker start times when there are lots > and lots of addresses and lots of and lots of settings. Do either (or > both) of these situations apply to you? > > > Justin > > On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford <danlangf...@gmail.com> > wrote: > > > Dang I can’t easily upgrade past 2.1.0 because of the > config-delete-queues > > deserialization bug introduced in 2.2.0. Unless that bug was squashed in > > 2.6+. I don’t think I made a jira for it (vacation and work load) but we > > discussed it back in April. I should go confirm that bug on 2.6 and make > a > > jira for that > > > > Thanks > > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic < > clebert.suco...@gmail.com > > > > > wrote: > > > > > There is an issue I remember where the journal would have some dirt > that > > > was fixed on 2.3/0. > > > > > > I would ipgrade to 2.6.2. > > > > > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <danlangf...@gmail.com> > > > wrote: > > > > > > > would you be willing to help me translate these thread dumps? > > > > > > > > i attached a Zip file with some thread dumps in them. i will also > share > > > > the fasthread.io links for each file. (i was struggling getting > > > > fastthread to do a combo report with the threads in the correct > order) > > > > > > > > artemis04-20180719-1525 https://goo.gl/d88azU > > > > artemis04-20180719-1530 https://goo.gl/G78qn3 > > > > artemis04-20180719-1535 https://goo.gl/aMBSBw > > > > artemis04-20180719-1540 https://goo.gl/brKxxk > > > > artemis04-20180719-1545 https://goo.gl/RaXXCs > > > > artemis04-20180719-1550 https://goo.gl/r5dndK > > > > artemis04-20180719-1555 https://goo.gl/YJRLxe > > > > > > > > at :35, :45, and :55 the young+old gen space gets bigger than at the > > > other > > > > sample times. but i dont know what to look for in here to determine > > what > > > > the broker is actually during during this time. > > > > > > > > thanks > > > > > > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <jbert...@apache.org> > > > > wrote: > > > > > > > >> The first place I would start is grabbing thread dumps every so > often > > to > > > >> see what the broker is actually doing during the 30+ minutes. > > > >> > > > >> > > > >> Justin > > > >> > > > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford < > danlangf...@gmail.com> > > > >> wrote: > > > >> > > > >> > so my server startup times and failover times are growing pretty > > big. > > > >> but i > > > >> > dont really know where to start looking. > > > >> > > > > >> > here is a snippet of some logs to show you the time stamps: > > > >> > > > > >> > 08:11:31,801 INFO [org.apache.activemq.artemis. > > integration.bootstrap] > > > >> > AMQ101000: Starting ActiveMQ Artemis Server > > > >> > > > > >> > 08:11:31,818 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221000: > > > >> > live Message Broker is starting with configuration Broker > > > Configuration > > > >> > (clustered=true,journalDirectory=./data/ > > journal,bindingsDirectory=./ > > > >> > data/bindings,largeMessagesDirectory=./data/large-messages, > > > >> > pagingDirectory=./data/paging) > > > >> > > > > >> > 08:11:34,462 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221012: > > > >> > Using AIO Journal > > > >> > > > > >> > 08:11:34,493 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221057: > > > >> > Global Max Size is being adjusted to 1/2 of the JVM max size > (-Xmx). > > > >> being > > > >> > defined as 1,073,741,824 > > > >> > > > > >> > 08:11:34,555 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221043: > > > >> > Protocol module found: [artemis-server]. Adding protocol support > > for: > > > >> CORE > > > >> > > > > >> > 08:11:34,555 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221043: > > > >> > Protocol module found: [artemis-amqp-protocol]. Adding protocol > > > support > > > >> > for: AMQP > > > >> > > > > >> > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221043: > > > >> > Protocol module found: [artemis-hornetq-protocol]. Adding protocol > > > >> support > > > >> > for: HORNETQ > > > >> > > > > >> > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221043: > > > >> > Protocol module found: [artemis-mqtt-protocol]. Adding protocol > > > support > > > >> > for: MQTT > > > >> > > > > >> > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221043: > > > >> > Protocol module found: [artemis-openwire-protocol]. Adding > protocol > > > >> support > > > >> > for: OPENWIRE > > > >> > > > > >> > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] > > > AMQ221043: > > > >> > Protocol module found: [artemis-stomp-protocol]. Adding protocol > > > support > > > >> > for: STOMP > > > >> > > > > >> > 08:41:38,963 WARN [org.apache.activemq.artemis.core.server] > > > AMQ222165: > > > >> No > > > >> > Dead Letter Address configured for queue DLQ in AddressSettings > > > >> > > > > >> > 08:41:38,963 WARN [org.apache.activemq.artemis.core.server] > > > AMQ222166: > > > >> No > > > >> > Expiry Address configured for queue DLQ in AddressSettings > > > >> > > > > >> > 08:41:38,983 WARN [org.apache.activemq.artemis.core.server] > > > AMQ222165: > > > >> No > > > >> > Dead Letter Address configured for queue ExpiryQueue in > > > AddressSettings > > > >> > > > > >> > 08:41:38,983 WARN [org.apache.activemq.artemis.core.server] > > > AMQ222166: > > > >> No > > > >> > Expiry Address configured for queue ExpiryQueue in AddressSettings > > > >> > > > > >> > 08:41:38,984 WARN [org.apache.activemq.artemis.core.server] > > > AMQ222165: > > > >> No > > > >> > Dead Letter Address configured for queue example in > AddressSettings > > > >> > > > > >> > 08:41:38,985 WARN [org.apache.activemq.artemis.core.server] > > > AMQ222166: > > > >> No > > > >> > Expiry Address configured for queue example in AddressSettings > > > >> > > > > >> > 08:41:38,985 WARN [org.apache.activemq.artemis.core.server] > > > AMQ222165: > > > >> No > > > >> > Dead Letter Address configured for queue exampleQueue in > > > AddressSettings > > > >> > > > > >> > 08:41:38,986 WARN [org.apache.activemq.artemis.core.server] > > > AMQ222166: > > > >> No > > > >> > Expiry Address configured for queue exampleQueue in > AddressSettings > > > >> > > > > >> > > > > >> > and it continues. i have 138 queues. i wonder if i need to be > > looking > > > at > > > >> > PAGE configuration or some cache sizes. do i need to be looking at > > the > > > >> > number of messages persisted on these queues? where would you look > > to > > > >> > determine why the startup times are so long? > > > >> > > > > >> > also, i know i need to upgrade but i cannot upgrade off of 2.1.0 > due > > > to > > > >> > some deseralization changes introduced in 2.2.0. i think i have > > > another > > > >> > thread on here (that i need to update) regarding that upgrade > issue. > > > >> > > > > >> > thanks for any ideas or insight you have for me > > > >> > > > > >> > > > > -- > > > Clebert Suconic > > > > > > -- Clebert Suconic
