And spending some more time in jprofiler, looks like 20% of this is do to schedulerSupport and the other 80% of this is do to advisorySupport.
if I set both to false the total runtime of my tests drops in half… and the latencies fall from max create producer latency: 10,176 ms > max create message on existing producer and consumer: 2 ms … to max create producer latency: 1 ms > max create message on existing producer and consumer: 1 ms and this isn’t without fixing the purge background lock. So the question now is what the heck is the advisory support doing that can result in such massive performance overhead. … and I think advisorySupport is enabled by default so that’s problematic as well. On Sun, Feb 22, 2015 at 4:45 PM, Kevin Burton <[email protected]> wrote: > OK. Loaded up JProfiler and confirmed that it’s not LevelDB. > > This is a non-persistent broker I’m testing on. > > Looks like it’s spending all it’s time in CopyOnWriteArrayList.remove and > Timer.purge… > > Which is hopeful because this is ALL due to ActiveMQ internals and in > theory LevelDB should perform well if we improve the performance of > ActiveMQ internals and fix this lock bug. > > Which would rock! > > It should ALSO make queue creation faster. > > > On Sun, Feb 22, 2015 at 4:10 PM, Kevin Burton <[email protected]> wrote: > >> >> >> On Sun, Feb 22, 2015 at 3:30 PM, Tim Bain <[email protected]> wrote: >> >>> So if LevelDB cleanup during removeDestination() is the presumed culprit, >>> can we spin off the LevelDB cleanup work into a separate thread (better: >>> a >>> task object to be serviced by a ThreadPool so you can avoid a fork bomb >>> if >>> we remove many destinations at once) so the call to removeDestination() >>> can >>> return quickly and LevelDB can do its record-keeping in the background >>> without blocking message-processing? >>> >> >> Would that be possible? If the delete is pending on ActiveMQ there is a >> race where a producer could re-create it unless the lock is held. >> >> Though I guess if you dispatched to the GC thread WITH the lock still >> held you would be ok but I think if we use the existing purge thread then >> we’re fine. >> >> OK. I think I’m wrong about LevelDB being the issue. To be fair I wasn’t >> 100% certain before but I should have specified. >> >> Our current production broker is running with persistent=false.. .and I >> just re-ran the tests without disk persistence and it has the same problem. >> >> So the main issue how is why the heck is ActiveMQ taking SO LONG to GC a >> queue. It’s taking about 100ms which is an insane amount of time >> considering this is done all in memory. >> >> Kevin >> >> -- >> >> Founder/CEO Spinn3r.com >> Location: *San Francisco, CA* >> blog: http://burtonator.wordpress.com >> … or check out my Google+ profile >> <https://plus.google.com/102718274791889610666/posts> >> <http://spinn3r.com> >> >> > > > -- > > Founder/CEO Spinn3r.com > Location: *San Francisco, CA* > blog: http://burtonator.wordpress.com > … or check out my Google+ profile > <https://plus.google.com/102718274791889610666/posts> > <http://spinn3r.com> > > -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile <https://plus.google.com/102718274791889610666/posts> <http://spinn3r.com>
