A few qns to help with the situation: 1. what's the hardware where you're running the broker - #machines, type of machines, config, etc?
2. how much heap are you giving the amq processes? anything else running on the broker nodes or are they dedicated to amq? 3. are messages persistent? if so, what's the backing store? running transactions? how big is the message payload roughly? 4. how many producers and consumers and how do they connect? what clients are you using? what kinds of destinations and how many are there? 5. ran any perf/bench tests against this cluster to figure how far you can go with your workload type? if you don't measure, no way to tune and improve and iterate. amq is a very flexible distributed messaging system but like any of the others, does best if you pay great attention to detail. On Nov 3, 2014, at 20:54, Vikas Agarwal <vi...@infoobjects.com> wrote: By crash, I mean message producers are not able to connect to ActiveMQ and queue listeners stop processing messages because of the same reason. I have to restart ActiveMQ to make everything working fine. > On Tue, Nov 4, 2014 at 12:00 AM, Tim Bain <tb...@alumni.duke.edu> wrote: > > Most developers I know would define "crash" to be "the process > terminated". Since that doesn't seem to be the case here (and since you > applied the term to an instance of Producer Flow Control in a previous > situation, which is most definitely not an ActiveMQ crash nor an ActiveMQ > error of any sort), I assume that's not your definition of the term. Can > you describe the symptoms you're seeing, without using the word "crash", to > get everyone on the same page about what your problem actually is? > > On Mon, Nov 3, 2014 at 10:15 AM, Vikas Agarwal <vi...@infoobjects.com> > wrote: > >> I mean 50,000-100,000 messages are there in queue after which it crashes >> and this range is not definitive, it some crashes below a pile of 50,000 >> messages too. Actually, I am not able to predict when exactly it is >> crashing, if its is crashing due to CPU usage or memory usage or IO. >> >> Logs are behaving fine before the crash, so I believe everything is fine >> with permissions and log level. Log rotation would not remove the >> activemq.log completely. Further, after crash, activemq process is still >> alive (checked using ps command), however, Admin UI (port 8161) is not >> available after crash. >> >> On Mon, Nov 3, 2014 at 7:03 PM, James Green <james.mk.gr...@gmail.com> >> wrote: >> >>>> On 3 November 2014 09:42, Vikas Agarwal <vi...@infoobjects.com> wrote: >>>> >>>> Hi, >>>> >>>> We are using ActiveMQ for 3+ years, however, we didn't test it under >>> heavy >>>> load. Recently, we started using ActiveMQ in another project where >>>> sometimes load increases exponentially as we are listening to twitter >>>> stream which can pile up a lot of messages depending of occurrence of >>> some >>>> global event like iPhone 6 launch. Now, what we are seeing in > ActiveMQ >>>> admin UI is that one of the queues gets piled up with thousands of >>> message >>>> ranging from 50k-100k. Now, sometimes (almost daily) the ActiveMQ >> crashes >>> >>> Do you mean: >>> >>> 1. The queues have 50-100,000 messages each, or >>> 2. The queues have thousands of messages each of which may be 50-100k > in >>> size? >>> >>> >>>> in between of my sleeping hours. :( And the strange thing is that > when >>> try >>>> to view the log file for the reason of the crash, I am not able to > find >>> the >>>> activemq.log file at all. We earlier faced ActiveMQ crash issues, but >>> that >>> >>> Ensure you have INFO logging switched on and that AMQ writes to this > log >> on >>> start-up. Also, that log could be rotated - ensure the process has >>> permission to write to it. >>> >>> >>>> was due to producer flow control and we disabled that to fix the > issue >>> and >>>> main point is that in those cases we never had the case where the log >>> file >>>> was missing. I have to restart the process to fix the issue and that >> too, >>>> twice because on first restart, it shows error about missing or > corrupt >>>> data log. >>>> >>>> So, please suggest me where to look for such issue or what should be > my >>>> next move for debugging. I am stuck with missing log file. >>>> >>>> For information here are the memory settings: >>>> >>>> <systemUsage> >>>> <systemUsage> >>>> <memoryUsage> >>>> <memoryUsage limit="*256 mb*"/> >>>> </memoryUsage> >>>> <storeUsage> >>>> <storeUsage limit="*100 gb*"/> >>>> </storeUsage> >>>> <tempUsage> >>>> <tempUsage limit="*50 gb*"/> >>>> </tempUsage> >>>> </systemUsage> >>>> </systemUsage> >>> >>> You might want to start here and work out the model of usage you're >> seeing: >>> http://activemq.apache.org/javalangoutofmemory.html >>> >>> For instance: thousands of messages, few clients; or thousands of >> messages >>> and lots of clients. >> >> >> >> -- >> Regards, >> Vikas Agarwal >> 91 – 9928301411 >> >> InfoObjects, Inc. >> Execution Matters >> http://www.infoobjects.com >> 2041 Mission College Boulevard, #280 >> Santa Clara, CA 95054 >> +1 (408) 988-2000 Work >> +1 (408) 716-2726 Fax -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 +1 (408) 988-2000 Work +1 (408) 716-2726 Fax
