Can you try increasing the wire format's maxInactivityDurationInitialDelay as described by Christian in a similar-sounding problem last year: http://activemq.2283324.n4.nabble.com/Wire-Format-Negotiation-Timeout-Say-what-td4666972.html
That may not be the final solution (there might be underlying network problems that need to be addressed by your admins), but it should help diagnose if that's what's going on... On Mon, Nov 3, 2014 at 10:17 PM, Vikas Agarwal <vi...@infoobjects.com> wrote: > For more information, I see following messages in logs (whenever, ActiveMQ > fails to respond to producer and consumer but logs are available) > > 2014-11-04 00:08:03,893 | WARN | Transport Connection to: tcp://<ip>:45550 > failed: java.io.EOFException | > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ > Transport: tcp:///<ip>:45550 > > > And on client side > > WARN org.apache.activemq.transport.failover.FailoverTransport - Transport > (tcp://<ip>:61616) failed, reason: java.io.IOException: Wire format > negotiation timeout: peer did not send his wire format., attempting to > automatically reconnect > > > Strange things is that with this state, everything works sometimes while > does not work at rest of the time. > > On Tue, Nov 4, 2014 at 10:24 AM, Vikas Agarwal <vi...@infoobjects.com> > wrote: > > > By crash, I mean message producers are not able to connect to ActiveMQ > and > > queue listeners stop processing messages because of the same reason. I > have > > to restart ActiveMQ to make everything working fine. > > > > On Tue, Nov 4, 2014 at 12:00 AM, Tim Bain <tb...@alumni.duke.edu> wrote: > > > >> Most developers I know would define "crash" to be "the process > >> terminated". Since that doesn't seem to be the case here (and since you > >> applied the term to an instance of Producer Flow Control in a previous > >> situation, which is most definitely not an ActiveMQ crash nor an > ActiveMQ > >> error of any sort), I assume that's not your definition of the term. > Can > >> you describe the symptoms you're seeing, without using the word "crash", > >> to > >> get everyone on the same page about what your problem actually is? > >> > >> On Mon, Nov 3, 2014 at 10:15 AM, Vikas Agarwal <vi...@infoobjects.com> > >> wrote: > >> > >> > I mean 50,000-100,000 messages are there in queue after which it > crashes > >> > and this range is not definitive, it some crashes below a pile of > 50,000 > >> > messages too. Actually, I am not able to predict when exactly it is > >> > crashing, if its is crashing due to CPU usage or memory usage or IO. > >> > > >> > Logs are behaving fine before the crash, so I believe everything is > fine > >> > with permissions and log level. Log rotation would not remove the > >> > activemq.log completely. Further, after crash, activemq process is > still > >> > alive (checked using ps command), however, Admin UI (port 8161) is not > >> > available after crash. > >> > > >> > On Mon, Nov 3, 2014 at 7:03 PM, James Green <james.mk.gr...@gmail.com > > > >> > wrote: > >> > > >> > > On 3 November 2014 09:42, Vikas Agarwal <vi...@infoobjects.com> > >> wrote: > >> > > > >> > > > Hi, > >> > > > > >> > > > We are using ActiveMQ for 3+ years, however, we didn't test it > under > >> > > heavy > >> > > > load. Recently, we started using ActiveMQ in another project where > >> > > > sometimes load increases exponentially as we are listening to > >> twitter > >> > > > stream which can pile up a lot of messages depending of occurrence > >> of > >> > > some > >> > > > global event like iPhone 6 launch. Now, what we are seeing in > >> ActiveMQ > >> > > > admin UI is that one of the queues gets piled up with thousands of > >> > > message > >> > > > ranging from 50k-100k. Now, sometimes (almost daily) the ActiveMQ > >> > crashes > >> > > > > >> > > > >> > > Do you mean: > >> > > > >> > > 1. The queues have 50-100,000 messages each, or > >> > > 2. The queues have thousands of messages each of which may be > 50-100k > >> in > >> > > size? > >> > > > >> > > > >> > > > in between of my sleeping hours. :( And the strange thing is that > >> when > >> > > try > >> > > > to view the log file for the reason of the crash, I am not able to > >> find > >> > > the > >> > > > activemq.log file at all. We earlier faced ActiveMQ crash issues, > >> but > >> > > that > >> > > > > >> > > > >> > > Ensure you have INFO logging switched on and that AMQ writes to this > >> log > >> > on > >> > > start-up. Also, that log could be rotated - ensure the process has > >> > > permission to write to it. > >> > > > >> > > > >> > > > was due to producer flow control and we disabled that to fix the > >> issue > >> > > and > >> > > > main point is that in those cases we never had the case where the > >> log > >> > > file > >> > > > was missing. I have to restart the process to fix the issue and > that > >> > too, > >> > > > twice because on first restart, it shows error about missing or > >> corrupt > >> > > > data log. > >> > > > > >> > > > So, please suggest me where to look for such issue or what should > >> be my > >> > > > next move for debugging. I am stuck with missing log file. > >> > > > > >> > > > For information here are the memory settings: > >> > > > > >> > > > <systemUsage> > >> > > > <systemUsage> > >> > > > <memoryUsage> > >> > > > <memoryUsage limit="*256 mb*"/> > >> > > > </memoryUsage> > >> > > > <storeUsage> > >> > > > <storeUsage limit="*100 gb*"/> > >> > > > </storeUsage> > >> > > > <tempUsage> > >> > > > <tempUsage limit="*50 gb*"/> > >> > > > </tempUsage> > >> > > > </systemUsage> > >> > > > </systemUsage> > >> > > > > >> > > > >> > > You might want to start here and work out the model of usage you're > >> > seeing: > >> > > http://activemq.apache.org/javalangoutofmemory.html > >> > > > >> > > For instance: thousands of messages, few clients; or thousands of > >> > messages > >> > > and lots of clients. > >> > > > >> > > >> > > >> > > >> > -- > >> > Regards, > >> > Vikas Agarwal > >> > 91 – 9928301411 > >> > > >> > InfoObjects, Inc. > >> > Execution Matters > >> > http://www.infoobjects.com > >> > 2041 Mission College Boulevard, #280 > >> > Santa Clara, CA 95054 > >> > +1 (408) 988-2000 Work > >> > +1 (408) 716-2726 Fax > >> > > >> > > > > > > > > -- > > Regards, > > Vikas Agarwal > > 91 – 9928301411 > > > > InfoObjects, Inc. > > Execution Matters > > http://www.infoobjects.com > > 2041 Mission College Boulevard, #280 > > Santa Clara, CA 95054 > > +1 (408) 988-2000 Work > > +1 (408) 716-2726 Fax > > > > > > > -- > Regards, > Vikas Agarwal > 91 – 9928301411 > > InfoObjects, Inc. > Execution Matters > http://www.infoobjects.com > 2041 Mission College Boulevard, #280 > Santa Clara, CA 95054 > +1 (408) 988-2000 Work > +1 (408) 716-2726 Fax >
