@someword: might be best to take stack traces of both brokers at 10s intervals for about 4-5 periods... i saw some suspicious threads, but would need to see what kind of progress they made. and curious about what the network bridge was doing on the spoke.
in any event, it would be best to try and reproduce this in a unit test.. especially if you can recreate this on the latest 5.9 snapshots. i've gone ahead and put together the unit test here: https://github.com/christian-posta/activemq/blob/someword-from-mailing-list/activemq-unit-tests/src/test/java/org/apache/activemq/usecases/TwoBrokerReqeustReplyStompTempDestTest.java It's a branch of the activemq code base, and the test is in there. I would recommend running the test to see that it passes, then tweak it to get it just right (like the environment you guys have) to get it to fail. I realize might be a lot going on there in the test, but if you and your team can recreate the issue there, it would be much easier to solve :) On Wed, Sep 25, 2013 at 3:01 PM, someword <derek.ol...@jivesoftware.com>wrote: > > "Maybe you can post more about your clients? > So your flow is Client -> Hub -> Spoke -> Server at which point the server > responds to a temporary destination which flows messages Server -> Spoke -> > Hub -> Client right?" > > > That is correct. > > "How many messages does the server respond w/ over the temporary > destination > for each message the client sends in?" > > The server will respond with just one response to each message it gets. > The server responds just fine for every single request it gets. What's > interesting is that during the course of about 1-2 seconds wall clock time > 500 - 800 messages get to the server and responses get back to the reply-to > queue before the blocking occurs. > > > The underlying application we are using is called MCollective > http://puppetlabs.com/mcollective. I will post the code we are using > which > uses mcollective and stomp to recreate the problem to assist in > troubleshooting. > > The code which runs on the server is here http://pastebin.com/7UgTpT32 > The code which runs on the client is here http://pastebin.com/kDYXYndv > > Please let me know what else I can provide. > Thanks. Derek. > > > > > > > On Wed, Sep 25, 2013 at 2:02 PM, someword <derek.olsen@>wrote: > > > I tried using 5.9 SNAPSHOT and get the same behavior. Here is a link to > a > > thread dump from the broker where the blocking is occurring - > > http://pastebin.com/ew0cL6Kd. Additionally I've captured some > strace/gdb > > output of the ruby client in case that's helpful - > > http://pastebin.com/FTKfczVd. > > > > Any ideas? > > Thanks. Derek. > > > > > > > > -- > > View this message in context: > > > http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html > > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > > > > > > -- > *Christian Posta* > http://www.christianposta.com/blog > twitter: @christianposta > > > Maybe you can post more about your clients? > So your flow is Client -> Hub -> Spoke -> Server at which point the server > responds to a temporary destination which flows messages Server -> Spoke -> > Hub -> Client right? > > How many messages does the server respond w/ over the temporary destination > for each message the client sends in? > > > On Wed, Sep 25, 2013 at 2:02 PM, someword <derek.olsen@>wrote: > > > I tried using 5.9 SNAPSHOT and get the same behavior. Here is a link to > a > > thread dump from the broker where the blocking is occurring - > > http://pastebin.com/ew0cL6Kd. Additionally I've captured some > strace/gdb > > output of the ruby client in case that's helpful - > > http://pastebin.com/FTKfczVd. > > > > Any ideas? > > Thanks. Derek. > > > > > > > > -- > > View this message in context: > > > http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html > > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > > > > > > -- > *Christian Posta* > http://www.christianposta.com/blog > twitter: @christianposta > > > Maybe you can post more about your clients? > So your flow is Client -> Hub -> Spoke -> Server at which point the server > responds to a temporary destination which flows messages Server -> Spoke -> > Hub -> Client right? > > How many messages does the server respond w/ over the temporary destination > for each message the client sends in? > > > On Wed, Sep 25, 2013 at 2:02 PM, someword <derek.olsen@>wrote: > > > I tried using 5.9 SNAPSHOT and get the same behavior. Here is a link to > a > > thread dump from the broker where the blocking is occurring - > > http://pastebin.com/ew0cL6Kd. Additionally I've captured some > strace/gdb > > output of the ruby client in case that's helpful - > > http://pastebin.com/FTKfczVd. > > > > Any ideas? > > Thanks. Derek. > > > > > > > > -- > > View this message in context: > > > http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html > > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > > > > > > -- > *Christian Posta* > http://www.christianposta.com/blog > twitter: @christianposta > > > Maybe you can post more about your clients? > So your flow is Client -> Hub -> Spoke -> Server at which point the server > responds to a temporary destination which flows messages Server -> Spoke -> > Hub -> Client right? > > How many messages does the server respond w/ over the temporary destination > for each message the client sends in? > > > On Wed, Sep 25, 2013 at 2:02 PM, someword <derek.olsen@>wrote: > > > I tried using 5.9 SNAPSHOT and get the same behavior. Here is a link to > a > > thread dump from the broker where the blocking is occurring - > > http://pastebin.com/ew0cL6Kd. Additionally I've captured some > strace/gdb > > output of the ruby client in case that's helpful - > > http://pastebin.com/FTKfczVd. > > > > Any ideas? > > Thanks. Derek. > > > > > > > > -- > > View this message in context: > > > http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671836.html > > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > > > > > > -- > *Christian Posta* > http://www.christianposta.com/blog > twitter: @christianposta > > > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/Did-I-turn-off-producer-flow-control-correctly-tp4671619p4671839.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > -- *Christian Posta* http://www.christianposta.com/blog twitter: @christianposta
