Based on this thread http://stackoverflow.com/questions/26536525/what-is-the-point-of-timing-out-tuples it sounds like timed-out tuples still flow through the topology. So that would explain why my queue never cleared.
From: Nathan Leung [mailto:[email protected]] Sent: Wednesday, January 28, 2015 5:20 PM To: user Subject: RE: Timed-out tuples replayed over and over I'm not sure, I would have to run a test to see if this is possible. Maybe someone else knows off hand. If it's possible to re-emit while the tuple is still enqueued, I can see how you might run into a situation where you can never work the queue off. On Jan 28, 2015 4:56 PM, "Tao, Jing" <[email protected]<mailto:[email protected]>> wrote: Can multiple copies of the same tuple be in the queue at the same time? Say 1999 tuples timeout, the queue is still reduced to size 1999, correct? What I’m seeing is that the queue size seems to never shrink once timeouts start happening. I’ve seen timeouts previously due to large volume of data, but they did clear up after a few minutes. This time, it did not clear after even 2 hours. The Spout’s Acked # stayed the same for 2 hours. From: Nathan Leung [mailto:[email protected]<mailto:[email protected]>] Sent: Wednesday, January 28, 2015 4:20 PM To: user Subject: Re: Timed-out tuples replayed over and over Your max spout pending is 2000. In a worst case scenario, where you have 1 spout task, 1 bolt task, and every request is timing out, if your spout output queue is full it will take 4000 seconds to drain. This is not accounting for any tuples that need to be replayed, which will cause the full queue drain to take even longer. On Wed, Jan 28, 2015 at 4:13 PM, Tao, Jing <[email protected]<mailto:[email protected]>> wrote: Hi, I have a topology where one of the bolts is taking 2 seconds to complete due to a webservice call which is timing out (I set request timeout to 2 sec). I have the following configs for the topology: config.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS, 60) config.setMaxSpoutPending(2000) config.setNumAckers(4) config.setNumWorkers(3) What I am seeing is, many message will timeout, and gets replayed every few minutes. I’m assuming they are repeatedly timing out? But if each tuple takes 2 seconds, and TOPOLOGY_MESSAGE_TIMEOUT_SECS is 60, why are they timing out so much? I had a total of 2250 tuples, and the queue still has not cleared up after almost 2 hours. When the webservice call does not timeout, 2250 tuples only takes a few minutes to complete. Any ideas? Thanks, Jing
