Your max spout pending is 2000. In a worst case scenario, where you have 1 spout task, 1 bolt task, and every request is timing out, if your spout output queue is full it will take 4000 seconds to drain. This is not accounting for any tuples that need to be replayed, which will cause the full queue drain to take even longer.
On Wed, Jan 28, 2015 at 4:13 PM, Tao, Jing <[email protected]> wrote: > Hi, > > > > I have a topology where one of the bolts is taking 2 seconds to complete > due to a webservice call which is timing out (I set request timeout to 2 > sec). > > > > I have the following configs for the topology: > > config.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS, 60) > > config.setMaxSpoutPending(2000) > > config.setNumAckers(4) > > config.setNumWorkers(3) > > > > What I am seeing is, many message will timeout, and gets replayed every > few minutes. I’m assuming they are repeatedly timing out? But if each > tuple takes 2 seconds, and TOPOLOGY_MESSAGE_TIMEOUT_SECS is 60, why are > they timing out so much? I had a total of 2250 tuples, and the queue still > has not cleared up after almost 2 hours. > > > > When the webservice call does not timeout, 2250 tuples only takes a few > minutes to complete. > > > > Any ideas? > > > > Thanks, > > Jing >
