Hi Spicon florin, info provided by you really helpful but still I see the tuples failing in the spout. I increased max spending to the value of 2500 still my spout fails with in seconds of times.is there any logic of calculating the incoming tuples from kafka and setting of setMaxSpoutPending according to calculation? what is the maximum value that one can set to setMaxSpoutPending.
*Best regards,* *K.Sai Dilip Reddy.* On Fri, May 6, 2016 at 11:24 PM, Spico Florin <[email protected]> wrote: > hi! > can you please write the hint parallesim that you are using for each > sout and bolts. > there is formula on what amount on data you can have unacked nd the > amound of data that you can process in your bolts. > there is a good book Storm applied where this formul is explained nd > asloong with other goodies. > you can read the book from creating a an account. > https://www.safaribooksonline.com > i cite from the book > so if you have 2 spouts for example with 2 task for each of them and the > max spout pending is 10 the number of possible unacked tuples is: > 2 spouts * 2 tasks * 10 max spout pending = 40 unacked tuples possible > > if the number of the possible uncked tuples is lower than total > parallelism you have set for your topology then it coud be a bottleneck. > the total number of parralellism is the sum of all the maxtasks that you > have set up for your bolts or in case that you have used the > hintparallelims is the sum of all hintparralelsim. > > suppose that you have bolt b1 with 10 hint p, and bolt b2 with 12 hp then > the total p is 10hpb1+12hpb2=22thp. so considering the above max speding > setup we are fine becuse we have 40 greater than 22 so max pending is not > an issue. > i really recommend all the book Storm applied. is rhe best book that > explains also the storm internals > > i hope it helps. > regards, > florin > > > On Friday, May 6, 2016, Sai Dilip Reddy Kiralam < > [email protected]> wrote: > >> Hi All, >> >> >> after increment of parallelism in bolts also spout is failing ? how can I >> know how much time my bolts are taking to complete the tuple processing.I'm >> using kafka spout with 10 partitions so I kept my >> topology.max.spout.pending as 10 .do I need to increase the value more than >> that?if yes to extent i can increase the number ? as a stupid play i kept >> the value 100 to 1 partition kafka it give me metrics error. >> >> >> >> *Best regards,* >> >> *K.Sai Dilip Reddy.* >> >> On Thu, May 5, 2016 at 12:23 PM, Sai Dilip Reddy Kiralam < >> [email protected]> wrote: >> >>> >>> >>> >>> *Best regards,* >>> >>> *K.Sai Dilip Reddy.* >>> >>> ---------- Forwarded message ---------- >>> From: Spico Florin <[email protected]> >>> Date: Tue, May 3, 2016 at 1:02 PM >>> Subject: Re: Why tuples fail in spout >>> To: [email protected] >>> >>> >>> Hi! >>> It could be that your process time for that tuples is greater than >>> topology.message.timeout.secs >>> 30 seconds. In these case have a look at your bolts processing time or how >>> fast you generate your data from spout that could not cope with your bolts. >>> You can: >>> 1. Increase your level your parallelism for your bolts. >>> 2. Throttling the messages from the spouts by setting out >>> topology.max.spout.pending >>> Please have a look at these questions: >>> >>> http://stackoverflow.com/questions/32322682/apache-storm-what-happens-to-a-tuple-when-no-bolts-are-available-to-consume-it >>> >>> >>> http://stackoverflow.com/questions/26536525/what-is-the-point-of-timing-out-tuples >>> >>> I hope that these help. >>> Regards, >>> Florin >>> >>> On Tue, May 3, 2016 at 9:42 AM, Sai Dilip Reddy Kiralam < >>> [email protected]> wrote: >>> >>>> but no failed count is shown for bolts >>>> >>>> >>>> >>>> *Best regards,* >>>> >>>> *K.Sai Dilip Reddy.* >>>> >>>> On Tue, May 3, 2016 at 11:19 AM, John Fang < >>>> [email protected]> wrote: >>>> >>>>> Some tuples failed in the bolts. You can review the bolts' code. Maybe >>>>> your bolts' code trigger the fail() due to some reasons, Or the operation >>>>> of bolts need more time. >>>>> >>>>> ------------------------------------------------------------------ >>>>> 发件人：Sai Dilip Reddy Kiralam <[email protected]> >>>>> 发送时间：2016年5月3日(星期二) 12:06 >>>>> 收件人：user <[email protected]> >>>>> 主 题：Why tuples fail in spout >>>>> >>>>> Hi all, >>>>> >>>>> >>>>> I'm running storm topology in the local machine & storm UI shows some >>>>> tuples are failed in the spout.as per my knowledge spout tuples are >>>>> transferred to a bolts with out any failure.Can any of you help me out in >>>>> finding the reason of tuples failures in the spout. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *Best regards,* >>>>> >>>>> *K.Sai Dilip Reddy.* >>>>> >>>>> >>>>> >>>> >>> >>> >>
