Re: Strange storm problem

2015-11-02 Thread Santosh Pingale
Share storm UI screen shot so that we can have a look at stats of topology. Also a visualization screen shot to know the flow. On Mon, Nov 2, 2015 at 10:25 AM, Renjie Liu wrote: > The output speed is measured by the outpu of dstat, which shows the worker > traffic speed. > > On Mon, Nov 2, 2015

Re: Strange storm problem

2015-11-01 Thread Renjie Liu
The output speed is measured by the outpu of dstat, which shows the worker traffic speed. On Mon, Nov 2, 2015 at 10:52 AM, Nathan Leung wrote: > How are you measuring output speed? Is it possible that you are > experiencing problems with HBase? > > On Sun, Nov 1, 2015 at 9:22 PM, Renjie Liu >

Re: Strange storm problem

2015-11-01 Thread Nathan Leung
How are you measuring output speed? Is it possible that you are experiencing problems with HBase? On Sun, Nov 1, 2015 at 9:22 PM, Renjie Liu wrote: > The result of jstat shows that it's not in full gc cycle but the minor gc > takes more than 1s each time. However, the frequence of minor gc is q

Re: Strange storm problem

2015-11-01 Thread Renjie Liu
The result of jstat shows that it's not in full gc cycle but the minor gc takes more than 1s each time. However, the frequence of minor gc is quite low, which happens once every few seconds. On Mon, Nov 2, 2015 at 12:29 AM, Nathan Leung wrote: > The box with no throughput might be in a gc loop.

Re: Strange storm problem

2015-11-01 Thread Renjie Liu
Yes, we need to write result to hbase but they are written out asynchronously. On Sun, Nov 1, 2015 at 11:49 PM, Harsha wrote: > Do you have any calls to external data sources which might be increasing > the latency and causing tuple timeout? > > > On Sun, Nov 1, 2015, at 04:49 AM, Renjie Liu wro

Re: Strange storm problem

2015-11-01 Thread Nathan Leung
The box with no throughput might be in a gc loop. Check your heap utilization and maybe increase worker heap if necessary. Also consider decreasing the max spout pending, even without further details 20k seems high. On Nov 1, 2015 10:50 AM, "Harsha" wrote: > Do you have any calls to external data

Re: Strange storm problem

2015-11-01 Thread Harsha
Do you have any calls to external data sources which might be increasing the latency and causing tuple timeout? On Sun, Nov 1, 2015, at 04:49 AM, Renjie Liu wrote: > Yes, I've set it to 2 > > On Sun, Nov 1, 2015 at 6:40 PM, Santosh Pingale > wrote: >> Have you set 'topology.*max*.*spout*.*pe

Re: Strange storm problem

2015-11-01 Thread Renjie Liu
Yes, I've set it to 2 On Sun, Nov 1, 2015 at 6:40 PM, Santosh Pingale wrote: > Have you set 'topology.max.spout.pending'? > > On Sun, Nov 1, 2015 at 2:26 PM, Renjie Liu > wrote: > >> Hi, storm community: >> >> We have a storm cluster deployed with 15 workers and recently we often >> experie

Re: Strange storm problem

2015-11-01 Thread Santosh Pingale
Have you set 'topology.max.spout.pending'? On Sun, Nov 1, 2015 at 2:26 PM, Renjie Liu wrote: > Hi, storm community: > > We have a storm cluster deployed with 15 workers and recently we often > experience failure since ack timeout. Our input source is kafka and we used > ganglia to monitor our cl

Strange storm problem

2015-11-01 Thread Renjie Liu
Hi, storm community: We have a storm cluster deployed with 15 workers and recently we often experience failure since ack timeout. Our input source is kafka and we used ganglia to monitor our cluster. Recently we experience failures every 12 hours and following are my observations from some monitor