The result of jstat shows that it's not in full gc cycle but the minor gc takes more than 1s each time. However, the frequence of minor gc is quite low, which happens once every few seconds.
On Mon, Nov 2, 2015 at 12:29 AM, Nathan Leung <ncle...@gmail.com> wrote: > The box with no throughput might be in a gc loop. Check your heap > utilization and maybe increase worker heap if necessary. Also consider > decreasing the max spout pending, even without further details 20k seems > high. > On Nov 1, 2015 10:50 AM, "Harsha" <st...@harsha.io> wrote: > >> Do you have any calls to external data sources which might be increasing >> the latency and causing tuple timeout? >> >> >> On Sun, Nov 1, 2015, at 04:49 AM, Renjie Liu wrote: >> >> Yes, I've set it to 20000 >> >> On Sun, Nov 1, 2015 at 6:40 PM, Santosh Pingale <pingalesant...@gmail.com >> > wrote: >> >> Have you set 'topology.*max*.*spout*.*pending'?* >> >> On Sun, Nov 1, 2015 at 2:26 PM, Renjie Liu <liurenjie2...@gmail.com> >> wrote: >> >> Hi, storm community: >> >> We have a storm cluster deployed with 15 workers and recently we often >> experience failure since ack timeout. Our input source is kafka and we used >> ganglia to monitor our cluster. Recently we experience failures every 12 >> hours and following are my observations from some monitoring tools when >> problem happens: >> >> 1. Topology page shows that no worker was down since uptime of each >> task are nearly equal to topology uptime >> 2. I've checked ganglia, the cpu report and mem report does not give >> any clue about the problem. But network report shows something unusual: >> the >> in speed decreases a little while the out speed decreases to nearly zero >> on >> some workers. >> 3. I've logged in to one of machines mentioned above, and found out >> that one of the survivor areas always remains 100% full. >> 4. dstat show that csw turns to 4k+ every few seconds while it >> remains around 400 in normal condition. >> >> Can anyone give us some hint about this problem? >> >> >> >> >> >> >> -- >> Renjie Liu >> Department of Computer Science & Engineering >> Shanghai JiaoTong University >> >> >> > -- Renjie Liu Department of Computer Science & Engineering Shanghai JiaoTong University
