Hi, What would you recommend in our situation then? Packing together all operations into a single bolt,probably will not help in our case : our first bolt receives 400K tuples per/sec , most of them the bolt filters out but for each filtered tuple an ack is sent anyway.
Other question : When it's preferable to use shuffle over localOrShuffle? I was thinking that localOrShuffle is always better since it reduces the network's cost. Is there a scenario when shuffle is preferable? Thank you in advance, Vladi On Tue, Dec 2, 2014 at 6:45 AM, Nathan Marz <[email protected]> wrote: > Acking is 1 message per tuple, which is at most a 50% throughput drop. > However, messages sent for acking are quite small so it is not likely to be > nearly that much of a drop. In addition, the acker bolt is highly efficient > and uses very little CPU. > > Local acking does not really make sense. If you're in a situation where > you would benefit from "local acking", that means you have a lot of bolts > strung together with "localOrShuffle" – in which case you should try > packing together those operations into a single bolt (as Trident does). > > On Mon, Dec 1, 2014 at 2:41 PM, Vladi Feigin <[email protected]> wrote: > >> Hi All, >> >> We use Storm 0.82. Our throughput is 400K messages per sec. >> From Storm UI we calculate that the total latency of all spouts and bolts >> = ~3.5 min to process 10 min of data. But in reality it takes 13 min! >> Obviously it creates huge backlogs. >> We don't have time-out failures at all. We don't other exceptions. >> So our main suspect is Storm acking mechanism , which uses a lot of >> network. >> (BTW if you have other opinion , please let me know) >> We think the fact that the all ack messages go via 0mq ,even when acker >> bolt runs in the same worker, causes this huge performance drop. An ack is >> sent per tuple (micro-batches are not supported), which is inefficient. >> There is no a way as far as we know to define the acker bolt to work in >> local Shuffle (like it's possible for other bolts) >> We'd like to ask your opinion regarding the new proposed feature in Storm: >> Support local acking . That means if an acker runs locally in the same >> worker , send the ack messages via local Disraptor queue (like >> localShuffle) rather than via 0mq. >> >> Does it make sense? What do you think? >> >> If you think that a root cause of our problem is other one, please let us >> know. >> Thank you in advance, >> Vladi >> >> >> >> >> >> >> >> . >> >> >> > > > -- > Twitter: @nathanmarz > http://nathanmarz.com >
