No I just would like to say it's not strange. What you need to know about is that there could be some additional latency when your Spout spends some moments in nextTuple(), ack(), fail(). KafkaSpout reads the data from Kafka so I assume there's execution time in nextTuple() on KafkaSpout.
Apache Storm is planing to merge JStorm which has separate threads between nextTuple() and ack() & fail(). I guess it should help for the situation like this. Hope this helps. 2016년 4월 20일 (수) 오전 10:36, Kevin Conaway <[email protected]>님이 작성: > We're using the out of the box kafka spout, are there any issues related > to that here? > > > One thing you would like to check is, event handler on Spout is single > thread, which means that same thread calls nextTuple() and ack() and > fail(). > > I'm not following you here. What would I be checking here? > > On Tue, Apr 19, 2016 at 9:32 PM, Jungtaek Lim <[email protected]> wrote: > >> Oh right sorry it's introduced at Storm 1.0.0. >> >> And max-spout-pending controls the call of nextTuple() so tuple latency >> shouldn't be affected. >> One thing you would like to check is, event handler on Spout is single >> thread, which means that same thread calls nextTuple() and ack() and >> fail(). >> >> http://storm.apache.org/releases/1.0.0/Concepts.html >> "It is imperative that nextTuple does not block for any spout >> implementation, because Storm calls all the spout methods on the same >> thread." >> >> So if your topology spends non-ignorable time on nextTuple(), that could >> be affected to tuple latency. >> >> Hope this helps. >> >> Thanks, >> Jungtaek Lim (HeartSaVioR) >> >> 2016년 4월 20일 (수) 오전 10:21, Kevin Conaway <[email protected]>님이 >> 작성: >> >>> We are already sending our metrics to graphite but I don't see >>> __skipped-max-spout being logged. Was that added after 0.10? I don't even >>> see a reference to it in the codebase. >>> >>> We are capturing the queue metrics for each component (send_queue and >>> receive_queue) and the population for each is very low, like 0.01% of >>> capacity (< 10 entries at any given time) so I'm not sure that items are >>> sitting in those particular disruptor queues >>> >>> On Tue, Apr 19, 2016 at 8:54 PM, Jungtaek Lim <[email protected]> wrote: >>> >>>> Hi Kevin, >>>> >>>> You can attach metrics consumer to log additional informations for that >>>> topology like disruptor queue metrics, __skipped-max-spout for spout, and >>>> etc. >>>> Please refer >>>> https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/metric/LoggingMetricsConsumer.java >>>> to >>>> how to set up. >>>> >>>> Please note that it may affect whole throughput (I guess it would be >>>> relatively small) so you may want to attach when only testing / debugging >>>> performance issue. >>>> >>>> Thanks, >>>> Jungtaek Lim (HeartSaVioR) >>>> >>>> >>>> 2016년 4월 20일 (수) 오전 9:41, Kevin Conaway <[email protected]>님이 >>>> 작성: >>>> >>>>> In Storm 0.10, is there a way to monitor the _maxSpoutPending_ value? >>>>> I don' see it exposed in any of the metrics that storm publishes but I'd >>>>> like to be able to see how much time each tuple spends waiting and how big >>>>> the pending queue size is for each spout task. Is this possible? >>>>> >>>>> Our topology has one spout and one bolt. The average bolt execute >>>>> latency is ~20ms but the overall spout latency is ~60ms. I'd like to know >>>>> where those other 40ms are being spent. >>>>> >>>>> Thank you >>>>> >>>>> >>>>> -- >>>>> Kevin Conaway >>>>> http://www.linkedin.com/pub/kevin-conaway/7/107/580/ >>>>> https://github.com/kevinconaway >>>>> >>>> >>> >>> >>> -- >>> Kevin Conaway >>> http://www.linkedin.com/pub/kevin-conaway/7/107/580/ >>> https://github.com/kevinconaway >>> >> > > > -- > Kevin Conaway > http://www.linkedin.com/pub/kevin-conaway/7/107/580/ > https://github.com/kevinconaway >
