Hey Kashyap, Excellent points, especially regarding compression. I've thought about trying compression, and your results indicate that's worth a shot.
Also, I concur on fields grouping, especially with a dramatic fan-out followed by a fan-in, which is what I am currently working with. Sure glad I started this thread today because both you and Nick have shared lots of excellent thoughts--much appreciated, and thanks to you both! --John Sent from my iPhone > On Jan 30, 2016, at 7:34 PM, Kashyap Mhaisekar <[email protected]> wrote: > > John, Nick > I don't have direct answers but here is one test I did based on which I > concluded that tuple size does matter. > My use case was like this - > Spout S emits a number X (say 1 or 100 or 1024 etc) -> Bolt A (Which > generates a string of Xkb and emits it out 200 times) -> Bolt C (Bolt see > just prints the the length of the string). All are shuffle grouped and no > limits on max spout pending. > > As you notice, this is a pretty straight topology with really nothing much in > this except emitting out Strings of varying sizes. > > With increase in the size, i notice that the throughput (No. of acks on spout > divided by total time taken) decreases. The test was done on 1 machine so > that network can be ruled out. The only things in play here are the LMAX and > Kryo (de)serialization. > > Another test - if Bolt C was field grouped on X, then i see that the > performance drops much further, probably because all the desrialization is > being done on instance of the bolt AND also because the queues are filled up. > > This being said, when I compressed the emits from Bolt A (Use Snappy > compression), I see that the throuput increases drastically. - I interpret > this as the reduction in size due to compression has improved throughput). > > I unfortunately have not checked VisualVM at the time.. > > Hope this helps. > > Thanks > Kashyap >> On Sat, Jan 30, 2016 at 4:54 PM, John Yost <[email protected]> wrote: >> Also, I am wondering if this issue is actually fixed in 0.10.0: >> https://issues.apache.org/jira/browse/STORM-292 What do you guys think? >> >> --John >> >>> On Sat, Jan 30, 2016 at 5:53 PM, John Yost <[email protected]> wrote: >>> Hi Kashyap, >>> >>> Question--what percentage of time is spent in Kryo deserialization and how >>> much in LMAX disruptor? >>> >>> --John >>> >>>> On Sat, Jan 30, 2016 at 5:18 PM, Kashyap Mhaisekar <[email protected]> >>>> wrote: >>>> That is right. But for a decently well written code, disruptor is almost >>>> always the CPU hogger. That said, on the issue b of emits taking time, we >>>> found that the size of emitted object matters. Kryo times for serializing >>>> and deserialization increases with size. >>>> >>>> But does size have a correlation with disruptor showing up big time in >>>> profiling? >>>> >>>> Thanks >>>> Kashyap >>>> >>>> Kashyap, >>>> >>>> It is only expected to see the Disruptor dominating CPU time. It is the >>>> object responsible for sending/receiving tuples (at least when you have >>>> tuples produced by one executor thread for another executor thread on the >>>> same machine). Therefore, it is expected to see Disruptor having something >>>> like ~80% of the time. >>>> >>>> A nice experiment to check my statement above is to create a Bolt that for >>>> every tuple it receives, it performs a random CPU task (like nested for >>>> loops) and it emits a tuple only after receiving X number of tuples, where >>>> X > 1. Then, I expect that you will see the percentage of CPU time for the >>>> Disruptor object to drop. >>>> >>>> Cheers, >>>> Nick >>>> >>>>> On Sat, Jan 30, 2016 at 3:40 PM, Kashyap Mhaisekar <[email protected]> >>>>> wrote: >>>>> John, Nick >>>>> Thanks for broaching this topic. In my case, 1 tuple from spout gives out >>>>> 200 more tuples. I too see the same class listed in VisualVM profiling... >>>>> And tried bringing this down... I reduced parallelism hints, played with >>>>> buffers, changed lmax strategies, changed max spout pending... Nothing >>>>> seems to have an impact >>>>> >>>>> Any ideas on what could be done for this? >>>>> >>>>> Thanks >>>>> Kashyap >>>>> >>>>> Hello John, >>>>> >>>>> First off, let us agree on your definition of throughput. Do you define >>>>> throughput as the average number of tuples each of your last bolts >>>>> (sinks) emit per second? If yes, then OK. Otherwise, please provide us >>>>> with more details. >>>>> >>>>> Going back to the BlockingWaitStrategy observation you have, it (most >>>>> probably) means that since you are producing a large number of tuples >>>>> (15-20 tuples) the outgoing Disruptor queue gets full, and the emit() >>>>> function blocks. Also, since you are anchoring tuples (that might mean >>>>> exactly-once semantics), it basically takes more time to place something >>>>> in the queue, in order to guarantee deliver of all tuples to a downstream >>>>> bolt. >>>>> >>>>> Therefore, it makes sense to see so much time spent in the LMAX messaging >>>>> layer. A good experiment to verify your hypothesis, is to not anchor >>>>> tuples, and profile your topology again. However, I am not sure that you >>>>> will see a much different percentage, since for every tuple you are >>>>> receiving, you have at least one call to the Disruptor layer. Maybe in >>>>> your case (if I got it correctly from your description), you should have >>>>> one call every N tuples, where N is the size of your bin in tuples. Right? >>>>> >>>>> I hope I helped with my comments. >>>>> >>>>> Cheers, >>>>> Nick >>>>> >>>>>> On Sat, Jan 30, 2016 at 12:16 PM, John Yost <[email protected]> wrote: >>>>>> Hi Everyone, >>>>>> >>>>>> I have a large fan-out that I've posted questions about before with the >>>>>> following new, updated info: >>>>>> >>>>>> 1. Incoming tuple to Bolt A produces 15-20 tuples >>>>>> 2. Bolt A emits to Bolt B via fieldsGrouping >>>>>> 3. I cache outgoing tuples in bins within Bolt A and then emit anchored >>>>>> tuples to Bolt B with the OutputCollector emit(Collection<Tuple> >>>>>> anchors, List<Object> tuple) method >>>>>> 4. I have throughput where I need it to be if I just receive tuples in >>>>>> Bolt B, ack, and drop. If I do actual processing in Bolt B, throughput >>>>>> degrades a bunch. >>>>>> 5. I profiled the Bolt B worker yesterday and see that over 90% is spent >>>>>> in com.lmax.disruptor.BlockingWaitStrategy--irrespective if I drop the >>>>>> tuples or process in Bolt B >>>>>> >>>>>> I am wondering if the acking of the anchor tuples is what's resulting in >>>>>> so much time spent in the LMAX messaging layer. What do y'all think? >>>>>> Any ideas appreciated as always. >>>>>> >>>>>> Thanks! :) >>>>>> >>>>>> --John >>>>> >>>>> >>>>> >>>>> -- >>>>> Nick R. Katsipoulakis, >>>>> Department of Computer Science >>>>> University of Pittsburgh >>>> >>>> >>>> >>>> -- >>>> Nick R. Katsipoulakis, >>>> Department of Computer Science >>>> University of Pittsburgh >
