Hi Fang, Can you elaborate more on what kind of testing you performed with "Kafka's consumer console"?
Moreover, can you try creating more partitions in your kafka topics and then increase the kafka spout parallelism and see if that works? it will be great if you can paste how u are building your topology Regards, Nitin Kumar Sharma. On Thu, Jun 4, 2015 at 3:03 PM, Fang Chen <[email protected]> wrote: > One correction, my msg size is about 3k each. > > I did another round for comparison. I disabled acking all together, still > the throughput is only slightly better at 12k tuples/s. So I used kafka's > console consumer from one of the cluster nodes (different from where the > partition is located) in order to find out if network was the bottleneck. > This time I easily achieved 70k+ tuples/s. > > Any thoughts on this? > > Thanks > > On Wed, Jun 3, 2015 at 10:55 PM, Fang Chen <[email protected]> wrote: > >> My use case requires total order in kafka queue, so I tested with a topic >> with only 1 partition. My spout parallelism was set to 1, and bolt >> parallelism 20. The message size is less than 1k bytes each. >> >> No matter how I tune kafka spout configs, including those queue fetch >> related params, and max spout pending, I could only get about 10K tuples/s >> with very low complete latency (<10ms) >> >> I even tried with empty bolt that acks tuples immediately without any >> extra processing. Still the throughput is similar, though the complete >> latency was even lower. This makes me wonder if I hit some sort of perf. >> walls. >> >> My boxes are quite powerful baremetals (40 cores, lots of disk space, 96G >> memory, 1G network), also the worker jvm was tuned so negligible pauses >> there. >> >> Any advice on what I can tune or look into? >> >> Thanks a lot! >> >> Fang >> > >
