One correction, my msg size is about 3k each. I did another round for comparison. I disabled acking all together, still the throughput is only slightly better at 12k tuples/s. So I used kafka's console consumer from one of the cluster nodes (different from where the partition is located) in order to find out if network was the bottleneck. This time I easily achieved 70k+ tuples/s.
Any thoughts on this? Thanks On Wed, Jun 3, 2015 at 10:55 PM, Fang Chen <[email protected]> wrote: > My use case requires total order in kafka queue, so I tested with a topic > with only 1 partition. My spout parallelism was set to 1, and bolt > parallelism 20. The message size is less than 1k bytes each. > > No matter how I tune kafka spout configs, including those queue fetch > related params, and max spout pending, I could only get about 10K tuples/s > with very low complete latency (<10ms) > > I even tried with empty bolt that acks tuples immediately without any > extra processing. Still the throughput is similar, though the complete > latency was even lower. This makes me wonder if I hit some sort of perf. > walls. > > My boxes are quite powerful baremetals (40 cores, lots of disk space, 96G > memory, 1G network), also the worker jvm was tuned so negligible pauses > there. > > Any advice on what I can tune or look into? > > Thanks a lot! > > Fang >
