Kafka on test cluster: 2 Kafka nodes, 2GB, 2CPUs3 Zookeeper nodes, 2GB, 2CPUs
Storm:3 nodes, 3CPUs each, on the same Zookeeper cluster as Kafka.
1 topic, 5 partitions, replication x2
Whether I use 1 slot for the Kafka Spout or 5 slots (=#partitions), the 
throughput seems about the same.
I can't seem to read much more than 7000 events/sec.
Same, on writing, I set a generator spout and write to Kafka on 1 
topic/5partitions with a KafkaBolt with parallelism of 5 and I can't seem to 
write much more than 7000 events/sec.
Meanwhile, none of the CPU, IO or MEM seem to be a bottleneck: In Storm UI the 
bolts all show capacities <50%, sometimes much less (in the single digit %)Top 
shows CPUs being used at ~30% max
We have another process moving data from Kafka to Cassandra and it gives 
similar throughput, so it seems related to Kafka more than Storm.

What could be wrong? Sorry for the generic question but I would appreciate any 
hint on where to start to troubleshoot.
Thanks                                    

Reply via email to