Hi, We are evaluating Samza's performance, and our sample job with TestPerformanceTask is much slower than a program reading directly from Kafka.
Scenario: * Cluster: 1 master node for Zookeeper and yarn. 3 Kafka broker nodes 3 yarn worker nodes * Kafka: Topic has only 1 partition. Average message size is around 100 byte. On a yarn worker node, run the performance test program from Kafka repository to read the topic. The throughput is about 400k messages/sec *Samza Run TestPerformanceTask from Samza repository with no output stream defined, and the throughput no more than 130k messages/sec How can I explain/fix this performance difference? What I have done so far: 1. Monitor yarn worker node resource usage. When the job is running, cpu and memory usage are never more than 5% except at the beginning of the run. No significant network and disk IO either 2. Monitor worker node network traffic Tcpdump shows an interesting pattern. The yarn worker node will fetch a block of data from the kafka broker, and after that, it will handshake with the same kafka broker once every 100 ms for 300 ms before fetching the next block. If I increase systems.kafka.samza.fetch.threshold to 500k, i.e., 10x the default settings, this handshake lasts about 3 seconds. If I set fetch.threshold to 250k, this idle period then becomes 1.5 sec. It seems kafka consumer greatly outpaced process() call 3. Check Samza metrics I do not see any excessive network calls to master or kafka broker, i.e., no more than 30 calls/sec. However, process-ms and choose-ms are approaching 2ms Any input would be greatly appreciated. Thanks, George
