Hi, George, Could you share w/ us the code and configuration of your sample test job? Thanks!
-Yi On Wed, May 20, 2015 at 1:19 PM, George Li <g...@ca.ibm.com> wrote: > Hi, > > We are evaluating Samza's performance, and our sample job with > TestPerformanceTask is much slower than a program reading directly from > Kafka. > > Scenario: > > * Cluster: > 1 master node for Zookeeper and yarn. > 3 Kafka broker nodes > 3 yarn worker nodes > > * Kafka: > Topic has only 1 partition. Average message size is around 100 byte. > On a yarn worker node, run the performance test program from Kafka > repository to read the topic. The throughput is about 400k messages/sec > > *Samza > Run TestPerformanceTask from Samza repository with no output stream > defined, and the throughput no more than 130k messages/sec > > > How can I explain/fix this performance difference? > > What I have done so far: > > 1. Monitor yarn worker node resource usage. > When the job is running, cpu and memory usage are never more than 5% > except at the beginning of the run. No significant network and disk IO > either > > 2. Monitor worker node network traffic > Tcpdump shows an interesting pattern. The yarn worker node will fetch a > block of data from the kafka broker, and after that, it will handshake > with the same kafka broker once every 100 ms for 300 ms before fetching > the next block. > > If I increase systems.kafka.samza.fetch.threshold to 500k, i.e., 10x the > default settings, this handshake lasts about 3 seconds. If I set > fetch.threshold to 250k, this idle period then becomes 1.5 sec. It seems > kafka consumer greatly outpaced process() call > > 3. Check Samza metrics > I do not see any excessive network calls to master or kafka broker, i.e., > no more than 30 calls/sec. However, process-ms and choose-ms are > approaching 2ms > > Any input would be greatly appreciated. > > Thanks, > > George > > > > >
