Hi, We are testing flink and storm for our streaming pipelines on various features.
In terms of Latency,i see the flink comes up short on storm even if more CPU is given to it. Will Explain in detail. *Machine*. t2.large 4 core 16 gb. is used for Used for flink task manager and storm supervisor node. *Kafka Partitions* 4 *Messages tested:* 1million *Load* : 50k/sec *Scenario*: Read from Kafka -> Transform (Map to a different JSON format) - > Write to a Kafka topic. *Test 1* Storm Parallelism is set as 1. There are four processes. 1 Spout (Read from Kafka) and 3 bolts (Transformation and sink) . Flink. Operator level parallelism not set. Task Parallelism is set as 1. Task slot is 1 per core. Storm was 130 milliseconds faster in 1st record. Storm was 20 seconds faster in 1 millionth record. *Test 2* Storm Parallelism is set as 1. There are four processes. 1 Spout (Read from Kafka) and 3 bolts (Transformation and sink) Flink. Operator level parallelism not set. Task Parallelism is set as 4. Task slot is 1 per core. So all cores is used. Storm was 180 milliseconds faster in 1st record. Storm was 25 seconds faster in 1 millionth record. *Observations here* 1) Increasing Parallelism did not increase the performance in Flink rather it became 50ms to 5s slower. 2) Flink is slower in Reading from Kafka compared to storm. Thats where the bulk of the latency is. for the millionth record its 19-24 seconds slower. 3) Once message is read, flink takes lesser time to transform and write to kafka compared to storm. *Other Flink Config* jobmanager.heap.size: 1024m taskmanager.memory.process.size: 1568m *How do we improve the latency ? * *Why does latency becomes worse when parallelism is increased and matched to partitions?* Thanks, Prasanna.
