Hi All, In the past, we experienced lots of problems with running Kafka Stream Applications on AWS.
We've seen issues with state locking, memory spiking to 100% and the instance dying, very slow startup on pulling down initial rocksdbs and so on.... Today we realised that the instances were experiencing such high throughput that the IOPS credits on our EBSs storing the rocks EBS were completely used up resulting in very slow IO and most of our issues. Sharing this as others might have the same problems. We were running on EBSs of 100GB which give 300 "provisioned" IOPS (3 iops per 1gb). Most cost effective solution for us is to just increase the EBS size to get more "provisioned" IOPS. Alternative solutions are to use i3.* instances or IO1 EBS volumes but they are expensive. Hope this helps somebody else, Ian.
