Kafka Stream Window

2021-10-25 Thread Mohammed Kamaal
Hi, Is there a way to define a sliding window with a count (number of occurrence) of stream in Kafka Stream Window (KStream)? In Flink we have, stream.keyBy().countWindow(3,1) //where 3 is number/size of streams Can you suggest an equivalent of the above in KStream The below code takes only dur

Re: Flink Performance Issue

2021-09-27 Thread Mohammed Kamaal
zed ?( if you are checkpointing to RocksDB, the > disk can dramatically slow you down) > Idea 4: Are you under high memory pressure, and your JVMs are spending most > of their cycles garbage collecting? > > My bet is you are not getting data into your cluster as fast as you think

Re: Flink Performance Issue

2021-09-22 Thread Mohammed Kamaal
nce is > counter-productive if you don't exactly know that you need it. > >> On Thu, Sep 2, 2021 at 1:36 PM Mohammed Kamaal >> wrote: >> Hi Fabian, >> >> Just an update, >> >> Problem 2:- >> >> Caused by: org.a

Re: Flink Performance Issue

2021-09-02 Thread Mohammed Kamaal
. Thanks On Wed, Sep 1, 2021 at 10:55 PM Mohammed Kamaal wrote: > > Hi Fabian, > > Problem 1:- > - > I have removed the print out sink's and ran the test again. This time > the throughput is 17 minutes for 20K records (200 records every > second). Earl

Flink Performance Issue

2021-08-24 Thread Mohammed Kamaal
Hi, Apologize for the big message, to explain the issue in detail. We have a Flink (version 1.8) application running on AWS Kinesis Analytics. The application has a source which is a kafka topic with 15 partitions (AWS Managed Streaming Kafka) and the sink is again a kafka topic with 15 partiti