Re: Flink Streaming Job Tuning help

2020-05-18 Thread Arvid Heise
Thanks for your suggestions. We will keep it in mind! > > > > Kumar > > > > *From: *Zhijiang > *Reply-To: *Zhijiang > *Date: *Tuesday, May 12, 2020 at 10:10 PM > *To: *Senthil Kumar , "user@flink.apache.org" < > user@flink.apache.org> > *Subject

Re: Flink Streaming Job Tuning help

2020-05-13 Thread Senthil Kumar
Zhijiang, Thanks for your suggestions. We will keep it in mind! Kumar From: Zhijiang Reply-To: Zhijiang Date: Tuesday, May 12, 2020 at 10:10 PM To: Senthil Kumar , "user@flink.apache.org" Subject: Re: Flink Streaming Job Tuning help Hi Kumar, I can give some general ideas f

Re: Flink Streaming Job Tuning help

2020-05-12 Thread Zhijiang
Time:2020年5月13日(星期三) 00:49 To:user@flink.apache.org Subject:Re: Flink Streaming Job Tuning help I forgot to mention, we are consuming said records from AWS kinesis and writing out to S3. From: Senthil Kumar Date: Tuesday, May 12, 2020 at 10:47 AM To: "user@flink.apache.org" Subjec

Re: Flink Streaming Job Tuning help

2020-05-12 Thread Senthil Kumar
I forgot to mention, we are consuming said records from AWS kinesis and writing out to S3. From: Senthil Kumar Date: Tuesday, May 12, 2020 at 10:47 AM To: "user@flink.apache.org" Subject: Flink Streaming Job Tuning help Hello Flink Community! We have a fairly intensive flink

Flink Streaming Job Tuning help

2020-05-12 Thread Senthil Kumar
Hello Flink Community! We have a fairly intensive flink streaming application, processing 8-9 million records a minute, with each record being 10k. One of our steps is a keyBy operation. We are finding that flink lags seriously behind when we introduce the keyBy (presumably because of shuffle ac