Re: large sliding window perf question

2017-05-29 Thread Chen Qin
;> Stefan, in our case we are already on Flink 1.2.1 which should have the >>>> patched version of RocksDB, right? Because that patch did solve an issue >>>> we had in a different Flink job (a Kafka Source -> HDFS/Bucketing Sink >>>> which was

Re: large sliding window perf question

2017-05-29 Thread Chen Qin
Source -> HDFS/Bucketing Sink which >>> was stalling quite often under Flink 1.2.0) but did not solve this case, >>> which fits the “way too much RocksDB access” explanation better. >>> >>> >>> Thanks again, >>> Carst >>> &

Re: large sliding window perf question

2017-05-29 Thread Aljoscha Krettek
ain, >> Carst >> >> From: Aljoscha Krettek mailto:aljos...@apache.org>> >> Date: Wednesday, May 24, 2017 at 16:13 >> To: Stefan Richter > <mailto:s.rich...@data-artisans.com>> >> Cc: Carst Tankink mailto:ctank...@bol.com>>, >> &q

Re: large sliding window perf question

2017-05-24 Thread Chen Qin
> *From: *Aljoscha Krettek > *Date: *Wednesday, May 24, 2017 at 16:13 > *To: *Stefan Richter > *Cc: *Carst Tankink , "user@flink.apache.org" < > user@flink.apache.org> > *Subject: *Re: large sliding window perf question > > Hi, > > I’m afraid you’re runn

Re: large sliding window perf question

2017-05-24 Thread Stefan Richter
FoldingState, but > why this so slow is beyond me. > > Hope this info helps in figuring out what is going on, and hopefully it is > actually related to Chen’s case :) > > > Thanks, > Carst > > From: Stefan Richter <mailto:s.rich...@data-artisans.com>> &

Re: large sliding window perf question

2017-05-24 Thread Aljoscha Krettek
explanation better. > > > Thanks again, > Carst > > From: Aljoscha Krettek > Date: Wednesday, May 24, 2017 at 16:13 > To: Stefan Richter > Cc: Carst Tankink , "user@flink.apache.org" > > Subject: Re: large sliding window perf question > &g

Re: large sliding window perf question

2017-05-24 Thread Carst Tankink
2.0) but did not solve this case, which fits the “way too much RocksDB access” explanation better. Thanks again, Carst From: Aljoscha Krettek Date: Wednesday, May 24, 2017 at 16:13 To: Stefan Richter Cc: Carst Tankink , "user@flink.apache.org" Subject: Re: large sliding window perf

Re: large sliding window perf question

2017-05-24 Thread Aljoscha Krettek
e this info helps in figuring out what is going on, and hopefully it is >> actually related to Chen’s case :) >> >> >> Thanks, >> Carst >> >> From: Stefan Richter > <mailto:s.rich...@data-artisans.com>> >> Date: Tuesday, May 23, 2017 at

Re: large sliding window perf question

2017-05-24 Thread Stefan Richter
> actually related to Chen’s case :) > > > Thanks, > Carst > > From: Stefan Richter > Date: Tuesday, May 23, 2017 at 21:35 > To: "user@flink.apache.org" > Subject: Re: large sliding window perf question > > Hi, > > Which state backend

Re: large sliding window perf question

2017-05-24 Thread Carst Tankink
hopefully it is actually related to Chen’s case :) Thanks, Carst From: Stefan Richter Date: Tuesday, May 23, 2017 at 21:35 To: "user@flink.apache.org" Subject: Re: large sliding window perf question Hi, Which state backend and Flink version are you using? There was a problem with lar

Re: large sliding window perf question

2017-05-23 Thread Stefan Richter
Hi, Which state backend and Flink version are you using? There was a problem with large merging states on RocksDB, caused by some inefficiencies in the merge operator of RocksDB. We provide a custom patch for this with all newer versions of Flink. Best, Stefan > Am 23.05.2017 um 21:24 schrieb