Re: large sliding window perf question

2017-05-29 Thread Chen Qin
;> Stefan, in our case we are already on Flink 1.2.1 which should have the >>>> patched version of RocksDB, right? Because that patch did solve an issue >>>> we had in a different Flink job (a Kafka Source -> HDFS/Bucketing Sink >>>> which was

Re: large sliding window perf question

2017-05-29 Thread Chen Qin
Source -> HDFS/Bucketing Sink which >>> was stalling quite often under Flink 1.2.0) but did not solve this case, >>> which fits the “way too much RocksDB access” explanation better. >>> >>> >>> Thanks again, >>> Carst >>> &

Re: large sliding window perf question

2017-05-29 Thread Aljoscha Krettek
ain, >> Carst >> >> From: Aljoscha Krettek mailto:aljos...@apache.org>> >> Date: Wednesday, May 24, 2017 at 16:13 >> To: Stefan Richter > <mailto:s.rich...@data-artisans.com>> >> Cc: Carst Tankink mailto:ctank...@bol.com>>, >> &q

Re: large sliding window perf question

2017-05-24 Thread Chen Qin
> *From: *Aljoscha Krettek > *Date: *Wednesday, May 24, 2017 at 16:13 > *To: *Stefan Richter > *Cc: *Carst Tankink , "user@flink.apache.org" < > user@flink.apache.org> > *Subject: *Re: large sliding window perf question > > Hi, > > I’m afraid you’re runn

Re: large sliding window perf question

2017-05-24 Thread Stefan Richter
FoldingState, but > why this so slow is beyond me. > > Hope this info helps in figuring out what is going on, and hopefully it is > actually related to Chen’s case :) > > > Thanks, > Carst > > From: Stefan Richter <mailto:s.rich...@data-artisans.com>> &

Re: large sliding window perf question

2017-05-24 Thread Aljoscha Krettek
explanation better. > > > Thanks again, > Carst > > From: Aljoscha Krettek > Date: Wednesday, May 24, 2017 at 16:13 > To: Stefan Richter > Cc: Carst Tankink , "user@flink.apache.org" > > Subject: Re: large sliding window perf question > &g

Re: large sliding window perf question

2017-05-24 Thread Carst Tankink
2.0) but did not solve this case, which fits the “way too much RocksDB access” explanation better. Thanks again, Carst From: Aljoscha Krettek Date: Wednesday, May 24, 2017 at 16:13 To: Stefan Richter Cc: Carst Tankink , "user@flink.apache.org" Subject: Re: large sliding window perf

Re: large sliding window perf question

2017-05-24 Thread Aljoscha Krettek
e this info helps in figuring out what is going on, and hopefully it is >> actually related to Chen’s case :) >> >> >> Thanks, >> Carst >> >> From: Stefan Richter > <mailto:s.rich...@data-artisans.com>> >> Date: Tuesday, May 23, 2017 at

Re: large sliding window perf question

2017-05-24 Thread Stefan Richter
> actually related to Chen’s case :) > > > Thanks, > Carst > > From: Stefan Richter > Date: Tuesday, May 23, 2017 at 21:35 > To: "user@flink.apache.org" > Subject: Re: large sliding window perf question > > Hi, > > Which state backend

Re: large sliding window perf question

2017-05-24 Thread Carst Tankink
hopefully it is actually related to Chen’s case :) Thanks, Carst From: Stefan Richter Date: Tuesday, May 23, 2017 at 21:35 To: "user@flink.apache.org" Subject: Re: large sliding window perf question Hi, Which state backend and Flink version are you using? There was a problem with lar

Re: large sliding window perf question

2017-05-23 Thread Stefan Richter
Hi, Which state backend and Flink version are you using? There was a problem with large merging states on RocksDB, caused by some inefficiencies in the merge operator of RocksDB. We provide a custom patch for this with all newer versions of Flink. Best, Stefan > Am 23.05.2017 um 21:24 schrieb

large sliding window perf question

2017-05-23 Thread Chen Qin
Hi there, I have seen some weird perf issue while running event time based job with large sliding window (24 hours offset every 10s) pipeline looks simple, tail kafka topic and assign timestamp and watermark, forward to large sliding window (30days) and fire every 10 seconds and print out. what