Re: Porting batch percentile computation to streaming window

2017-05-31 Thread Gyula Fóra
s running on from Flink. > > > - Original Message - > From: > "Gyula Fóra" > > To: > "William Saar" , > Cc: > > Sent: > > Tue, 30 May 2017 13:56:08 + > > > Subject: > Re: Porting batch percentile computation

Re: Porting batch percentile computation to streaming window

2017-05-30 Thread William Saar
Flink. - Original Message - From: "Gyula Fóra" To: "William Saar" , Cc: Sent: Tue, 30 May 2017 13:56:08 +0000 Subject: Re: Porting batch percentile computation to streaming window I think you could actually do a window operation to get the tDigestStre

Re: Porting batch percentile computation to streaming window

2017-05-30 Thread Gyula Fóra
ess the watermark/window information in my fold > operation in line 1? > > Thanks! > > > - Original Message - > From: > "Gyula Fóra" > > To: > "William Saar" , > Cc: > > Sent: > Tue, 30 May 2017 08:56:28 + > Subject: > Re: Porti

Re: Porting batch percentile computation to streaming window

2017-05-30 Thread William Saar
May 2017 08:56:28 +0000 Subject:Re: Porting batch percentile computation to streaming window Hi William, I think basically the feature you are looking for are side inputs which is not implemented yet but let me try to give a workaround that might work.  If I understand correctly you have tw

Re: Porting batch percentile computation to streaming window

2017-05-30 Thread Gyula Fóra
Hi William, I think basically the feature you are looking for are side inputs which is not implemented yet but let me try to give a workaround that might work. If I understand correctly you have two windowed computations: TDigestStream = allMetrics.windowAll(...).reduce() windowMetricsByIP = allM

Porting batch percentile computation to streaming window

2017-05-29 Thread William Saar
I am porting a calculation from Spark batches that uses broadcast variables to compute percentiles from metrics and curious for tips on doing this with Flink streaming. I have a windowed computation where I am compute metrics for IP-addresses (a windowed stream of metrics objects grouped by IP-add