Re: Scheduling of GroupByKey and CombinePerKey operations

2018-01-26 Thread Aljoscha Krettek
Hi, What are the other stages in that program? Best, Aljoscha > On 18. Jan 2018, at 16:22, Fabian Hueske wrote: > > Hi Pawel, > > This question might be better suited for the Beam user list. > Beam includes the Beam Flink runner which translates Beam programs into Flink > programs. > > Best

Re: Scheduling of GroupByKey and CombinePerKey operations

2018-01-18 Thread Fabian Hueske
Hi Pawel, This question might be better suited for the Beam user list. Beam includes the Beam Flink runner which translates Beam programs into Flink programs. Best, Fabian 2018-01-18 16:02 GMT+01:00 Pawel Bartoszek : > Can I ask why some operations run only one slot? I understand that file > wr

Scheduling of GroupByKey and CombinePerKey operations

2018-01-18 Thread Pawel Bartoszek
Can I ask why some operations run only one slot? I understand that file writes should happen only one one slot but GroupByKey operation could be distributed across all slots. I am having around 20k distinct keys every minute. Is there any way to break this operator chain? I noticed that CombinePer