Re: How to setup Regions for Fault Tolerance in Flink when using Side Outputs

2020-11-25 Thread Till Rohrmann
Patrick Eifler > > > > Senior Software Engineer (BI) > > Cloud Gaming Engineering & Infrastructure > Sony Interactive Entertainment LLC > > Wilhelmstraße 118, 10963 Berlin > > > Germany > > E: patrick.eif...@sony.com > > > > *From: *Ti

Re: How to setup Regions for Fault Tolerance in Flink when using Side Outputs

2020-11-25 Thread Eifler, Patrick
many E: patrick.eif...@sony.com From: Till Rohrmann Date: Tuesday, 24. November 2020 at 17:20 To: "Eifler, Patrick" Cc: "user@flink.apache.org" Subject: Re: How to setup Regions for Fault Tolerance in Flink when using Side Outputs Hi Patrick, Flink supports regional

Re: How to setup Regions for Fault Tolerance in Flink when using Side Outputs

2020-11-24 Thread Till Rohrmann
Hi Patrick, Flink supports regional failover [1] which only restarts all tasks connected via pipelined data exchanges. Hence, either when having an embarrassingly parallel topology or running a batch job, Flink should not restart the whole job in case of a task failure. However, in the case of si

How to setup Regions for Fault Tolerance in Flink when using Side Outputs

2020-11-24 Thread Eifler, Patrick
Hi all, We are trying to setup regions to enable Flink to only stop failing tasks based on region instead of failing the entire stream. We are using one main stream that is reading from a kafka topic and a bunch of side outputs for processing each event from that topic differently. For the proce