Hi Patrick, Flink supports regional failover [1] which only restarts all tasks connected via pipelined data exchanges. Hence, either when having an embarrassingly parallel topology or running a batch job, Flink should not restart the whole job in case of a task failure.
However, in the case of side outputs, I think they are connected via pipelined data exchanges with the main stream and, hence, are part of the same failover region as the main stream. [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/task_failure_recovery.html#restart-pipelined-region-failover-strategy Cheers, Till On Tue, Nov 24, 2020 at 5:15 PM Eifler, Patrick <patrick.eif...@sony.com> wrote: > Hi all, > > > > We are trying to setup regions to enable Flink to only stop failing tasks > based on region instead of failing the entire stream. > > We are using one main stream that is reading from a kafka topic and a > bunch of side outputs for processing each event from that topic differently. > > For the processing in the side outputs we use the process function > provided by flink. > > > > So far when one side output stream failed, the whole stream job failed. > > > > Is there anything that needs to be done or set on the Side Outputs so that > Flink recognizes them as regions? > > Is it even possible to have Flink handle side outputs as regions and > restart only one specific side output stream on failure? > > > > Many thanks in advance! > > > > Cheers, > > > > Patrick > > -- > > Patrick Eifler > > > > Senior Software Engineer (BI) > > Cloud Gaming Engineering & Infrastructure > Sony Interactive Entertainment LLC > > Wilhelmstraße 118, 10963 Berlin > > > Germany > > E: patrick.eif...@sony.com >
