[
https://issues.apache.org/jira/browse/FLINK-30120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Weinwurm updated FLINK-30120:
-------------------------------------
Description:
Hey all,
We've noticed that a single task failure causes all of the Statefun tasks to be
restarted.
For example, a single task fails because of some Statefun Endpoint
unavailability or if one of our Kuberentes TaskManager pods go down.
Flink then determines that the _region_ failover strategy requires all tasks to
be restarted so we see this in the logs:
{code:java}
Nov 17 10:20:30
org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy
[] - Calculating tasks to restart to recover the failed task
31284d56d1e2112b0f20099ee448a6a9_11.
Nov 17 10:20:30
org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy
[] - 5650 tasks should be restarted to recover the failed task
31284d56d1e2112b0f20099ee448a6a9_11. {code}
Our tasks are all fully independent so I would like that only the one failed
task to get restarted or moved to a different TaskManager slot.
Is there a way to tell Flink to only restart the failed task? Or is there a
specific reason why the region failover strategy decides to restart all tasks?
If not, we'd really appreciate a way to enable individual task failovers.
Thanks in advance!
Stephan
was:
Hey all,
We've noticed that a single task failure causes all of the Statefun tasks to be
restarted.
For example, a single task fails because of some Statefun Endpoint
unavailability or if one of our Kuberentes TaskManager pods go down.
Flink then determines that the _region_ failover strategy requires all tasks to
be restarted so we see this in the logs:
{code:java}
Nov 17 10:20:30
org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy
[] - Calculating tasks to restart to recover the failed task
31284d56d1e2112b0f20099ee448a6a9_11.
Nov 17 10:20:30
org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy
[] - 5650 tasks should be restarted to recover the failed task
31284d56d1e2112b0f20099ee448a6a9_11. {code}
Our tasks are all fully independent so I would like that only the one failed
task to get restarted or moved to a different TaskManager slot.
Is there a way to tell Flink to only restart the failed task? Or is there a
specific reason why the region failover strategy decides to restart all tasks?
Thanks in advance!
Stephan
> Flink Statefun Task Failure causes Restart of all Tasks with Regional
> Failover Strategy
> ---------------------------------------------------------------------------------------
>
> Key: FLINK-30120
> URL: https://issues.apache.org/jira/browse/FLINK-30120
> Project: Flink
> Issue Type: Improvement
> Components: Stateful Functions
> Affects Versions: statefun-3.0.0, statefun-3.2.0, statefun-3.1.1
> Reporter: Stephan Weinwurm
> Priority: Major
>
> Hey all,
> We've noticed that a single task failure causes all of the Statefun tasks to
> be restarted.
> For example, a single task fails because of some Statefun Endpoint
> unavailability or if one of our Kuberentes TaskManager pods go down.
> Flink then determines that the _region_ failover strategy requires all tasks
> to be restarted so we see this in the logs:
>
> {code:java}
> Nov 17 10:20:30
> org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy
> [] - Calculating tasks to restart to recover the failed task
> 31284d56d1e2112b0f20099ee448a6a9_11.
> Nov 17 10:20:30
> org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy
> [] - 5650 tasks should be restarted to recover the failed task
> 31284d56d1e2112b0f20099ee448a6a9_11. {code}
> Our tasks are all fully independent so I would like that only the one failed
> task to get restarted or moved to a different TaskManager slot.
> Is there a way to tell Flink to only restart the failed task? Or is there a
> specific reason why the region failover strategy decides to restart all tasks?
> If not, we'd really appreciate a way to enable individual task failovers.
> Thanks in advance!
> Stephan
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
