Hi, 1. Does this mean that each task slot will contain an entire pipeline in > the job?
not exactly, each slot will run a subtask of each task. If the job is so simple that there is no keyby logic and we do not enable rebalance shuffle type, each slot could run all the pipeline. But if not we need to shuffle data to other subtasks. You can get some examples from [1]. 2. Upon a TM pod failure and after K8s brings back the TM pod, would flink > assign the same subtasks back to restarted TM again? Or will they be > distributed to different TaskManagers? If there is no shuffle data in your job (described in 1), only tasks on failure pods will be restarted, and they will be assigned to the new TM again. But if not, all the related tasks will be restarted. When these tasks re-scheduled, there are some strategy to assign slots. They will try to assign the task to previous slot to reduce the recovery time, But there is no guarantee. You can read [2] to get more information about failure recovery. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/concepts/flink-architecture/#tasks-and-operator-chains [2] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/task_failure_recovery/ Best, Weihua On Mon, Mar 27, 2023 at 3:22 PM santhosh venkat < santhoshvenkat1...@gmail.com> wrote: > Hi, > > I am trying to understand how subtask distribution works in Flink. Let's > assume a setup of a Flink cluster with a fixed number of TaskManagers in a > kubernetes cluster. > > Let's say I have a flink job with all the operators having the same > parallelism and with the same Slot sharing group. The operator parallelism > is computed as the number of task managers multiplied by number of task > slots per TM. > > 1. Does this mean that each task slot will contain an entire pipeline in > the job? > 2. Upon a TM pod failure and after K8s brings back the TM pod, would flink > assign the same subtasks back to restarted TM again? Or will they be > distributed to different TaskManagers? > > It would be great if someone can answer this question. > > Thanks. >
