Re: Subtask distribution in Flink

2023-03-29 Thread Weihua Hu
For metrics about failover, you can refer to [1] [1] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/#availability Best, Weihua On Tue, Mar 28, 2023 at 1:34 AM santhosh venkat < santhoshvenkat1...@gmail.com> wrote: > Hi, > > Thank you so much for taking time to answer my

Re: Subtask distribution in Flink

2023-03-27 Thread santhosh venkat
Hi, Thank you so much for taking time to answer my questions and pointing me to relevant documentation. Really appreciate it. When the task failover happens, are there internal metrics in Flink at a job level to track the new execution attempt? Is there a way for the application owner to figure

Re: Subtask distribution in Flink

2023-03-27 Thread Weihua Hu
Hi, 1. Does this mean that each task slot will contain an entire pipeline in > the job? not exactly, each slot will run a subtask of each task. If the job is so simple that there is no keyby logic and we do not enable rebalance shuffle type, each slot could run all the pipeline. But if not we n

Subtask distribution in Flink

2023-03-27 Thread santhosh venkat
Hi, I am trying to understand how subtask distribution works in Flink. Let's assume a setup of a Flink cluster with a fixed number of TaskManagers in a kubernetes cluster. Let's say I have a flink job with all the operators having the same parallelism and with the same Slot sharing group. The ope