The exception basically says the remote TM is unreachable, probably terminated due to some other reasons. This may not be the root cause. Is there any other exceptions in the log? Also, since the overall resource usage is almost full, could you try allocating more CPUs and see if the instability persists?
Best, Zhanghao Chen ________________________________ From: Oscar Perez <oscarfernando.pe...@n26.com> Sent: Monday, April 15, 2024 19:24 To: Zhanghao Chen <zhanghao.c...@outlook.com> Cc: Oscar Perez via user <user@flink.apache.org> Subject: Re: Flink job performance Hei, ok that is weird. Let me resend them. Regards, Oscar On Mon, 15 Apr 2024 at 14:00, Zhanghao Chen <zhanghao.c...@outlook.com<mailto:zhanghao.c...@outlook.com>> wrote: Hi, there seems to be sth wrong with the two images attached in the latest email. I cannot open them. Best, Zhanghao Chen ________________________________ From: Oscar Perez via user <user@flink.apache.org<mailto:user@flink.apache.org>> Sent: Monday, April 15, 2024 15:57 To: Oscar Perez via user <user@flink.apache.org<mailto:user@flink.apache.org>>; pi-team <pi-t...@n26.com<mailto:pi-t...@n26.com>>; Hermes Team <hermes-t...@n26.com<mailto:hermes-t...@n26.com>> Subject: Flink job performance Hi community! We have an interesting problem with Flink after increasing parallelism in a certain way. Here is the summary: 1) We identified that our job bottleneck were some Co-keyed process operators that were affecting on previous operators causing backpressure. 2( What we did was to increase the parallelism to all the operators from 6 to 12 but keeping 6 these operators that read from kafka. The main reason was that all our topics have 6 partitions so increasing the parallelism will not yield better performance See attached job layout prior and after the changes: What happens was that some operations that were chained in the same operator like reading - filter - map - filter now are rebalanced and the overall performance of the job is suffering (keeps throwing exceptions now and then) Is the rebalance operation going over the network or this happens in the same node? How can we effectively improve performance of this job with the given resources? Thanks for the input! Regards, Oscar
