1996fanrui opened a new pull request, #29374: URL: https://github.com/apache/flink/pull/29374
## What is the purpose of the change StreamTask replaces a ForwardPartitioner with a RebalancePartitioner when the parallelism of a FORWARD edge doesn't match (FLINK-30213). The check compares the producer parallelism with the number of subpartitions, which is not the consumer parallelism on a POINTWISE edge. Equal parallelism > 1 is taken as a mismatch, and N -> M with N > 1 can go undetected (e.g. 2 -> 4 with the DefaultScheduler, where half of the consumers get no data). Found in https://github.com/apache/flink/pull/29272#discussion_r4088147465. ## Brief change log - Pass the parallelism of the consuming job vertices through `ResultPartitionDeploymentDescriptor` and expose it via `Environment#getWriterConsumerParallelism`. - `StreamTask` compares the producer parallelism with it instead of the number of subpartitions, and logs both values. ## Verifying this change - `StreamTaskTest#testForwardPartitionerIsConvertedToRebalanceOnParallelismChanges` now simulates the change via the consumer parallelism. - `ResultPartitionDeploymentDescriptorTest` covers the new field. - End-to-end coverage for all schedulers and parallelism combinations is added in #29272, which is stacked on this PR. ## Does this pull request potentially affect one of the following parts - Dependencies: no - The public API: no - The serializers: no - The runtime per-record code paths: no (the check runs once when record writers are created) - Anything that affects deployment or recovery: yes (`ResultPartitionDeploymentDescriptor` carries one more int) - The S3 file system connector: no ## Documentation - New feature: no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
