Re: How to use pipeline.jobvertex-parallelism-overrides property.

2023-08-25 Thread Krzysztof Chmielewski
Hi, thank you for replaying. Hang Ruan, regarding: " If they have different parallelisms, we cannot chain them together." <- that is clear for me. What I'm still not sure about is if " JobVertexId#toHexString" that should be used as key of pipeline.jobvertex-parallelism-overrides map is vertex o

Re: How to use pipeline.jobvertex-parallelism-overrides property.

2023-08-24 Thread Hang Ruan
Hi, Krzysztof. As liu ron said, the key of the map for this configuration is the value from JobVertexId#toHexString. Maybe we could improve the docs to provide more details. The condition that two operators have the same parallelism is a must for chaining them. If they have different parallelisms,

Re: How to use pipeline.jobvertex-parallelism-overrides property.

2023-08-24 Thread liu ron
Hi, Krzysztof As stated in the description section, this option is used to override the parallelism of a JobVertex, where the key is JobVertex id, you can see [1] for double check. A JobVertex may contain more than one operator, so we cannot override the parallelism of a given operator alone. One