Re: Understanding blocking behavior

2021-02-16 Thread Arvid Heise
Hi Andreas, Julian already offered a good explanation, so here is one possible solution: you could try to run the whole first subpipeline with parallelism X and the second with P-X. However, most likely you need to run with P>X to finish in time. Another way is to use DataStream (your program is

Re: Understanding blocking behavior

2021-02-16 Thread Jaffe, Julian
Hey Andreas, Have a read through https://ci.apache.org/projects/flink/flink-docs-stable/dev/datastream_execution_mode.html#task-scheduling-and-network-shuffle and in particular the BATCH Execution Mode section. Your intuition is mostly correct – because your operators can’t be chained due to th