Re: Data sources and slices

2017-10-27 Thread Till Rohrmann
Hi David, in case of a streaming program with a degree of parallelism of 128, Flink would create 128 splits. One split for each parallel sub task. The logic is that one split will have the size of one block unless this would not give you enough splits such that every task receives at least one spl

Data sources and slices

2017-10-26 Thread David Dreyfus
Hello, If I am on a cluster with 2 task managers with 64 CPUs each, I can configure 128 slots in accordance with the documentation. If I set parallelism to 128 and read a 64 MB file (one datasource with a single file), will flink really create 500K slices? Or, will it check the default blocksize o