S3 file source parallelism reverting to 1

[Steve Whelan]

Hi,

I have an S3 file source being consumed
as FileProcessingMode.PROCESS_CONTINUOUSLY with a parallelism of 3. I can
confirm the parallelism is set by printing it out. However, in the UI, the
file source has a parallelism of 1. I'm not changing it after its being
initially set.


DataStream s = env.readFile(
            new JsonInputFormat(...),
            filePath,
            FileProcessingMode.PROCESS_CONTINUOUSLY,
            5,
            myTypeInformation)
            .setParallelism(3);
System.out.println(s.getParallelism());  // prints 3


The DataStreamSource is a parallel operator otherwise `setParallelism(3)`
would throw an `IllegalArgumentException`[1]. The only other thing I do
with the DataStream is register it with the TableEnvironment.


tableEnvironment.registerDataStream("my_table", dataStream);


Is Flink resetting it to 1 for some reason? I'm running v1.9.0.

Thanks,

Steve


[1]
https://github.com/apache/flink/blob/release-1.9.0/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStreamSource.java#L53