recommendation for this limitation like size of files or number of
files or checkpointing state backend etc?
Regards,
Kirti Dhar
From: Martijn Visser
Sent: 20 April 2023 18:14
To: Kirti Dhar Upadhyay K
Cc: user@flink.apache.org
Subject: Re: SplitEnumerator and SourceReader
Hi Kirti Dhar,
1
Hi Kirti Dhar,
1. The SourceReader downloads the file, which is assigned to him by the
SplitEnumerator
2. This depends on the format; a BulkFormat like Parquet or ORC can be read
in batches of records at a time.
3. The SplitEnumerator runs on the JobManager, not on a TaskManager. Have
you read som
Hi Community,
I have started using file source of Flink 1.17.x recently.
I was going through the FLIP-27 documentation and as much I understand
SplitEnumerator lists files (splits) and assigns to SourceReader. A single
instance of SplitEnumerator runs whereas parallelism can be done on
SourceR
