I thought for sure this was already the existing behavior with this operator. Does it not check the mailbox executor after every record read?
On Tue, Dec 5, 2023 at 6:48 AM Prabhu Joseph (Jira) <j...@apache.org> wrote: > Prabhu Joseph created FLINK-33753: > ------------------------------------- > > Summary: ContinuousFileReaderOperator consume records as mini > batch > Key: FLINK-33753 > URL: https://issues.apache.org/jira/browse/FLINK-33753 > Project: Flink > Issue Type: Improvement > Affects Versions: 1.18.0 > Reporter: Prabhu Joseph > > > The ContinuousFileReaderOperator reads and collects the records from a > split in a loop. If the split size is large, then the loop will take more > time, and then the mailbox executor won't have a chance to process the > checkpoint barrier. This leads to checkpoint timing out. > ContinuousFileReaderOperator could be improved to consume the records in a > mini batch, similar to Hudi's StreamReadOperator ( > https://issues.apache.org/jira/browse/HUDI-2485). > > > > -- > This message was sent by Atlassian Jira > (v8.20.10#820010) >
