So basically if I set startupOptions(StartupOptions.initial()) and also scan.incremental.snapshot.enabled = true Then it would read from the source in parallel, thereby reading the entire mongo collection faster.
Am I understanding that correctly? Also I am using Flink 1.8, would it work with this version of flink ? Thanks Sachin On Mon, Aug 19, 2024 at 2:29 PM Jiabao Sun <jiabao...@apache.org> wrote: > Hi Sachin, > > Incremental snapshot reading is a new feature introduced in Flink 2.0. > > It has the following capabilities: > - Source can be parallel during snapshot reading to improve snapshot speed > - Source can perform checkpoints in the chunk granularity during snapshot > reading > > Limitation: > - MongoDB version needs to be greater than 4.0 > > Best, > Jiabao > > On 2024/08/19 06:48:39 Sachin Mittal wrote: > > Hi, > > I am using mongodb cdc connector version 3.1.1 > > I am connecting to mongodb atlas, which uses mongodb version 7.0. > > > > In the cdc connector I find a property: > > > > scan.incremental.snapshot.enabled with default as false. > > > > I wanted to know in what cases we should set this as true and what does > > this property help with ? > > > > Please note that I am configuring my connector with: > > > > .startupOptions(StartupOptions.initial()) > > .batchSize(2048) > > > > Thanks > > Sachin > > >
