Re: Input Selectable & Checkpointing

2021-11-25 Thread Piotr Nowojski
you! > > Best wishes, > > Shazia > > > - Original message - > From: "Piotr Nowojski" > To: "Shazia Kayani" > Cc: mart...@ververica.com, "user" > Subject: [EXTERNAL] Re: Input Selectable & Checkpointing > Date: Wed, Nov

RE: Input Selectable & Checkpointing

2021-11-24 Thread Shazia Kayani
Hi Piotrek,Thanks for you message!Ok that does sound interesting and is a approach I had not considered before, will take a look into and further investigateThank you!Best wishes,Shazia   - Original message -From: "Piotr Nowojski" To: "Shazia Kayani" Cc: mart...@ververica.com, "user" Subjec

Re: Input Selectable & Checkpointing

2021-11-24 Thread Piotr Nowojski
there consistently being significantly more messages on one source > than another which causes issues when we there are too many messages on the > stream. > > Thanks > > Shazia > > > - Original message - > From: "Martijn Visser" > To: "Shazia Ka

RE: Input Selectable & Checkpointing

2021-11-23 Thread Shazia Kayani
Hi Martijn,Its a continuous requirement so always read from one input source over another, but its does not require a super strict guarantee, so it doesn't matter if on occasion a message is read from the wrong topic. It's mainly due to there consistently being significantly more messages on one so

Re: Input Selectable & Checkpointing

2021-11-23 Thread Martijn Visser
Hi, Do you have a requirement to continuously prioritise one input source over another (like always read topic X from Kafka before topic Y from Kafka) or is it a one-time effort, because you might need to bootstrap some state, so first read all data from file source A before switching over to topi