Qingsheng,
For the scenario described by Mason in the original email, I think it is
safe to remove split/topic upson recovery without worrying about data loss,
since it is a conscious choice by the user to switch to a different set of
topics.
I thought the problem is that KafkaSourceReader just r
Hi Mason,
Sorry for my late response!
> quote_type
> “there was no logic to filter/remove splits”
Yes we indeed miss a split removal mechanism. Actually this is quite a tricky
one considering exactly-once semantic: there’s risk of losing data if we remove
a partition / topic from Kafka. There w
Hi all,
Any update on this?
Best,
Mason
On Sat, Oct 30, 2021 at 5:56 AM Arvid Heise wrote:
> This seems to be a valid concern but I'm not deep enough to clearly say
> that this is indeed a bug. @renqschn could you
> please double-check?
>
> On Thu, Oct 28, 2021 at 8:39 PM Mason Chen wrote:
>
This seems to be a valid concern but I'm not deep enough to clearly say
that this is indeed a bug. @renqschn could you please
double-check?
On Thu, Oct 28, 2021 at 8:39 PM Mason Chen wrote:
> Hi all,
>
> I noticed that the KafkaSourceReader did not have a pointer to the
> KafkaSubscriber, so I
Hi all,
I noticed that the KafkaSourceReader did not have a pointer to the
KafkaSubscriber, so I was wondering if this could be a bug:
1. User has a flink job with topic set A and takes savepoint
2. User modifies flink job to read from topic set B; however, splits are
still read from topic set A
