Re: List of consumed kafka topics should not be restored from state

Hello Gordon, Thank sounds good to me. I prepared a patch for that. Will add you in the loop. Cheers, Feng Le sam. 16 févr. 2019 à 05:30, Tzu-Li (Gordon) Tai a écrit : > Hi, > > I indeed don't have a specific use case in mind that justifies the current > behaviour. > My only concern is that sin

Re: List of consumed kafka topics should not be restored from state

Hi, I indeed don't have a specific use case in mind that justifies the current behaviour. My only concern is that since this was the default behaviour for a quite a while already, I can't be 100% that there are some user who actually expects the current behaviour. That may be dangerous since we wo

Re: List of consumed kafka topics should not be restored from state

Hello Gordon, Thanks for adding the contributor permission. :) Agree with Gyula, I would vote +1 for changing this behavior unless we have use cases for maintaining it. I consider it more like a bug other than expected behavior. We had one counter example in production when migrating from one to

Re: List of consumed kafka topics should not be restored from state

Gordon, Do you have an example where the current default behaviour makes sense and it doesnt cause unexpected problems? Or an example where someone might reasonably expect the current behaviour instead of the newly suggested one. If we have such cases I would agree lets keep it as is. If we cant c

Re: List of consumed kafka topics should not be restored from state

Hi Feng, Thanks for working on a fix for this. I gave you contributor permission on JIRA. Before you jump right onto the code: do we have an agreement already on whether we change the default behaviour directly, or add a configuration option (e.g. add a `filterRestoredPartitionOffsetState()` meth

Re: List of consumed kafka topics should not be restored from state

Hello Aljoscha, Thanks for sharing the ticket, I think it makes sense to reopen the ticket. (I can work on the fix for this, should be a small patch, just add a filter when restoring Kafka partitions with those discovered partitions). (btw. Can I have a contributor access for jira, my username is

Re: List of consumed kafka topics should not be restored from state

I think these two Jira issues are relevant here: - https://issues.apache.org/jira/browse/FLINK-10342 - https://issues.apache.org/jira/browse/FLINK-9303 The second one only because it’s slight

Re: List of consumed kafka topics should not be restored from state

Hi! I agree that it’s very confusing if you explicitly specify the topics that are to be confusing and what happens is different. I would almost consider this to be a bug , can’t see any reasonable use case just hard to debug problems . Having an option would be a good start but I would rather t

Re: List of consumed kafka topics should not be restored from state

Hello there, I’m just wondering if there are real world use cases for maintaining this default behavior. It’s a bit counter intuitive and sometimes results in serious production issues. ( We had a similar issue when changing the topic name, and resulting reading every message twice - both from the

Re: List of consumed kafka topics should not be restored from state

Hi, Partition offsets stored in state will always be respected when the consumer is restored from checkpoints / savepoints. AFAIK, this seems to have been the behaviour for quite some time now (since FlinkKafkaConsumer08). I think in the past there were some discussion to at least allow some way

List of consumed kafka topics should not be restored from state

Hi! I have run into a weird issue which I could have sworn that it wouldnt happen :D I feel there was a discussion about this in the past but maybe im wrong, but I hope someone can point me to a ticket. Lets say you create a kafka consumer that consumes (t1,t2,t3), you take a savepoint and deploy