Hi, KIP is updated. Changes: 1. Approach discussed to keep both tools (streams application resetter and consumer group reset offset). 2. Options has been aligned between both tools. 3. Zookeeper option from streams-application-resetted will be removed, and replaced internally for Kafka AdminClient.
Looking forward to your feedback. El jue., 29 jun. 2017 a las 15:04, Matthias J. Sax (<matth...@confluent.io>) escribió: > Damian, > > > resets everything and clears up > >> the state stores. > > That's not correct. The reset tool does not touch local store. For this, > we have `KafkaStreams#clenup` -- otherwise, you would need to run the > tool in every machine you run an application instance. > > With regard to state, the tool only deletes the underlying changelog > topics. > > Just to clarify. The idea of allowing to rest with different offset is > to clear all state but just use a different start offset (instead of > zero). This is for use case where your topic has a larger retention time > than the amount of data you want to reprocess. But we always need to > start with an empty state. (Resetting with consistent state is something > we might do at some point, but it's much hard and not part of this KIP) > > > @matthias, could we remove the ZK dependency from the streams reset tool > > now? > > I think so. The new AdminClient provide the feature we need AFAIK. I > guess we can picky back this into the KIP (we would need a KIP anyway > because we deprecate `--zookeeper` what is an public API change). > > > I just want to point out, that even without ZK dependency, I prefer to > wrap the consumer offset tool and keep two tools. > > > -Matthias > > > On 6/29/17 9:14 AM, Damian Guy wrote: > > Hi, > > > > Thanks for the KIP. What is not clear is how is this going to handle > state > > stores? Right now the streams reset tool, resets everything and clears up > > the state stores. What are we going to do if we reset to a particular > > offset? If we clear the state then we've lost any previously aggregated > > values (which may or may not be what is expected). If we don't clear > them, > > then we will end up with incorrect aggregates. > > > > @matthias, could we remove the ZK dependency from the streams reset tool > > now? > > > > Thanks, > > Damian > > > > On Thu, 29 Jun 2017 at 15:22 Jorge Esteban Quilcate Otoya < > > quilcate.jo...@gmail.com> wrote: > > > >> You're right, I was not considering Zookeeper dependency. > >> > >> I'm starting to like the idea to wrap `reset-offset` from > >> `streams-application-reset`. > >> > >> I will wait a bit for more feedback and work on a draft with this > changes. > >> > >> > >> El mié., 28 jun. 2017 a las 0:20, Matthias J. Sax (< > matth...@confluent.io > >>> ) > >> escribió: > >> > >>> I agree, that we should not duplicate functionality. > >>> > >>> However, I am worried, that a non-streams users using the offset reset > >>> tool might delete topics unintentionally (even if the changes are > pretty > >>> small). Also, currently the stream reset tool required Zookeeper while > >>> the offset reset tool does not -- I don't think we should add this > >>> dependency to the offset reset tool. > >>> > >>> Thus, it think it might be better to keep both tools, but internally > >>> rewrite the streams reset entry class, to reuse as much as possible > from > >>> the offset reset tool. Ie. the streams tool would be a wrapper around > >>> the offset tool and add some functionality it needs that is Streams > >>> specific. > >>> > >>> I also think, that keeping separate tools for consumers and streams is > a > >>> good thing. We might want to add new features that don't apply to plain > >>> consumers -- note, a Streams applications is more than just a client. > >>> > >>> WDYT? > >>> > >>> Would be good to get some feedback from others, too. > >>> > >>> > >>> -Matthias > >>> > >>> > >>> On 6/27/17 9:05 AM, Jorge Esteban Quilcate Otoya wrote: > >>>> Thanks for the feedback Matthias! > >>>> > >>>> The main idea is to use only 1 tool to reset offsets and don't > >> replicate > >>>> functionality between tools. > >>>> Reset Offset (KIP-122) tool not only reset but support execute the > >> reset > >>>> but also export, import from files, etc. > >>>> If we extend the current tool (kafka-streams-application-reset.sh) we > >>> will > >>>> have to duplicate all this functionality also. > >>>> Maybe another option is to move the current implementation into > >>>> `kafka-consumer-group` and add a new command `--reset-offset-streams` > >>> with > >>>> the current implementation functionality and add `--reset-offset` > >> options > >>>> for input topics. Does this make sense? > >>>> > >>>> > >>>> El lun., 26 jun. 2017 a las 23:32, Matthias J. Sax (< > >>> matth...@confluent.io>) > >>>> escribió: > >>>> > >>>>> Jorge, > >>>>> > >>>>> thanks a lot for this KIP. Allowing the reset streams applications > >> with > >>>>> arbitrary start offset is something we got multiple requests already. > >>>>> > >>>>> Couple of clarification question: > >>>>> > >>>>> - why do you want to deprecate the current tool instead of extending > >>>>> the current tool with the stuff the offset reset tool can do (ie, use > >>>>> the offset reset tool internally) > >>>>> > >>>>> - you suggest to extend the offset reset tool to replace the stream > >>>>> reset tool: how would the reset tool know if it is resetting a > streams > >>>>> applications or a regular consumer group? > >>>>> > >>>>> > >>>>> > >>>>> -Matthias > >>>>> > >>>>> > >>>>> On 6/26/17 1:28 PM, Jorge Esteban Quilcate Otoya wrote: > >>>>>> Hi all, > >>>>>> > >>>>>> I'd like to start the discussion to add reset offset tooling for > >> Stream > >>>>>> applications. > >>>>>> The KIP can be found here: > >>>>>> > >>>>> > >>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-171+-+Extend+Consumer+Group+Reset+Offset+for+Stream+Application > >>>>>> > >>>>>> Thanks, > >>>>>> Jorge. > >>>>>> > >>>>> > >>>>> > >>>> > >>> > >>> > >> > > > >
