Thanks a lot for the update! I like the KIP!
One more question about `--dry-run` vs `--execute`: While I agree that we should use the same flag for both tools, I am not sure which one is the better one... My personal take is, that I like `--dry-run` better. Not sure what others think. One more comment: with the removal of ZK, we can also tackle this JIRA: https://issues.apache.org/jira/browse/KAFKA-4327 If we do so, I think we should mention it in the KIP. I am also not sure about backward compatibility issue for this case. Actually, I don't expect people to call `StreamsResetter` from Java code, but you can never know. So if we break this, we need to make sure to cover it in the KIP and later on in the release notes. -Matthias On 7/14/17 7:15 AM, Jorge Esteban Quilcate Otoya wrote: > Hi, > > KIP is updated. > Changes: > 1. Approach discussed to keep both tools (streams application resetter and > consumer group reset offset). > 2. Options has been aligned between both tools. > 3. Zookeeper option from streams-application-resetted will be removed, and > replaced internally for Kafka AdminClient. > > Looking forward to your feedback. > > El jue., 29 jun. 2017 a las 15:04, Matthias J. Sax (<matth...@confluent.io>) > escribió: > >> Damian, >> >>> resets everything and clears up >>>> the state stores. >> >> That's not correct. The reset tool does not touch local store. For this, >> we have `KafkaStreams#clenup` -- otherwise, you would need to run the >> tool in every machine you run an application instance. >> >> With regard to state, the tool only deletes the underlying changelog >> topics. >> >> Just to clarify. The idea of allowing to rest with different offset is >> to clear all state but just use a different start offset (instead of >> zero). This is for use case where your topic has a larger retention time >> than the amount of data you want to reprocess. But we always need to >> start with an empty state. (Resetting with consistent state is something >> we might do at some point, but it's much hard and not part of this KIP) >> >>> @matthias, could we remove the ZK dependency from the streams reset tool >>> now? >> >> I think so. The new AdminClient provide the feature we need AFAIK. I >> guess we can picky back this into the KIP (we would need a KIP anyway >> because we deprecate `--zookeeper` what is an public API change). >> >> >> I just want to point out, that even without ZK dependency, I prefer to >> wrap the consumer offset tool and keep two tools. >> >> >> -Matthias >> >> >> On 6/29/17 9:14 AM, Damian Guy wrote: >>> Hi, >>> >>> Thanks for the KIP. What is not clear is how is this going to handle >> state >>> stores? Right now the streams reset tool, resets everything and clears up >>> the state stores. What are we going to do if we reset to a particular >>> offset? If we clear the state then we've lost any previously aggregated >>> values (which may or may not be what is expected). If we don't clear >> them, >>> then we will end up with incorrect aggregates. >>> >>> @matthias, could we remove the ZK dependency from the streams reset tool >>> now? >>> >>> Thanks, >>> Damian >>> >>> On Thu, 29 Jun 2017 at 15:22 Jorge Esteban Quilcate Otoya < >>> quilcate.jo...@gmail.com> wrote: >>> >>>> You're right, I was not considering Zookeeper dependency. >>>> >>>> I'm starting to like the idea to wrap `reset-offset` from >>>> `streams-application-reset`. >>>> >>>> I will wait a bit for more feedback and work on a draft with this >> changes. >>>> >>>> >>>> El mié., 28 jun. 2017 a las 0:20, Matthias J. Sax (< >> matth...@confluent.io >>>>> ) >>>> escribió: >>>> >>>>> I agree, that we should not duplicate functionality. >>>>> >>>>> However, I am worried, that a non-streams users using the offset reset >>>>> tool might delete topics unintentionally (even if the changes are >> pretty >>>>> small). Also, currently the stream reset tool required Zookeeper while >>>>> the offset reset tool does not -- I don't think we should add this >>>>> dependency to the offset reset tool. >>>>> >>>>> Thus, it think it might be better to keep both tools, but internally >>>>> rewrite the streams reset entry class, to reuse as much as possible >> from >>>>> the offset reset tool. Ie. the streams tool would be a wrapper around >>>>> the offset tool and add some functionality it needs that is Streams >>>>> specific. >>>>> >>>>> I also think, that keeping separate tools for consumers and streams is >> a >>>>> good thing. We might want to add new features that don't apply to plain >>>>> consumers -- note, a Streams applications is more than just a client. >>>>> >>>>> WDYT? >>>>> >>>>> Would be good to get some feedback from others, too. >>>>> >>>>> >>>>> -Matthias >>>>> >>>>> >>>>> On 6/27/17 9:05 AM, Jorge Esteban Quilcate Otoya wrote: >>>>>> Thanks for the feedback Matthias! >>>>>> >>>>>> The main idea is to use only 1 tool to reset offsets and don't >>>> replicate >>>>>> functionality between tools. >>>>>> Reset Offset (KIP-122) tool not only reset but support execute the >>>> reset >>>>>> but also export, import from files, etc. >>>>>> If we extend the current tool (kafka-streams-application-reset.sh) we >>>>> will >>>>>> have to duplicate all this functionality also. >>>>>> Maybe another option is to move the current implementation into >>>>>> `kafka-consumer-group` and add a new command `--reset-offset-streams` >>>>> with >>>>>> the current implementation functionality and add `--reset-offset` >>>> options >>>>>> for input topics. Does this make sense? >>>>>> >>>>>> >>>>>> El lun., 26 jun. 2017 a las 23:32, Matthias J. Sax (< >>>>> matth...@confluent.io>) >>>>>> escribió: >>>>>> >>>>>>> Jorge, >>>>>>> >>>>>>> thanks a lot for this KIP. Allowing the reset streams applications >>>> with >>>>>>> arbitrary start offset is something we got multiple requests already. >>>>>>> >>>>>>> Couple of clarification question: >>>>>>> >>>>>>> - why do you want to deprecate the current tool instead of extending >>>>>>> the current tool with the stuff the offset reset tool can do (ie, use >>>>>>> the offset reset tool internally) >>>>>>> >>>>>>> - you suggest to extend the offset reset tool to replace the stream >>>>>>> reset tool: how would the reset tool know if it is resetting a >> streams >>>>>>> applications or a regular consumer group? >>>>>>> >>>>>>> >>>>>>> >>>>>>> -Matthias >>>>>>> >>>>>>> >>>>>>> On 6/26/17 1:28 PM, Jorge Esteban Quilcate Otoya wrote: >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I'd like to start the discussion to add reset offset tooling for >>>> Stream >>>>>>>> applications. >>>>>>>> The KIP can be found here: >>>>>>>> >>>>>>> >>>>> >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-171+-+Extend+Consumer+Group+Reset+Offset+for+Stream+Application >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Jorge. >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>> >>> >> >> >
signature.asc
Description: OpenPGP digital signature
