I am not sure about --reset-plus and --reset-minus Would this skip n messages forward/backward for each partitions?
-Matthias On 2/8/17 2:23 PM, Jorge Esteban Quilcate Otoya wrote: > Great. I think I got the idea. What about this options: > > Scenarios: > > 1. Current status > > ´kafka-consumer-groups.sh --reset-offset --group cg1´ > > 2. To Datetime > > ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-datetime > 2017-01-01T00:00:00.000´ > > 3. To Period > > ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-period P2D´ > > 4. To Earliest > > ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-earliest´ > > 5. To Latest > > ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-latest´ > > 6. Minus 'n' offsets > > ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-minus n´ > > 7. Plus 'n' offsets > > ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-plus n´ > > 8. To specific offset > > ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to x´ > > Scopes: > > a. All topics used by Consumer Group > > Don't specify --topics > > b. Specific List of Topics > > Add list of values in --topics t1,t2,tn > > c. One Topic, all Partitions > > Add one topic and no partitions values: --topic t1 > > d. One Topic, List of Partitions > > Add one topic and partitions values: --topic t1 --partitions 0,1,2 > > About Reset Plan (JSON file): > > I think is still valid to have the option to persist reset configuration as > a file, but I agree to give the option to run the tool without going down > to the JSON file. > > Execution options: > > 1. Without execution argument (No args): > > Print out results (reset plan) > > 2. With --execute argument: > > Run reset process > > 3. With --output argument: > > Save result in a JSON format. > > 4. Only with --execute option and --reset-file (path to JSON) > > Reset based on file > > 4. Only with --verify option and --reset-file (path to JSON) > > Verify file values with current offsets > > I think we can remove --generate-and-execute because is a bit clumsy. > > With this options we will be able to execute with manual JSON configuration. > > > El mié., 8 feb. 2017 a las 22:43, Ben Stopford (<b...@confluent.io>) > escribió: > >> Yes - using a tool like this to skip a set of consumer groups over a >> corrupt/bad message is definitely appealing. >> >> B >> >> On Wed, Feb 8, 2017 at 9:37 PM Gwen Shapira <g...@confluent.io> wrote: >> >>> I like the --reset-to-earliest and --reset-to-latest. In general, >>> since the JSON route is the most challenging for users, we want to >>> provide a lot of ways to do useful things without going there. >>> >>> Two things that can help: >>> >>> 1. A lot of times, users want to skip few messages that cause issues >>> and continue. maybe just specifying the topic, partition and delta >>> will be better than having to find the offset and write a JSON and >>> validate the JSON etc. >>> >>> 2. Thinking if there are other common use-cases that we can make easy >>> rather than just one generic but not very usable method. >>> >>> Gwen >>> >>> On Wed, Feb 8, 2017 at 3:25 AM, Jorge Esteban Quilcate Otoya >>> <quilcate.jo...@gmail.com> wrote: >>>> Thanks for the feedback! >>>> >>>> @Onur, @Gwen: >>>> >>>> Agree. Actually at the first draft I considered to have it inside >>>> ´kafka-consumer-groups.sh´, but I decide to propose it as a standalone >>> tool >>>> to describe it clearly and focus it on reset functionality. >>>> >>>> But now that you mentioned, it does make sense to have it in >>>> ´kafka-consumer-groups.sh´. How would be a consistent way to introduce >>> it? >>>> >>>> Maybe something like this: >>>> >>>> ´kafka-consumer-groups.sh --reset-offset --generate --group cg1 >> --topics >>> t1 >>>> --reset-from 2017-01-01T00:00:00.000 --output plan.json´ >>>> >>>> ´kafka-consumer-groups.sh --reset-offset --verify --reset-json-file >>>> plan.json´ >>>> >>>> ´kafka-consumer-groups.sh --reset-offset --execute --reset-json-file >>>> plan.json´ >>>> >>>> ´kafka-consumer-groups.sh --reset-offset --generate-and-execute --group >>> cg1 >>>> --topics t1 --reset-from 2017-01-01T00:00:00.000´ >>>> >>>> @Gwen: >>>> >>>>> It looks exactly like the replica assignment tool >>>> >>>> It was influenced by ;-) I use the generate-verify-execute process here >>> to >>>> make sure user will be aware of the result of this operation. At the >>>> beginning we considered only add a couple of options to Consumer Group >>>> Command: >>>> >>>> --rewind-to-timestamp and --rewind-to-period >>>> >>>> @Onur: >>>> >>>>> You can actually get away with overriding while members of the group >>> are live >>>> with method 2 by using group information from DescribeGroupsRequest. >>>> >>>> This means that we need to have Consumer Group stopped before executing >>> and >>>> start a new consumer internally to do this? Therefore, we won't be able >>> to >>>> consider executing reset when ConsumerGroup is active? (trying to >> relate >>> it >>>> with @Dong 5th question) >>>> >>>> @Dong: >>>> >>>>> Should we allow user to use wildcard to reset offset of all groups >> for a >>>> given topic as well? >>>> >>>> I haven't thought about this scenario. Could be interesting. Following >>> the >>>> recommendation to add it into Consumer Group Command, in this case >> Group >>>> argument will be optional if there are only 1 topic. I think for >> multiple >>>> topic won't be that useful. >>>> >>>>> Should we allow user to specify timestamp per topic partition in the >>> json >>>> file as well? >>>> >>>> Don't think this could be a valid from the tool, but if Reset Plan is >>>> generated, and user want to set the offset for a specific partition to >>>> other offset (eventually based on another timestamp), and execute it, >> it >>>> will be up to her/him. >>>> >>>>> Should the script take some credential file to make sure that this >>>> operation is authenticated given the potential impact of this >> operation? >>>> >>>> Haven't tried to secure brokers yet, but the tool should support >>>> authorization if it's enabled in the broker. >>>> >>>>> Should we provide constant to reset committed offset to >> earliest/latest >>>> offset of a partition, e.g. -1 indicates earliest offset and -2 >> indicates >>>> latest offset. >>>> >>>> I will go for something like ´--reset-to-earliest´ and >>> ´--reset-to-latest´ >>>> >>>>> Should we allow dynamic change of the comitted offset when consumer >> are >>>> running, such that consumer will seek to the newly committed offset and >>>> start consuming from there? >>>> >>>> Not sure about this. I will recommend to keep it simple and ask user to >>>> stop consumers first. But I would considered it if the trade-offs are >>>> clear. >>>> >>>> @Matthias >>>> >>>> Added :). And thanks a lot for your help to define this KIP! >>>> >>>> >>>> >>>> El mié., 8 feb. 2017 a las 7:47, Gwen Shapira (<g...@confluent.io>) >>>> escribió: >>>> >>>>> As long as the CLI is a bit consistent? Like, not just adding 3 >>>>> arguments and a JSON parser to the existing tool, right? >>>>> >>>>> On Tue, Feb 7, 2017 at 10:29 PM, Onur Karaman >>>>> <onurkaraman.apa...@gmail.com> wrote: >>>>>> I think it makes sense to just add the feature to >>>>> kafka-consumer-groups.sh >>>>>> >>>>>> On Tue, Feb 7, 2017 at 10:24 PM, Gwen Shapira <g...@confluent.io> >>> wrote: >>>>>> >>>>>>> Thanks for the KIP. I'm super happy about adding the capability. >>>>>>> >>>>>>> I hate the interface, though. It looks exactly like the replica >>>>>>> assignment tool. A tool everyone loves so much that there are >>> multiple >>>>>>> projects, open and closed, that try to fix it. >>>>>>> >>>>>>> Can we swap it with something that looks a bit more like the >> consumer >>>>>>> group tool? or the kafka streams reset tool? Consistency is helpful >>> in >>>>>>> such cases. I spent some time learning existing tools and learning >>> yet >>>>>>> another one is a deterrent. >>>>>>> >>>>>>> Gwen >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Feb 7, 2017 at 6:43 PM, Jorge Esteban Quilcate Otoya >>>>>>> <quilcate.jo...@gmail.com> wrote: >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I would like to propose a KIP to Add a tool to Reset Consumer >> Group >>>>>>> Offsets. >>>>>>>> >>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>>>>>> 122%3A+Add+a+tool+to+Reset+Consumer+Group+Offsets >>>>>>>> >>>>>>>> Please, take a look at the proposal and share your feedback. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Jorge. >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Gwen Shapira >>>>>>> Product Manager | Confluent >>>>>>> 650.450.2760 <(650)%20450-2760> <(650)%20450-2760> >> <(650)%20450-2760> | @gwenshap >>>>>>> Follow us: Twitter | blog >>>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Gwen Shapira >>>>> Product Manager | Confluent >>>>> 650.450.2760 <(650)%20450-2760> <(650)%20450-2760> >> <(650)%20450-2760> | @gwenshap >>>>> Follow us: Twitter | blog >>>>> >>> >>> >>> >>> -- >>> Gwen Shapira >>> Product Manager | Confluent >>> 650.450.2760 <(650)%20450-2760> <(650)%20450-2760> | @gwenshap >>> Follow us: Twitter | blog >>> >> >
signature.asc
Description: OpenPGP digital signature
