Just to clarify, we'll need to allow specifying topic and partition. I don't think we want this on ALL partitions at once.
On Wed, Feb 8, 2017 at 3:35 PM, Gwen Shapira <g...@confluent.io> wrote: > That's what I'd like to see. For example, suppose a Connect task fails > because it can't deserialize an event from a partition. Stop > connector, move offset forward, start connector. Boom! > > > On Wed, Feb 8, 2017 at 3:22 PM, Matthias J. Sax <matth...@confluent.io> wrote: >> I am not sure about --reset-plus and --reset-minus >> >> Would this skip n messages forward/backward for each partitions? >> >> >> -Matthias >> >> On 2/8/17 2:23 PM, Jorge Esteban Quilcate Otoya wrote: >>> Great. I think I got the idea. What about this options: >>> >>> Scenarios: >>> >>> 1. Current status >>> >>> ´kafka-consumer-groups.sh --reset-offset --group cg1´ >>> >>> 2. To Datetime >>> >>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-datetime >>> 2017-01-01T00:00:00.000´ >>> >>> 3. To Period >>> >>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-period P2D´ >>> >>> 4. To Earliest >>> >>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-earliest´ >>> >>> 5. To Latest >>> >>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-latest´ >>> >>> 6. Minus 'n' offsets >>> >>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-minus n´ >>> >>> 7. Plus 'n' offsets >>> >>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-plus n´ >>> >>> 8. To specific offset >>> >>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to x´ >>> >>> Scopes: >>> >>> a. All topics used by Consumer Group >>> >>> Don't specify --topics >>> >>> b. Specific List of Topics >>> >>> Add list of values in --topics t1,t2,tn >>> >>> c. One Topic, all Partitions >>> >>> Add one topic and no partitions values: --topic t1 >>> >>> d. One Topic, List of Partitions >>> >>> Add one topic and partitions values: --topic t1 --partitions 0,1,2 >>> >>> About Reset Plan (JSON file): >>> >>> I think is still valid to have the option to persist reset configuration as >>> a file, but I agree to give the option to run the tool without going down >>> to the JSON file. >>> >>> Execution options: >>> >>> 1. Without execution argument (No args): >>> >>> Print out results (reset plan) >>> >>> 2. With --execute argument: >>> >>> Run reset process >>> >>> 3. With --output argument: >>> >>> Save result in a JSON format. >>> >>> 4. Only with --execute option and --reset-file (path to JSON) >>> >>> Reset based on file >>> >>> 4. Only with --verify option and --reset-file (path to JSON) >>> >>> Verify file values with current offsets >>> >>> I think we can remove --generate-and-execute because is a bit clumsy. >>> >>> With this options we will be able to execute with manual JSON configuration. >>> >>> >>> El mié., 8 feb. 2017 a las 22:43, Ben Stopford (<b...@confluent.io>) >>> escribió: >>> >>>> Yes - using a tool like this to skip a set of consumer groups over a >>>> corrupt/bad message is definitely appealing. >>>> >>>> B >>>> >>>> On Wed, Feb 8, 2017 at 9:37 PM Gwen Shapira <g...@confluent.io> wrote: >>>> >>>>> I like the --reset-to-earliest and --reset-to-latest. In general, >>>>> since the JSON route is the most challenging for users, we want to >>>>> provide a lot of ways to do useful things without going there. >>>>> >>>>> Two things that can help: >>>>> >>>>> 1. A lot of times, users want to skip few messages that cause issues >>>>> and continue. maybe just specifying the topic, partition and delta >>>>> will be better than having to find the offset and write a JSON and >>>>> validate the JSON etc. >>>>> >>>>> 2. Thinking if there are other common use-cases that we can make easy >>>>> rather than just one generic but not very usable method. >>>>> >>>>> Gwen >>>>> >>>>> On Wed, Feb 8, 2017 at 3:25 AM, Jorge Esteban Quilcate Otoya >>>>> <quilcate.jo...@gmail.com> wrote: >>>>>> Thanks for the feedback! >>>>>> >>>>>> @Onur, @Gwen: >>>>>> >>>>>> Agree. Actually at the first draft I considered to have it inside >>>>>> ´kafka-consumer-groups.sh´, but I decide to propose it as a standalone >>>>> tool >>>>>> to describe it clearly and focus it on reset functionality. >>>>>> >>>>>> But now that you mentioned, it does make sense to have it in >>>>>> ´kafka-consumer-groups.sh´. How would be a consistent way to introduce >>>>> it? >>>>>> >>>>>> Maybe something like this: >>>>>> >>>>>> ´kafka-consumer-groups.sh --reset-offset --generate --group cg1 >>>> --topics >>>>> t1 >>>>>> --reset-from 2017-01-01T00:00:00.000 --output plan.json´ >>>>>> >>>>>> ´kafka-consumer-groups.sh --reset-offset --verify --reset-json-file >>>>>> plan.json´ >>>>>> >>>>>> ´kafka-consumer-groups.sh --reset-offset --execute --reset-json-file >>>>>> plan.json´ >>>>>> >>>>>> ´kafka-consumer-groups.sh --reset-offset --generate-and-execute --group >>>>> cg1 >>>>>> --topics t1 --reset-from 2017-01-01T00:00:00.000´ >>>>>> >>>>>> @Gwen: >>>>>> >>>>>>> It looks exactly like the replica assignment tool >>>>>> >>>>>> It was influenced by ;-) I use the generate-verify-execute process here >>>>> to >>>>>> make sure user will be aware of the result of this operation. At the >>>>>> beginning we considered only add a couple of options to Consumer Group >>>>>> Command: >>>>>> >>>>>> --rewind-to-timestamp and --rewind-to-period >>>>>> >>>>>> @Onur: >>>>>> >>>>>>> You can actually get away with overriding while members of the group >>>>> are live >>>>>> with method 2 by using group information from DescribeGroupsRequest. >>>>>> >>>>>> This means that we need to have Consumer Group stopped before executing >>>>> and >>>>>> start a new consumer internally to do this? Therefore, we won't be able >>>>> to >>>>>> consider executing reset when ConsumerGroup is active? (trying to >>>> relate >>>>> it >>>>>> with @Dong 5th question) >>>>>> >>>>>> @Dong: >>>>>> >>>>>>> Should we allow user to use wildcard to reset offset of all groups >>>> for a >>>>>> given topic as well? >>>>>> >>>>>> I haven't thought about this scenario. Could be interesting. Following >>>>> the >>>>>> recommendation to add it into Consumer Group Command, in this case >>>> Group >>>>>> argument will be optional if there are only 1 topic. I think for >>>> multiple >>>>>> topic won't be that useful. >>>>>> >>>>>>> Should we allow user to specify timestamp per topic partition in the >>>>> json >>>>>> file as well? >>>>>> >>>>>> Don't think this could be a valid from the tool, but if Reset Plan is >>>>>> generated, and user want to set the offset for a specific partition to >>>>>> other offset (eventually based on another timestamp), and execute it, >>>> it >>>>>> will be up to her/him. >>>>>> >>>>>>> Should the script take some credential file to make sure that this >>>>>> operation is authenticated given the potential impact of this >>>> operation? >>>>>> >>>>>> Haven't tried to secure brokers yet, but the tool should support >>>>>> authorization if it's enabled in the broker. >>>>>> >>>>>>> Should we provide constant to reset committed offset to >>>> earliest/latest >>>>>> offset of a partition, e.g. -1 indicates earliest offset and -2 >>>> indicates >>>>>> latest offset. >>>>>> >>>>>> I will go for something like ´--reset-to-earliest´ and >>>>> ´--reset-to-latest´ >>>>>> >>>>>>> Should we allow dynamic change of the comitted offset when consumer >>>> are >>>>>> running, such that consumer will seek to the newly committed offset and >>>>>> start consuming from there? >>>>>> >>>>>> Not sure about this. I will recommend to keep it simple and ask user to >>>>>> stop consumers first. But I would considered it if the trade-offs are >>>>>> clear. >>>>>> >>>>>> @Matthias >>>>>> >>>>>> Added :). And thanks a lot for your help to define this KIP! >>>>>> >>>>>> >>>>>> >>>>>> El mié., 8 feb. 2017 a las 7:47, Gwen Shapira (<g...@confluent.io>) >>>>>> escribió: >>>>>> >>>>>>> As long as the CLI is a bit consistent? Like, not just adding 3 >>>>>>> arguments and a JSON parser to the existing tool, right? >>>>>>> >>>>>>> On Tue, Feb 7, 2017 at 10:29 PM, Onur Karaman >>>>>>> <onurkaraman.apa...@gmail.com> wrote: >>>>>>>> I think it makes sense to just add the feature to >>>>>>> kafka-consumer-groups.sh >>>>>>>> >>>>>>>> On Tue, Feb 7, 2017 at 10:24 PM, Gwen Shapira <g...@confluent.io> >>>>> wrote: >>>>>>>> >>>>>>>>> Thanks for the KIP. I'm super happy about adding the capability. >>>>>>>>> >>>>>>>>> I hate the interface, though. It looks exactly like the replica >>>>>>>>> assignment tool. A tool everyone loves so much that there are >>>>> multiple >>>>>>>>> projects, open and closed, that try to fix it. >>>>>>>>> >>>>>>>>> Can we swap it with something that looks a bit more like the >>>> consumer >>>>>>>>> group tool? or the kafka streams reset tool? Consistency is helpful >>>>> in >>>>>>>>> such cases. I spent some time learning existing tools and learning >>>>> yet >>>>>>>>> another one is a deterrent. >>>>>>>>> >>>>>>>>> Gwen >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Feb 7, 2017 at 6:43 PM, Jorge Esteban Quilcate Otoya >>>>>>>>> <quilcate.jo...@gmail.com> wrote: >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I would like to propose a KIP to Add a tool to Reset Consumer >>>> Group >>>>>>>>> Offsets. >>>>>>>>>> >>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>>>>>>>> 122%3A+Add+a+tool+to+Reset+Consumer+Group+Offsets >>>>>>>>>> >>>>>>>>>> Please, take a look at the proposal and share your feedback. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Jorge. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Gwen Shapira >>>>>>>>> Product Manager | Confluent >>>>>>>>> 650.450.2760 <(650)%20450-2760> <(650)%20450-2760> >>>> <(650)%20450-2760> | @gwenshap >>>>>>>>> Follow us: Twitter | blog >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Gwen Shapira >>>>>>> Product Manager | Confluent >>>>>>> 650.450.2760 <(650)%20450-2760> <(650)%20450-2760> >>>> <(650)%20450-2760> | @gwenshap >>>>>>> Follow us: Twitter | blog >>>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Gwen Shapira >>>>> Product Manager | Confluent >>>>> 650.450.2760 <(650)%20450-2760> <(650)%20450-2760> | @gwenshap >>>>> Follow us: Twitter | blog >>>>> >>>> >>> >> > > > > -- > Gwen Shapira > Product Manager | Confluent > 650.450.2760 | @gwenshap > Follow us: Twitter | blog -- Gwen Shapira Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter | blog
