I am not sure about --reset-plus and --reset-minus

Would this skip n messages forward/backward for each partitions?


-Matthias

On 2/8/17 2:23 PM, Jorge Esteban Quilcate Otoya wrote:
> Great. I think I got the idea. What about this options:
> 
> Scenarios:
> 
> 1. Current status
> 
> ´kafka-consumer-groups.sh --reset-offset --group cg1´
> 
> 2. To Datetime
> 
> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-datetime
> 2017-01-01T00:00:00.000´
> 
> 3. To Period
> 
> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-period P2D´
> 
> 4. To Earliest
> 
> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-earliest´
> 
> 5. To Latest
> 
> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-latest´
> 
> 6. Minus 'n' offsets
> 
> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-minus n´
> 
> 7. Plus 'n' offsets
> 
> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-plus n´
> 
> 8. To specific offset
> 
> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to x´
> 
> Scopes:
> 
> a. All topics used by Consumer Group
> 
> Don't specify --topics
> 
> b. Specific List of Topics
> 
> Add list of values in --topics t1,t2,tn
> 
> c. One Topic, all Partitions
> 
> Add one topic and no partitions values: --topic t1
> 
> d. One Topic, List of Partitions
> 
> Add one topic and partitions values: --topic t1 --partitions 0,1,2
> 
> About Reset Plan (JSON file):
> 
> I think is still valid to have the option to persist reset configuration as
> a file, but I agree to give the option to run the tool without going down
> to the JSON file.
> 
> Execution options:
> 
> 1. Without execution argument (No args):
> 
> Print out results (reset plan)
> 
> 2. With --execute argument:
> 
> Run reset process
> 
> 3. With --output argument:
> 
> Save result in a JSON format.
> 
> 4. Only with --execute option and --reset-file (path to JSON)
> 
> Reset based on file
> 
> 4. Only with --verify option and --reset-file (path to JSON)
> 
> Verify file values with current offsets
> 
> I think we can remove --generate-and-execute because is a bit clumsy.
> 
> With this options we will be able to execute with manual JSON configuration.
> 
> 
> El mié., 8 feb. 2017 a las 22:43, Ben Stopford (<b...@confluent.io>)
> escribió:
> 
>> Yes - using a tool like this to skip a set of consumer groups over a
>> corrupt/bad message is definitely appealing.
>>
>> B
>>
>> On Wed, Feb 8, 2017 at 9:37 PM Gwen Shapira <g...@confluent.io> wrote:
>>
>>> I like the --reset-to-earliest and --reset-to-latest. In general,
>>> since the JSON route is the most challenging for users, we want to
>>> provide a lot of ways to do useful things without going there.
>>>
>>> Two things that can help:
>>>
>>> 1. A lot of times, users want to skip few messages that cause issues
>>> and continue. maybe just specifying the topic, partition and delta
>>> will be better than having to find the offset and write a JSON and
>>> validate the JSON etc.
>>>
>>> 2. Thinking if there are other common use-cases that we can make easy
>>> rather than just one generic but not very usable method.
>>>
>>> Gwen
>>>
>>> On Wed, Feb 8, 2017 at 3:25 AM, Jorge Esteban Quilcate Otoya
>>> <quilcate.jo...@gmail.com> wrote:
>>>> Thanks for the feedback!
>>>>
>>>> @Onur, @Gwen:
>>>>
>>>> Agree. Actually at the first draft I considered to have it inside
>>>> ´kafka-consumer-groups.sh´, but I decide to propose it as a standalone
>>> tool
>>>> to describe it clearly and focus it on reset functionality.
>>>>
>>>> But now that you mentioned, it does make sense to have it in
>>>> ´kafka-consumer-groups.sh´. How would be a consistent way to introduce
>>> it?
>>>>
>>>> Maybe something like this:
>>>>
>>>> ´kafka-consumer-groups.sh --reset-offset --generate --group cg1
>> --topics
>>> t1
>>>> --reset-from 2017-01-01T00:00:00.000 --output plan.json´
>>>>
>>>> ´kafka-consumer-groups.sh --reset-offset --verify --reset-json-file
>>>> plan.json´
>>>>
>>>> ´kafka-consumer-groups.sh --reset-offset --execute --reset-json-file
>>>> plan.json´
>>>>
>>>> ´kafka-consumer-groups.sh --reset-offset --generate-and-execute --group
>>> cg1
>>>> --topics t1 --reset-from 2017-01-01T00:00:00.000´
>>>>
>>>> @Gwen:
>>>>
>>>>> It looks exactly like the replica assignment tool
>>>>
>>>> It was influenced by ;-) I use the generate-verify-execute process here
>>> to
>>>> make sure user will be aware of the result of this operation. At the
>>>> beginning we considered only add a couple of options to Consumer Group
>>>> Command:
>>>>
>>>> --rewind-to-timestamp and --rewind-to-period
>>>>
>>>> @Onur:
>>>>
>>>>> You can actually get away with overriding while members of the group
>>> are live
>>>> with method 2 by using group information from DescribeGroupsRequest.
>>>>
>>>> This means that we need to have Consumer Group stopped before executing
>>> and
>>>> start a new consumer internally to do this? Therefore, we won't be able
>>> to
>>>> consider executing reset when ConsumerGroup is active? (trying to
>> relate
>>> it
>>>> with @Dong 5th question)
>>>>
>>>> @Dong:
>>>>
>>>>> Should we allow user to use wildcard to reset offset of all groups
>> for a
>>>> given topic as well?
>>>>
>>>> I haven't thought about this scenario. Could be interesting. Following
>>> the
>>>> recommendation to add it into Consumer Group Command, in this case
>> Group
>>>> argument will be optional if there are only 1 topic. I think for
>> multiple
>>>> topic won't be that useful.
>>>>
>>>>> Should we allow user to specify timestamp per topic partition in the
>>> json
>>>> file as well?
>>>>
>>>> Don't think this could be a valid from the tool, but if Reset Plan is
>>>> generated, and user want to set the offset for a specific partition to
>>>> other offset (eventually based on another timestamp), and execute it,
>> it
>>>> will be up to her/him.
>>>>
>>>>> Should the script take some credential file to make sure that this
>>>> operation is authenticated given the potential impact of this
>> operation?
>>>>
>>>> Haven't tried to secure brokers yet, but the tool should support
>>>> authorization if it's enabled in the broker.
>>>>
>>>>> Should we provide constant to reset committed offset to
>> earliest/latest
>>>> offset of a partition, e.g. -1 indicates earliest offset and -2
>> indicates
>>>> latest offset.
>>>>
>>>> I will go for something like ´--reset-to-earliest´ and
>>> ´--reset-to-latest´
>>>>
>>>>> Should we allow dynamic change of the comitted offset when consumer
>> are
>>>> running, such that consumer will seek to the newly committed offset and
>>>> start consuming from there?
>>>>
>>>> Not sure about this. I will recommend to keep it simple and ask user to
>>>> stop consumers first. But I would considered it if the trade-offs are
>>>> clear.
>>>>
>>>> @Matthias
>>>>
>>>> Added :). And thanks a lot for your help to define this KIP!
>>>>
>>>>
>>>>
>>>> El mié., 8 feb. 2017 a las 7:47, Gwen Shapira (<g...@confluent.io>)
>>>> escribió:
>>>>
>>>>> As long as the CLI is a bit consistent? Like, not just adding 3
>>>>> arguments and a JSON parser to the existing tool, right?
>>>>>
>>>>> On Tue, Feb 7, 2017 at 10:29 PM, Onur Karaman
>>>>> <onurkaraman.apa...@gmail.com> wrote:
>>>>>> I think it makes sense to just add the feature to
>>>>> kafka-consumer-groups.sh
>>>>>>
>>>>>> On Tue, Feb 7, 2017 at 10:24 PM, Gwen Shapira <g...@confluent.io>
>>> wrote:
>>>>>>
>>>>>>> Thanks for the KIP. I'm super happy about adding the capability.
>>>>>>>
>>>>>>> I hate the interface, though. It looks exactly like the replica
>>>>>>> assignment tool. A tool everyone loves so much that there are
>>> multiple
>>>>>>> projects, open and closed, that try to fix it.
>>>>>>>
>>>>>>> Can we swap it with something that looks a bit more like the
>> consumer
>>>>>>> group tool? or the kafka streams reset tool? Consistency is helpful
>>> in
>>>>>>> such cases. I spent some time learning existing tools and learning
>>> yet
>>>>>>> another one is a deterrent.
>>>>>>>
>>>>>>> Gwen
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Feb 7, 2017 at 6:43 PM, Jorge Esteban Quilcate Otoya
>>>>>>> <quilcate.jo...@gmail.com> wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I would like to propose a KIP to Add a tool to Reset Consumer
>> Group
>>>>>>> Offsets.
>>>>>>>>
>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>> 122%3A+Add+a+tool+to+Reset+Consumer+Group+Offsets
>>>>>>>>
>>>>>>>> Please, take a look at the proposal and share your feedback.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Jorge.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Gwen Shapira
>>>>>>> Product Manager | Confluent
>>>>>>> 650.450.2760 <(650)%20450-2760> <(650)%20450-2760>
>> <(650)%20450-2760> | @gwenshap
>>>>>>> Follow us: Twitter | blog
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Gwen Shapira
>>>>> Product Manager | Confluent
>>>>> 650.450.2760 <(650)%20450-2760> <(650)%20450-2760>
>> <(650)%20450-2760> | @gwenshap
>>>>> Follow us: Twitter | blog
>>>>>
>>>
>>>
>>>
>>> --
>>> Gwen Shapira
>>> Product Manager | Confluent
>>> 650.450.2760 <(650)%20450-2760> <(650)%20450-2760> | @gwenshap
>>> Follow us: Twitter | blog
>>>
>>
> 

Attachment: signature.asc
Description: OpenPGP digital signature

Reply via email to