Just to clarify, we'll need to allow specifying topic and partition. I
don't think we want this on ALL partitions at once.

On Wed, Feb 8, 2017 at 3:35 PM, Gwen Shapira <g...@confluent.io> wrote:
> That's what I'd like to see. For example, suppose a Connect task fails
> because it can't deserialize an event from a partition. Stop
> connector, move offset forward, start connector. Boom!
>
>
> On Wed, Feb 8, 2017 at 3:22 PM, Matthias J. Sax <matth...@confluent.io> wrote:
>> I am not sure about --reset-plus and --reset-minus
>>
>> Would this skip n messages forward/backward for each partitions?
>>
>>
>> -Matthias
>>
>> On 2/8/17 2:23 PM, Jorge Esteban Quilcate Otoya wrote:
>>> Great. I think I got the idea. What about this options:
>>>
>>> Scenarios:
>>>
>>> 1. Current status
>>>
>>> ´kafka-consumer-groups.sh --reset-offset --group cg1´
>>>
>>> 2. To Datetime
>>>
>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-datetime
>>> 2017-01-01T00:00:00.000´
>>>
>>> 3. To Period
>>>
>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-period P2D´
>>>
>>> 4. To Earliest
>>>
>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-earliest´
>>>
>>> 5. To Latest
>>>
>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-latest´
>>>
>>> 6. Minus 'n' offsets
>>>
>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-minus n´
>>>
>>> 7. Plus 'n' offsets
>>>
>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-plus n´
>>>
>>> 8. To specific offset
>>>
>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to x´
>>>
>>> Scopes:
>>>
>>> a. All topics used by Consumer Group
>>>
>>> Don't specify --topics
>>>
>>> b. Specific List of Topics
>>>
>>> Add list of values in --topics t1,t2,tn
>>>
>>> c. One Topic, all Partitions
>>>
>>> Add one topic and no partitions values: --topic t1
>>>
>>> d. One Topic, List of Partitions
>>>
>>> Add one topic and partitions values: --topic t1 --partitions 0,1,2
>>>
>>> About Reset Plan (JSON file):
>>>
>>> I think is still valid to have the option to persist reset configuration as
>>> a file, but I agree to give the option to run the tool without going down
>>> to the JSON file.
>>>
>>> Execution options:
>>>
>>> 1. Without execution argument (No args):
>>>
>>> Print out results (reset plan)
>>>
>>> 2. With --execute argument:
>>>
>>> Run reset process
>>>
>>> 3. With --output argument:
>>>
>>> Save result in a JSON format.
>>>
>>> 4. Only with --execute option and --reset-file (path to JSON)
>>>
>>> Reset based on file
>>>
>>> 4. Only with --verify option and --reset-file (path to JSON)
>>>
>>> Verify file values with current offsets
>>>
>>> I think we can remove --generate-and-execute because is a bit clumsy.
>>>
>>> With this options we will be able to execute with manual JSON configuration.
>>>
>>>
>>> El mié., 8 feb. 2017 a las 22:43, Ben Stopford (<b...@confluent.io>)
>>> escribió:
>>>
>>>> Yes - using a tool like this to skip a set of consumer groups over a
>>>> corrupt/bad message is definitely appealing.
>>>>
>>>> B
>>>>
>>>> On Wed, Feb 8, 2017 at 9:37 PM Gwen Shapira <g...@confluent.io> wrote:
>>>>
>>>>> I like the --reset-to-earliest and --reset-to-latest. In general,
>>>>> since the JSON route is the most challenging for users, we want to
>>>>> provide a lot of ways to do useful things without going there.
>>>>>
>>>>> Two things that can help:
>>>>>
>>>>> 1. A lot of times, users want to skip few messages that cause issues
>>>>> and continue. maybe just specifying the topic, partition and delta
>>>>> will be better than having to find the offset and write a JSON and
>>>>> validate the JSON etc.
>>>>>
>>>>> 2. Thinking if there are other common use-cases that we can make easy
>>>>> rather than just one generic but not very usable method.
>>>>>
>>>>> Gwen
>>>>>
>>>>> On Wed, Feb 8, 2017 at 3:25 AM, Jorge Esteban Quilcate Otoya
>>>>> <quilcate.jo...@gmail.com> wrote:
>>>>>> Thanks for the feedback!
>>>>>>
>>>>>> @Onur, @Gwen:
>>>>>>
>>>>>> Agree. Actually at the first draft I considered to have it inside
>>>>>> ´kafka-consumer-groups.sh´, but I decide to propose it as a standalone
>>>>> tool
>>>>>> to describe it clearly and focus it on reset functionality.
>>>>>>
>>>>>> But now that you mentioned, it does make sense to have it in
>>>>>> ´kafka-consumer-groups.sh´. How would be a consistent way to introduce
>>>>> it?
>>>>>>
>>>>>> Maybe something like this:
>>>>>>
>>>>>> ´kafka-consumer-groups.sh --reset-offset --generate --group cg1
>>>> --topics
>>>>> t1
>>>>>> --reset-from 2017-01-01T00:00:00.000 --output plan.json´
>>>>>>
>>>>>> ´kafka-consumer-groups.sh --reset-offset --verify --reset-json-file
>>>>>> plan.json´
>>>>>>
>>>>>> ´kafka-consumer-groups.sh --reset-offset --execute --reset-json-file
>>>>>> plan.json´
>>>>>>
>>>>>> ´kafka-consumer-groups.sh --reset-offset --generate-and-execute --group
>>>>> cg1
>>>>>> --topics t1 --reset-from 2017-01-01T00:00:00.000´
>>>>>>
>>>>>> @Gwen:
>>>>>>
>>>>>>> It looks exactly like the replica assignment tool
>>>>>>
>>>>>> It was influenced by ;-) I use the generate-verify-execute process here
>>>>> to
>>>>>> make sure user will be aware of the result of this operation. At the
>>>>>> beginning we considered only add a couple of options to Consumer Group
>>>>>> Command:
>>>>>>
>>>>>> --rewind-to-timestamp and --rewind-to-period
>>>>>>
>>>>>> @Onur:
>>>>>>
>>>>>>> You can actually get away with overriding while members of the group
>>>>> are live
>>>>>> with method 2 by using group information from DescribeGroupsRequest.
>>>>>>
>>>>>> This means that we need to have Consumer Group stopped before executing
>>>>> and
>>>>>> start a new consumer internally to do this? Therefore, we won't be able
>>>>> to
>>>>>> consider executing reset when ConsumerGroup is active? (trying to
>>>> relate
>>>>> it
>>>>>> with @Dong 5th question)
>>>>>>
>>>>>> @Dong:
>>>>>>
>>>>>>> Should we allow user to use wildcard to reset offset of all groups
>>>> for a
>>>>>> given topic as well?
>>>>>>
>>>>>> I haven't thought about this scenario. Could be interesting. Following
>>>>> the
>>>>>> recommendation to add it into Consumer Group Command, in this case
>>>> Group
>>>>>> argument will be optional if there are only 1 topic. I think for
>>>> multiple
>>>>>> topic won't be that useful.
>>>>>>
>>>>>>> Should we allow user to specify timestamp per topic partition in the
>>>>> json
>>>>>> file as well?
>>>>>>
>>>>>> Don't think this could be a valid from the tool, but if Reset Plan is
>>>>>> generated, and user want to set the offset for a specific partition to
>>>>>> other offset (eventually based on another timestamp), and execute it,
>>>> it
>>>>>> will be up to her/him.
>>>>>>
>>>>>>> Should the script take some credential file to make sure that this
>>>>>> operation is authenticated given the potential impact of this
>>>> operation?
>>>>>>
>>>>>> Haven't tried to secure brokers yet, but the tool should support
>>>>>> authorization if it's enabled in the broker.
>>>>>>
>>>>>>> Should we provide constant to reset committed offset to
>>>> earliest/latest
>>>>>> offset of a partition, e.g. -1 indicates earliest offset and -2
>>>> indicates
>>>>>> latest offset.
>>>>>>
>>>>>> I will go for something like ´--reset-to-earliest´ and
>>>>> ´--reset-to-latest´
>>>>>>
>>>>>>> Should we allow dynamic change of the comitted offset when consumer
>>>> are
>>>>>> running, such that consumer will seek to the newly committed offset and
>>>>>> start consuming from there?
>>>>>>
>>>>>> Not sure about this. I will recommend to keep it simple and ask user to
>>>>>> stop consumers first. But I would considered it if the trade-offs are
>>>>>> clear.
>>>>>>
>>>>>> @Matthias
>>>>>>
>>>>>> Added :). And thanks a lot for your help to define this KIP!
>>>>>>
>>>>>>
>>>>>>
>>>>>> El mié., 8 feb. 2017 a las 7:47, Gwen Shapira (<g...@confluent.io>)
>>>>>> escribió:
>>>>>>
>>>>>>> As long as the CLI is a bit consistent? Like, not just adding 3
>>>>>>> arguments and a JSON parser to the existing tool, right?
>>>>>>>
>>>>>>> On Tue, Feb 7, 2017 at 10:29 PM, Onur Karaman
>>>>>>> <onurkaraman.apa...@gmail.com> wrote:
>>>>>>>> I think it makes sense to just add the feature to
>>>>>>> kafka-consumer-groups.sh
>>>>>>>>
>>>>>>>> On Tue, Feb 7, 2017 at 10:24 PM, Gwen Shapira <g...@confluent.io>
>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for the KIP. I'm super happy about adding the capability.
>>>>>>>>>
>>>>>>>>> I hate the interface, though. It looks exactly like the replica
>>>>>>>>> assignment tool. A tool everyone loves so much that there are
>>>>> multiple
>>>>>>>>> projects, open and closed, that try to fix it.
>>>>>>>>>
>>>>>>>>> Can we swap it with something that looks a bit more like the
>>>> consumer
>>>>>>>>> group tool? or the kafka streams reset tool? Consistency is helpful
>>>>> in
>>>>>>>>> such cases. I spent some time learning existing tools and learning
>>>>> yet
>>>>>>>>> another one is a deterrent.
>>>>>>>>>
>>>>>>>>> Gwen
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Feb 7, 2017 at 6:43 PM, Jorge Esteban Quilcate Otoya
>>>>>>>>> <quilcate.jo...@gmail.com> wrote:
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> I would like to propose a KIP to Add a tool to Reset Consumer
>>>> Group
>>>>>>>>> Offsets.
>>>>>>>>>>
>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>>> 122%3A+Add+a+tool+to+Reset+Consumer+Group+Offsets
>>>>>>>>>>
>>>>>>>>>> Please, take a look at the proposal and share your feedback.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Jorge.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Gwen Shapira
>>>>>>>>> Product Manager | Confluent
>>>>>>>>> 650.450.2760 <(650)%20450-2760> <(650)%20450-2760>
>>>> <(650)%20450-2760> | @gwenshap
>>>>>>>>> Follow us: Twitter | blog
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Gwen Shapira
>>>>>>> Product Manager | Confluent
>>>>>>> 650.450.2760 <(650)%20450-2760> <(650)%20450-2760>
>>>> <(650)%20450-2760> | @gwenshap
>>>>>>> Follow us: Twitter | blog
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Gwen Shapira
>>>>> Product Manager | Confluent
>>>>> 650.450.2760 <(650)%20450-2760> <(650)%20450-2760> | @gwenshap
>>>>> Follow us: Twitter | blog
>>>>>
>>>>
>>>
>>
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Reply via email to