I like the --reset-to-earliest and --reset-to-latest. In general, since the JSON route is the most challenging for users, we want to provide a lot of ways to do useful things without going there.
Two things that can help: 1. A lot of times, users want to skip few messages that cause issues and continue. maybe just specifying the topic, partition and delta will be better than having to find the offset and write a JSON and validate the JSON etc. 2. Thinking if there are other common use-cases that we can make easy rather than just one generic but not very usable method. Gwen On Wed, Feb 8, 2017 at 3:25 AM, Jorge Esteban Quilcate Otoya <quilcate.jo...@gmail.com> wrote: > Thanks for the feedback! > > @Onur, @Gwen: > > Agree. Actually at the first draft I considered to have it inside > ´kafka-consumer-groups.sh´, but I decide to propose it as a standalone tool > to describe it clearly and focus it on reset functionality. > > But now that you mentioned, it does make sense to have it in > ´kafka-consumer-groups.sh´. How would be a consistent way to introduce it? > > Maybe something like this: > > ´kafka-consumer-groups.sh --reset-offset --generate --group cg1 --topics t1 > --reset-from 2017-01-01T00:00:00.000 --output plan.json´ > > ´kafka-consumer-groups.sh --reset-offset --verify --reset-json-file > plan.json´ > > ´kafka-consumer-groups.sh --reset-offset --execute --reset-json-file > plan.json´ > > ´kafka-consumer-groups.sh --reset-offset --generate-and-execute --group cg1 > --topics t1 --reset-from 2017-01-01T00:00:00.000´ > > @Gwen: > >> It looks exactly like the replica assignment tool > > It was influenced by ;-) I use the generate-verify-execute process here to > make sure user will be aware of the result of this operation. At the > beginning we considered only add a couple of options to Consumer Group > Command: > > --rewind-to-timestamp and --rewind-to-period > > @Onur: > >> You can actually get away with overriding while members of the group are live > with method 2 by using group information from DescribeGroupsRequest. > > This means that we need to have Consumer Group stopped before executing and > start a new consumer internally to do this? Therefore, we won't be able to > consider executing reset when ConsumerGroup is active? (trying to relate it > with @Dong 5th question) > > @Dong: > >> Should we allow user to use wildcard to reset offset of all groups for a > given topic as well? > > I haven't thought about this scenario. Could be interesting. Following the > recommendation to add it into Consumer Group Command, in this case Group > argument will be optional if there are only 1 topic. I think for multiple > topic won't be that useful. > >> Should we allow user to specify timestamp per topic partition in the json > file as well? > > Don't think this could be a valid from the tool, but if Reset Plan is > generated, and user want to set the offset for a specific partition to > other offset (eventually based on another timestamp), and execute it, it > will be up to her/him. > >> Should the script take some credential file to make sure that this > operation is authenticated given the potential impact of this operation? > > Haven't tried to secure brokers yet, but the tool should support > authorization if it's enabled in the broker. > >> Should we provide constant to reset committed offset to earliest/latest > offset of a partition, e.g. -1 indicates earliest offset and -2 indicates > latest offset. > > I will go for something like ´--reset-to-earliest´ and ´--reset-to-latest´ > >> Should we allow dynamic change of the comitted offset when consumer are > running, such that consumer will seek to the newly committed offset and > start consuming from there? > > Not sure about this. I will recommend to keep it simple and ask user to > stop consumers first. But I would considered it if the trade-offs are > clear. > > @Matthias > > Added :). And thanks a lot for your help to define this KIP! > > > > El mié., 8 feb. 2017 a las 7:47, Gwen Shapira (<g...@confluent.io>) > escribió: > >> As long as the CLI is a bit consistent? Like, not just adding 3 >> arguments and a JSON parser to the existing tool, right? >> >> On Tue, Feb 7, 2017 at 10:29 PM, Onur Karaman >> <onurkaraman.apa...@gmail.com> wrote: >> > I think it makes sense to just add the feature to >> kafka-consumer-groups.sh >> > >> > On Tue, Feb 7, 2017 at 10:24 PM, Gwen Shapira <g...@confluent.io> wrote: >> > >> >> Thanks for the KIP. I'm super happy about adding the capability. >> >> >> >> I hate the interface, though. It looks exactly like the replica >> >> assignment tool. A tool everyone loves so much that there are multiple >> >> projects, open and closed, that try to fix it. >> >> >> >> Can we swap it with something that looks a bit more like the consumer >> >> group tool? or the kafka streams reset tool? Consistency is helpful in >> >> such cases. I spent some time learning existing tools and learning yet >> >> another one is a deterrent. >> >> >> >> Gwen >> >> >> >> >> >> >> >> On Tue, Feb 7, 2017 at 6:43 PM, Jorge Esteban Quilcate Otoya >> >> <quilcate.jo...@gmail.com> wrote: >> >> > Hi all, >> >> > >> >> > I would like to propose a KIP to Add a tool to Reset Consumer Group >> >> Offsets. >> >> > >> >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP- >> >> 122%3A+Add+a+tool+to+Reset+Consumer+Group+Offsets >> >> > >> >> > Please, take a look at the proposal and share your feedback. >> >> > >> >> > Thanks, >> >> > Jorge. >> >> >> >> >> >> >> >> -- >> >> Gwen Shapira >> >> Product Manager | Confluent >> >> 650.450.2760 <(650)%20450-2760> | @gwenshap >> >> Follow us: Twitter | blog >> >> >> >> >> >> -- >> Gwen Shapira >> Product Manager | Confluent >> 650.450.2760 <(650)%20450-2760> | @gwenshap >> Follow us: Twitter | blog >> -- Gwen Shapira Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter | blog
