Re: [DISCUSS] KIP-379: Multiple Consumer Group Management

[Alex D]

Hello guys,

Thank you for your suggestions!
I've made a short resume of all suggestions proposed for further
possible code corrections.
Since not all opinions match, let's review once again and decide.

#1. Support old csv format. Proposed by Jason.
Yes: Jason, Vahid

If backwards compatibility is important for this specific (and, I
believe, infrequent) case, ready to make corrections. Final word?

#2. Do not show group name for `--describe` output in case a single
`--group` is specified. Proposed by Jason.
Yes: Jason

Alternatively, the user will always expect the output to be the same
for any `--describe` query. Ready to make corrections if this is
important. Final word?

#3. GROUP column should not be the first in the row. Proposed by Jason.
Yes: Jason
No:  Vahid

For the group offset configuration, the group entity appears to be
the top priority and starting a table with a GROUP column makes more
sense, I believe. Plus, it's quicker and easier to spot to which group
the offsets belong to.
Apply corrections or leave as is?

#4. Single regex vs multiple `--group` flags. Proposed by eazama..

There are a few reasons behind this. Firstly, there are no rules for
defining group names unlike for topic names that have their own
validation routine according to a simple regex. Group names may
contain any symbols possible and simply splitting them by comma won't
work, at least without using escape characters maybe. Secondly,
repetition of the `--group` flag had already been implemented for the
group deletion logic and we don't not want to break the backwards
compatibility. Finally, visually, it's a bit easier to read and
compose a long query with a large number of groups than throwing
everything into one very long string.

#5. Valid scenario where we would want to delete all consumer groups.
Asked by Vahid.

There should be one, I believe ;) Already received a few requests
from colleagues.

# KIP approvals:
Suman: +1

> Sat, 20 Oct 2018 17:10:16 GMT,  wrote:
> Is there a reason for using multiple --group flags over having it accept a 
> regex?
>
> The topics command currently accepts a regex for most operations and doesn't 
> support using
> multiple topics flags. It seems like it would be better to take a more 
> standardized approach
> to providing this type of information.
>
>
>> On Oct 19, 2018, at 10:28 AM, Suman B N  wrote:
>>
>> This eases debugging metadata information of consumer groups and offsets in
>> case of client hungs which we have been facing frequently.
>> +1 from me. Well done Alex!
>>
>> -Suman
>>
>> On Fri, Oct 19, 2018 at 8:36 PM Vahid Hashemian 
>> wrote:
>>
>>> Thanks for proposing the KIP. Looks good to me overall.
>>>
>>> I agree with Jason's suggestion that it would be best to keep the current
>>> output format when a single '--group' is present. Because otherwise, there
>>> would be an impact to users who rely on the current output format. Also,
>>> starting with a GROUP column makes more sense to me.
>>>
>>> Also, and for my own info, is there a valid scenario where we would want to
>>> delete all consumer groups? It sounds to me like a potentially dangerous
>>> feature. I would imagine that it would help more with dev/test
>>> environments, where we normally have a few groups (for which the repeating
>>> '--group' option should work).
>>>
>>> Regards!
>>> --Vahid
>>>
>>> On Thu, Oct 18, 2018 at 11:28 PM Jason Gustafson 
>>> wrote:
>>>
>>>> Hi Alex,
>>>>
>>>> Thanks for the KIP. I think it makes sense, especially since most of the
>>>> group apis are intended for batching anyway.
>>>>
>>>> The only questions I have are about compatibility. For example, the csv
>>>> format for resetting offsets is changed, so will we continue to support
>>> the
>>>> old format? Also, if only one `--group` option is passed, do you think
>>> it's
>>>> worth leaving it the group column out of the `--describe` output? That
>>>> would keep the describe output consistent with the current implementation
>>>> for the current usage. Finally, and this is just a nitpick, but do you
>>>> think it makes sense to put the group column first in the describe
>>> output?
>>>>
>>>> Thanks,
>>>> Jason
>>>>
>>>>
>>>>> On Wed, Oct 3, 2018 at 7:11 AM, Alex D  wrote:
>>>>>
>>>>> Hello, friends!
>>>>>
>>>>> Welcome to the Multiple Consumer Group Management feature for
>>>>> kafka-consumer-groups utility discussion thread.
>>>>>
>>>>> KIP is available here:
>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-379%
>>>>> 3A+Multiple+Consumer+Group+Management
>>>>>
>>>>> Pull Request: https://github.com/apache/kafka/pull/5726
>>>>>
>>>>> JIRA ticket: https://issues.apache.org/jira/browse/KAFKA-7471
>>>>>
>>>>> What do you think?
>>>>>
>>>>> Thanks,
>>>>> Alexander Dunayevsky
>>>>>
>>>>
>>>
>>
>>
>> --
>> *Suman*
>> *OlaCabs*
>
>
>

Re: [DISCUSS] KIP-379: Multiple Consumer Group Management

[Alex D]

Hi Jason,

Sorry for the late reply too.
All proposed/dismissed changes done.
Please review the latest PR updates: https://github.com/apache/kafka/pull/5726/

#1. Done. Backwards compatibility with old csv format is preserved.
#2. Not. Considered not to omit the GROUP column for `--describe`
output of a single `--group` since this is less important from a
compatibility perspective.
#3. Done. All tables of --describe output now start from the GROUP column.
#4. Not. Can be implemented as a separate feature.

Do you confirm changes?

Thank you,
Alex Dunayevsky

>Hi Alex,
>
>Sorry for the late reply. Your message didn't get attached to the main
>thread and I missed it.
>
>#1. Yeah, I think we should continue to accept the old csv format.
>Compatibility is important for all public APIs.
>#2. I think this is less important from a compatibility perspective. On the
>one hand, it makes the output compatible with currently supported usage. On
>the other, it makes it more annoying to write tools which invoke this
>command because they need to treat the single group case separately. I'm
>probably leaning toward not doing this one, but I don't have a strong
>opinion.
>#3. To clarify, my suggestion was to put the group id first. I think Vahid
>was in agreement. From your comment, it sounds like you agree as well?
>#4. I agree supporting regex can be left for future work.
>
>Thanks,
>Jason
>
>On Mon, Nov 5, 2018 at 7:55 AM Alex D  wrote:
>
>>Hello guys,
>>
>>Thank you for your suggestions!
>>I've made a short resume of all suggestions proposed for further
>>possible code corrections.
>>Since not all opinions match, let's review once again and decide.
>>
>>#1. Support old csv format. Proposed by Jason.
>>Yes: Jason, Vahid
>>
>>If backwards compatibility is important for this specific (and, I
>>believe, infrequent) case, ready to make corrections. Final word?
>>
>>#2. Do not show group name for `--describe` output in case a single
>>`--group` is specified. Proposed by Jason.
>>Yes: Jason
>>
>>Alternatively, the user will always expect the output to be the
>>same
>>for any `--describe` query. Ready to make corrections if this is
>>important. Final word?
>>
>>#3. GROUP column should not be the first in the row. Proposed by Jason.
>>Yes: Jason
>>No:  Vahid
>>
>>For the group offset configuration, the group entity appears to be
>>the top priority and starting a table with a GROUP column makes more
>>sense, I believe. Plus, it's quicker and easier to spot to which group
>>the offsets belong to.
>>Apply corrections or leave as is?
>>
>>#4. Single regex vs multiple `--group` flags. Proposed by eazama..
>>
>>There are a few reasons behind this. Firstly, there are no rules
>>for
>>defining group names unlike for topic names that have their own
>>validation routine according to a simple regex. Group names may
>>contain any symbols possible and simply splitting them by comma won't
>>work, at least without using escape characters maybe. Secondly,
>>repetition of the `--group` flag had already been implemented for the
>>group deletion logic and we don't not want to break the backwards
>>compatibility. Finally, visually, it's a bit easier to read and
>>compose a long query with a large number of groups than throwing
>>everything into one very long string.
>>
>>#5. Valid scenario where we would want to delete all consumer groups.
>>Asked by Vahid.
>>
>>There should be one, I believe ;) Already received a few requests
>>from colleagues.
>>
>># KIP approvals:
>>Suman: +1
>>
>>> Sat, 20 Oct 2018 17:10:16 GMT,  wrote:
>>> Is there a reason for using multiple --group flags over having it accept
>>a regex?
>>>
>>> The topics command currently accepts a regex for most operations and
>>doesn't support using
>>> multiple topics flags. It seems like it would be better to take a more
>>standardized approach
>>> to providing this type of information.
>>>
>>>
>>>> On Oct 19, 2018, at 10:28 AM, Suman B N  wrote:
>>>>
>>>> This eases debugging metadata information of consumer groups and
>>offsets in
>>>> case of client hungs which we have been facing frequently.
>>>> +1 from me. Well done Alex!
>>>>
>>>> -Suman
>>>>
>>>> On Fri, Oct 19, 2018 at 8:36 PM Vahid Hashemian <
>>vahid.hashem...@gmail.com>
>>>

[DISCUSS] KIP-379: Multiple Consumer Group Management

[Alex D]

Hi Jason,


Sorry for the late reply too.
All proposed/dismissed changes done.
Please review the latest PR updates: https://github.com/apache/kafka/pull/5726/

#1. Done. Backwards compatibility with old csv format is preserved.
#2. Not. Considered not to omit the GROUP column for `--describe`
output of a single `--group` since this is less important from a
compatibility perspective.
#3. Done. All tables of --describe output now start from the GROUP column.
#4. Not. Can be implemented as a separate feature.

Do you confirm changes?

Thank you,
Alex Dunayevsky

>Hi Alex,
>
>Sorry for the late reply. Your message didn't get attached to the main
>thread and I missed it.
>
>#1. Yeah, I think we should continue to accept the old csv format.
>Compatibility is important for all public APIs.
>#2. I think this is less important from a compatibility perspective. On the
>one hand, it makes the output compatible with currently supported usage. On
>the other, it makes it more annoying to write tools which invoke this
>command because they need to treat the single group case separately. I'm
>probably leaning toward not doing this one, but I don't have a strong
>opinion.
>#3. To clarify, my suggestion was to put the group id first. I think Vahid
>was in agreement. From your comment, it sounds like you agree as well?
>#4. I agree supporting regex can be left for future work.
>
>Thanks,
>Jason
>
>On Mon, Nov 5, 2018 at 7:55 AM Alex D  wrote:
>
>>Hello guys,
>>
>>Thank you for your suggestions!
>>I've made a short resume of all suggestions proposed for further
>>possible code corrections.
>>Since not all opinions match, let's review once again and decide.
>>
>>#1. Support old csv format. Proposed by Jason.
>>Yes: Jason, Vahid
>>
>>If backwards compatibility is important for this specific (and, I
>>believe, infrequent) case, ready to make corrections. Final word?
>>
>>#2. Do not show group name for `--describe` output in case a single
>>`--group` is specified. Proposed by Jason.
>>Yes: Jason
>>
>>Alternatively, the user will always expect the output to be the
>>same
>>for any `--describe` query. Ready to make corrections if this is
>>important. Final word?
>>
>>#3. GROUP column should not be the first in the row. Proposed by Jason.
>>Yes: Jason
>>No:  Vahid
>>
>>For the group offset configuration, the group entity appears to be
>>the top priority and starting a table with a GROUP column makes more
>>sense, I believe. Plus, it's quicker and easier to spot to which group
>>the offsets belong to.
>>Apply corrections or leave as is?
>>
>>#4. Single regex vs multiple `--group` flags. Proposed by eazama..
>>
>>There are a few reasons behind this. Firstly, there are no rules
>>for
>>defining group names unlike for topic names that have their own
>>validation routine according to a simple regex. Group names may
>>contain any symbols possible and simply splitting them by comma won't
>>work, at least without using escape characters maybe. Secondly,
>>repetition of the `--group` flag had already been implemented for the
>>group deletion logic and we don't not want to break the backwards
>>compatibility. Finally, visually, it's a bit easier to read and
>>compose a long query with a large number of groups than throwing
>>everything into one very long string.
>>
>>#5. Valid scenario where we would want to delete all consumer groups.
>>Asked by Vahid.
>>
>>There should be one, I believe ;) Already received a few requests
>>from colleagues.
>>
>># KIP approvals:
>>Suman: +1
>>
>>> Sat, 20 Oct 2018 17:10:16 GMT,  wrote:
>>> Is there a reason for using multiple --group flags over having it accept
>>a regex?
>>>
>>> The topics command currently accepts a regex for most operations and
>>doesn't support using
>>> multiple topics flags. It seems like it would be better to take a more
>>standardized approach
>>> to providing this type of information.
>>>
>>>
>>>> On Oct 19, 2018, at 10:28 AM, Suman B N  wrote:
>>>>
>>>> This eases debugging metadata information of consumer groups and
>>offsets in
>>>> case of client hungs which we have been facing frequently.
>>>> +1 from me. Well done Alex!
>>>>
>>>> -Suman
>>>>
>>>> On Fri, Oct 19, 2018 at 8:36 PM Vahid Hashemian <
>>vahid.hashem...@gmail.com>
>>>

[VOTE] KIP-379: Multiple Consumer Group Management

[Alex D]

Hello guys,

We need your VOTES for the KIP-379: Multiple Consumer Group Management.

KIP-379:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-379%3A+Multiple+Consumer+Group+Management

PR: https://github.com/apache/kafka/pull/5726/

Let's start the voting session.

Thank you,

Alex Dunayevsky

Re: [VOTE] KIP-379: Multiple Consumer Group Management

[Alex D]

Hello guys,

2 votes from Jason, Gwen
Any binding votes?


Hello M Manna,
If migrating to JSON makes sense, you can post a new KIP for this purpose.

Thanks,
Alex Dunayevsky

On Tue, 15 Jan 2019, 04:24 Jason Gustafson  +1 Thanks for the KIP!
>
> -Jason
>
> On Mon, Jan 14, 2019 at 5:15 PM Gwen Shapira  wrote:
>
> > I was also wondering about that. I think its for compatibility with
> > the existing output.
> >
> > We can have a separate KIP to add JSON output.
> >
> > On Mon, Jan 14, 2019 at 7:55 AM M. Manna  wrote:
> > >
> > > Hi,
> > >
> > > Thanks for this KIP. Could you kindly clarify why CSV format is useful,
> > but
> > > not anything else?
> > >
> > > CSV format is ancient and the only reason it keep existing is various
> > > application is because legacy applications aren't moving away from
> using
> > > them. Would you would consider JSON or YAML?
> > >
> > > Also, if you think about the kafka-reassign-partitions - it's also
> using
> > > JSON, not CSV. That is my only point. However, if majority feels that
> > it's
> > > not
> > > an issue I believe it's a team decision after all :).
> > >
> > > Thanks,
> > >
> > >
> > > On Mon, 14 Jan 2019 at 15:06, Gwen Shapira  wrote:
> > >
> > > > +1. Thanks, that will be very helpful.
> > > >
> > > > On Mon, Jan 14, 2019, 4:43 AM Alex D  wrote:
> > > >
> > > > > Hello guys,
> > > > >
> > > > > We need your VOTES for the KIP-379: Multiple Consumer Group
> > Management.
> > > > >
> > > > > KIP-379:
> > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-379%3A+Multiple+Consumer+Group+Management
> > > > >
> > > > > PR: https://github.com/apache/kafka/pull/5726/
> > > > >
> > > > > Let's start the voting session.
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Alex Dunayevsky
> > > > >
> > > >
> >
> >
> >
> > --
> > Gwen Shapira
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
>

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Alex D]

Congrats, Vahid!

On Wed, 16 Jan 2019, 05:54 Gwen Shapira  Congrats, Vahid. Thank you for all your contribution!
>
> On Tue, Jan 15, 2019, 2:45 PM Jason Gustafson 
> > Hi All,
> >
> > The PMC for Apache Kafka has invited Vahid Hashemian as a project
> > committer and
> > we are
> > pleased to announce that he has accepted!
> >
> > Vahid has made numerous contributions to the Kafka community over the
> past
> > few years. He has authored 13 KIPs with core improvements to the consumer
> > and the tooling around it. He has also contributed nearly 100 patches
> > affecting all parts of the codebase. Additionally, Vahid puts a lot of
> > effort into community engagement, helping others on the mail lists and
> > sharing his experience at conferences and meetups.
> >
> > We appreciate the contributions and we are looking forward to more.
> > Congrats Vahid!
> >
> > Jason, on behalf of the Apache Kafka PMC
> >
>

Re: [DISCUSS] Kafka 2.2.0 in February 2018

[Alex D]

KIP-379?

On Fri, 18 Jan 2019, 22:33 Matthias J. Sax  Just a quick update on the release.
>
>
> We have 22 KIP atm:
>
> 81, 207, 258, 289, 313, 328, 331, 339, 341, 351, 359, 361, 367, 368,
> 371, 376, 377, 380, 386, 393, 394, 414
>
> Let me know if I missed any KIP that is targeted for 2.2 release.
>
> 21 of those KIPs are accepted, and the vote for the last one is open and
> can be closed on time.
>
> The KIP deadline is Jan 24, so if any late KIPs are coming in, the vote
> must be started latest next Monday Jan 21, to be open for at least 72h
> and to meet the deadline.
>
> Also keep the feature freeze deadline in mind (31 Jan).
>
>
> Besides this, there are 91 open tickets and 41 ticket in progress. I
> will start to go through those tickets soon to see what will make it
> into 2.2 and what we need to defer. If you have any tickets assigned to
> yourself that are target for 2.2 and you know you cannot make it, I
> would appreciate if you could update those ticket yourself to help
> streamlining the release process. Thanks a lot for you support!
>
>
> -Matthias
>
>
> On 1/8/19 7:27 PM, Ismael Juma wrote:
> > Thanks for volunteering Matthias! The plan sounds good to me.
> >
> > Ismael
> >
> > On Tue, Jan 8, 2019, 1:07 PM Matthias J. Sax  wrote:
> >
> >> Hi all,
> >>
> >> I would like to propose a release plan (with me being release manager)
> >> for the next time-based feature release 2.2.0 in February.
> >>
> >> The recent Kafka release history can be found at
> >> https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan.
> >> The release plan (with open issues and planned KIPs) for 2.2.0 can be
> >> found at
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827512
> >> .
> >>
> >>
> >> Here are the suggested dates for Apache Kafka 2.2.0 release:
> >>
> >> 1) KIP Freeze: Jan 24, 2019.
> >>
> >> A KIP must be accepted by this date in order to be considered for this
> >> release)
> >>
> >> 2) Feature Freeze: Jan 31, 2019
> >>
> >> Major features merged & working on stabilization, minor features have
> >> PR, release branch cut; anything not in this state will be automatically
> >> moved to the next release in JIRA.
> >>
> >> 3) Code Freeze: Feb 14, 2019
> >>
> >> The KIP and feature freeze date is about 2-3 weeks from now. Please plan
> >> accordingly for the features you want push into Apache Kafka 2.2.0
> release.
> >>
> >> 4) Release Date: Feb 28, 2019 (tentative)
> >>
> >>
> >> -Matthias
> >>
> >>
> >
>
>

[DISCUSS] KIP-379: Multiple Consumer Group Management

[Alex D]

Hello, friends!

Welcome to the Multiple Consumer Group Management feature for
kafka-consumer-groups utility discussion thread.

KIP is available here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-379%3A+Multiple+Consumer+Group+Management

Pull Request: https://github.com/apache/kafka/pull/5726

JIRA ticket: https://issues.apache.org/jira/browse/KAFKA-7471

What do you think?

Thanks,
Alexander Dunayevsky

KAFKA-7471: Multiple Consumer Group Management PR merging with trunk

[Alex D]

Hello guys,

Any comments/updates on Multiple Consumer Group Management PR:
https://github.com/apache/kafka/pull/5726 ?

Best Regards,
Alex Dunayevsky

Re: ConsumerGroupCommand tool accepting more than one group.

[Alex D]

Hello Ana,

Glad you also like this feature. It has already been implemented by me and
merged into trunk.
Not yet released though.

Best Regs,
Alexander Dunayevsky

On Tue, 14 May 2019, 23:35 Ana Czarnitzki,  wrote:

> Hello,
>
> I wanted to ask if it would be possible to query for more than one group
> with the ConsumerGroupCommand tool. Specifically, I would like to use the
> command with --describe and various groups to get their lag information for
> each topic without having to make the same call many times.
>
> Regards,
> Ana Cz.
>

[DISCUSS] KIP-561: Regex Expressions Support for ConsumerGroupCommand

[Alex D]

Hello, guys,

Please review Regex Expressions Support for ConsumerGroupCommand
improvement proposal

   - *Previous Discussion 1*: Re: Multiple Consumer Group Management
   
   - *Previous Discussion 2*: Re: ConsumerGroupCommand tool improvement?
   

*JIRA*: KAFKA-7817 Multiple Consumer Group Management with Regex


*PR*: #6700