Re: [DISCUSS] KIP-40 - Extend GroupMetadata request to return group and member status

2015-10-29 Thread Jason Gustafson
Forgot to include a link to the changed KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-40%3A+ListGroups+and+DescribeGroup -Jason On Thu, Oct 29, 2015 at 9:43 AM, Jason Gustafson wrote: > Hi All, > > I've updated the KIP for ListGroups and DescribeGroup. This does simplify > things q

Re: [DISCUSS] KIP-40 - Extend GroupMetadata request to return group and member status

2015-10-29 Thread Jason Gustafson
Hi All, I've updated the KIP for ListGroups and DescribeGroup. This does simplify things quite a bit. Since this is a blocker for 0.9 and there doesn't seem to be any major objections, let's go ahead and start the vote. Minor issues with the schemas can be addressed on the JIRA. Thanks, Jason

Re: [DISCUSS] KIP-40 - Extend GroupMetadata request to return group and member status

2015-10-29 Thread Neha Narkhede
I like Jun's suggestion of adding ListGroups and DescribeGroup. It keeps the abstractions simple and avoids overloading existing requests. On Wed, Oct 28, 2015 at 8:08 PM, Jason Gustafson wrote: > Hey Jun, > > Thanks for taking a look at this. Initially the thought was to make > GroupMetadataReq

Re: [DISCUSS] KIP-40 - Extend GroupMetadata request to return group and member status

2015-10-28 Thread Jason Gustafson
Hey Jun, Thanks for taking a look at this. Initially the thought was to make GroupMetadataRequest analogous to TopicMetadataRequest: one accepts an array of topics and returns topic metadata, the other accepts an array of groupIds and returns group metadata. However, the analogy doesn't quite fit

Re: [DISCUSS] KIP-40 - Extend GroupMetadata request to return group and member status

2015-10-28 Thread Jun Rao
Jason, Thanks for the writeup. Perhaps we can have two new requests: DescribeConsumerGroup and ListConsumerGroup. The former takes a list of groups and returns a list of metadata (members, group metadata, member metadata, etc) for each group. The latter takes nothing and just returns a list of gro

Re: [DISCUSS] KIP-40 - Extend GroupMetadata request to return group and member status

2015-10-28 Thread Ismael Juma
On Wed, Oct 28, 2015 at 10:25 PM, Jason Gustafson wrote: > Hey Ashish, > > Yeah, that's fine with me too. I thought people kind of frowned upon the > use of an empty topic list to get all topics, but perhaps it's more of an > issue at the user API level. > Yes, empty list to represent "all" is q

Re: [DISCUSS] KIP-40 - Extend GroupMetadata request to return group and member status

2015-10-28 Thread Jason Gustafson
Hey Ashish, Yeah, that's fine with me too. I thought people kind of frowned upon the use of an empty topic list to get all topics, but perhaps it's more of an issue at the user API level. -Jason On Wed, Oct 28, 2015 at 2:58 PM, Ashish Singh wrote: > Jason, thanks for the great write up. I am o

Re: [DISCUSS] KIP-40 - Extend GroupMetadata request to return group and member status

2015-10-28 Thread Jason Gustafson
Hey Neha, Thanks for the feedback. I don't have a strong position on either of the points you mentioned. If we're fine having an additional request type, then maybe we could do something like this: 1. GroupMetadata accepts an array of groupIds and just returns the coordinator for each group. An e

Re: [DISCUSS] KIP-40 - Extend GroupMetadata request to return group and member status

2015-10-28 Thread Ashish Singh
Jason, thanks for the great write up. I am overall in favor of changes suggested. However, I too think that there is no specific need of *IncludeAllGroups* flag, but that could be due to me not being aware of why this pattern is frowned upon for TopicMetadataRequest. To me it simply eases the use.

Re: [DISCUSS] KIP-40 - Extend GroupMetadata request to return group and member status

2015-10-28 Thread Neha Narkhede
I slightly prefer one of the rejected alternatives over the currently suggested one - which is to add a separate DescribeGroupRequest that always returns the member metadata for the group and any other information useful for monitoring and tooling. It helps keep the abstractions clean and also redu

Re: [DISCUSS] KIP-40 - Extend GroupMetadata request to return group and member status

2015-10-28 Thread Gwen Shapira
Looks awesome to me :) This will allow to both list all groups and to retrieve offsets for specific groups. Since 3 days passed with no comments, would you like to start a vote? On Sun, Oct 25, 2015 at 6:29 PM, Jason Gustafson wrote: > Hi Kafka Devs, > > Currently, the new consumer provides no

[DISCUSS] KIP-40 - Extend GroupMetadata request to return group and member status

2015-10-25 Thread Jason Gustafson
Hi Kafka Devs, Currently, the new consumer provides no way to view a group's status except by inspecting coordinator and consumer logs. This includes listing the members of the group and their partition assignments. For the old consumer, tools could read this information directly from Zookeeper, b