Hi everyone, There has been some changes at the API level to support:
* Request every node to list consumer groups: ``` public class ListConsumerGroupsResult { final KafkaFuture<Map<Node, KafkaFuture<Collection<ConsumerGroupListing>>>> future; //... } ``` * Request coordinators to describe/list Offset/delete consumer groups: ``` public class DescribeConsumerGroupResult { private final KafkaFuture<Map<String, KafkaFuture<ConsumerGroupDescription>>> futures; ... } ... public class ListConsumerGroupOffsetsResult { final KafkaFuture<Map<TopicPartition, OffsetAndMetadata>> future; //... } ... public class DeleteConsumerGroupsResult { final KafkaFuture<Map<String, KafkaFuture<Void>>> future; } ``` Looking forward to your feedback. If these changes look good we can keep the discussion on the PR. Cheers, Jorge. El mar., 6 feb. 2018 a las 17:47, Jorge Esteban Quilcate Otoya (< quilcate.jo...@gmail.com>) escribió: > Hi everyone, > > I have add some changes to the KIP based on the Pull Request: > https://github.com/apache/kafka/pull/4454#issuecomment-360553277 : > > * Reduce the scope of the operations to Consumer Groups to avoid > complexity of making assignments generic for Consumer and Connect groups. > If Connect Group operations are required we can handle it in another KIP or > add it here if needed. > * Consider adding `deleteConsumerGroups` API once > https://cwiki.apache.org/confluence/display/KAFKA/KIP-229%3A+DeleteGroups+API > is > merged. > > Looking forward to your feedback. > > If these changes look good we can keep the discussion on the PR. > > Cheers, > Jorge. > > El dom., 7 ene. 2018 a las 21:00, Jorge Esteban Quilcate Otoya (< > quilcate.jo...@gmail.com>) escribió: > >> Great! >> >> I have added `listGroupOffsets` to the KIP. >> >> If there are no additional feedback, VOTE thread is already open. >> >> Cheers, >> Jorge >> >> >> El mar., 2 ene. 2018 a las 17:49, Gwen Shapira (<g...@confluent.io>) >> escribió: >> >>> listGroups and listGroupOffsets will make it a snap to transition the >>> existing ConsumerGroups CLI to depend on client libraries only. >>> >>> Thanks for adding them :) >>> >>> On Sun, Dec 31, 2017 at 1:39 PM Jorge Esteban Quilcate Otoya < >>> quilcate.jo...@gmail.com> wrote: >>> >>> > Thanks all for your feedback, and sorry for late response. >>> > >>> > I'm considering the following: >>> > >>> > ```AdminClient.java >>> > public abstract ListGroupsResult listGroups(ListGroupsOptions >>> options); >>> > >>> > public ListGroupsResult listGroups() { >>> > return listGroups(new ListGroupsOptions()); >>> > } >>> > >>> > public ListGroupsResult listConsumerGroups(ListGroupsOptions >>> options) { >>> > //filtering groups by ConsumerProtocol.PROTOCOL_TYPE >>> > } >>> > >>> > public ListGroupsResult listConsumerGroups() { >>> > return listConsumerGroups(new ListGroupsOptions()); >>> > } >>> > ``` >>> > >>> > About `describeConsumerGroups`, I'm considering renaming to >>> > `describeGroups` and rename `ConsumerGroupDescription` and >>> > `ConsumerDescription` to `GroupDescription` to `MemberDescription`. >>> > Not sure we need a deserializer, we can access `DescribeGroupsResponse` >>> > members directly. >>> > >>> > As @dan says, I also think `listGroupOffsets` could be added to this >>> KIP to >>> > make it complete. >>> > >>> > I'm thinking about renaming this KIP to "Add Consumer Group operations >>> to >>> > Admin API". >>> > >>> > I'm updating the KIP accordingly. >>> > >>> > Cheers and happy 2018! >>> > >>> > Jorge. >>> > >>> > El mié., 13 dic. 2017 a las 19:06, Colin McCabe (<cmcc...@apache.org>) >>> > escribió: >>> > >>> > > On Tue, Dec 12, 2017, at 09:39, Jason Gustafson wrote: >>> > > > Hi Colin, >>> > > > >>> > > > They do share the same namespace. We have a "protocol type" field >>> in >>> > the >>> > > > JoinGroup request to make sure that all members are of the same >>> kind. >>> > > >>> > > Hi Jason, >>> > > >>> > > Thanks. That makes sense. >>> > > >>> > > > Very roughly what I was thinking is something like this. First we >>> > > introduce an >>> > > > interface for deserialization: >>> > > > >>> > > > interface GroupMetadataDeserializer<Metadata, Assignment> { >>> > > > String protocolType(); >>> > > > Metadata desrializeMetadata(ByteBuffer); >>> > > > Assignment deserializeAssignment(ByteBuffer); >>> > > > } >>> > > > >>> > > > Then we add some kind of generic container: >>> > > > >>> > > > class MemberMetadata<Metadata, Assignment> { >>> > > > Metadata metadata; >>> > > > Assignment assignment; >>> > > > } >>> > > > >>> > > > Then we have two APIs: one generic and one specific to consumer >>> groups: >>> > > > >>> > > > <M, A> Map<String, MemberMetadata<M,A>> describeGroup(String >>> groupId, >>> > > > GroupMetadataDeserializer<M, A> deserializer); >>> > > > >>> > > > Map<String, ConsumerGroupMetadata> describeConsumerGroup(String >>> > groupId); >>> > > > >>> > > > (This is just a sketch, so obviously we can change them to use >>> futures >>> > or >>> > > > to batch or whatever.) >>> > > > >>> > > > I think it would be fine to not provide a connect-specific API >>> since >>> > this >>> > > > usage will probably be limited to Connect itself. >>> > > >>> > > Yeah, it probably makes sense to have a separation between >>> describeGroup >>> > > and describeConsumerGroup. >>> > > >>> > > We will have to be pretty careful with cross-version compatibility in >>> > > describeConsumerGroup. It should be possible for an old client to >>> talk >>> > > to a new broker, and a new client to talk to an old broker. So we >>> > > should be prepared to read data in multiple formats. >>> > > >>> > > I'm not sure if we need to have a 'deserializer' argument to >>> > > describeGroup. We can just let them access a byte array, right? >>> > > Theoretically they might also just want to check for the presence or >>> > > absence of a group, but not deserialize anything. >>> > > >>> > > best, >>> > > Colin >>> > > >>> > > > >>> > > > Thanks, >>> > > > Jason >>> > > > >>> > > > >>> > > > On Mon, Dec 11, 2017 at 9:15 PM, Colin McCabe <cmcc...@apache.org> >>> > > wrote: >>> > > > >>> > > > > Sorry... this is probably a silly question, but do Kafka Connect >>> > groups >>> > > > > share a namespace with consumer groups? If we had a separate >>> API for >>> > > > > Kafka Connect groups vs. Consumer groups, would that make >>> sense? Or >>> > > > > should we unify them? >>> > > > > >>> > > > > best, >>> > > > > Colin >>> > > > > >>> > > > > >>> > > > > On Mon, Dec 11, 2017, at 16:11, Jason Gustafson wrote: >>> > > > > > Hi Jorge, >>> > > > > > >>> > > > > > Kafka group management is actually more general than consumer >>> > groups >>> > > > > > (e.g. >>> > > > > > there are kafka connect groups). If we are adding these APIs, I >>> > would >>> > > > > > suggest we consider the more general protocol and how to expose >>> > > > > > group-protocol-specific metadata. For example, it might be >>> > > reasonable to >>> > > > > > have both an API to access to the low-level bytes as well as >>> some >>> > > > > > higher-level convenience APIs for accessing consumer groups. >>> > > > > > >>> > > > > > Thanks, >>> > > > > > Jason >>> > > > > > >>> > > > > > On Mon, Dec 4, 2017 at 4:07 PM, Matthias J. Sax < >>> > > matth...@confluent.io> >>> > > > > > wrote: >>> > > > > > >>> > > > > > > Jorge, >>> > > > > > > >>> > > > > > > is there any update regarding this KIP? >>> > > > > > > >>> > > > > > > >>> > > > > > > -Matthias >>> > > > > > > >>> > > > > > > On 11/17/17 9:14 AM, Guozhang Wang wrote: >>> > > > > > > > Hello Jorge, >>> > > > > > > > >>> > > > > > > > I made a pass over the wiki, and here are a few comments: >>> > > > > > > > >>> > > > > > > > 1. First, regarding to Tom's comment #2 above, I think if >>> we >>> > are >>> > > only >>> > > > > > > going >>> > > > > > > > to include the String groupId. Then it is Okay to keep as a >>> > > String >>> > > > > than >>> > > > > > > > using a new wrapper class. However, I think we could >>> include >>> > the >>> > > > > > > > protocol_type returned from the ListGroupsResponse along >>> with >>> > the >>> > > > > > > groupId. >>> > > > > > > > This is a very useful information to tell which consumer >>> groups >>> > > are >>> > > > > from >>> > > > > > > > Connect, which ones are from Streams, which ones are >>> > > user-customized >>> > > > > etc. >>> > > > > > > > With this, it is reasonable to keep a wrapper class. >>> > > > > > > > >>> > > > > > > > 2. In ConsumerDescription, could we also add the state, >>> > > protocol_type >>> > > > > > > > (these two are form DescribeGroupResponse), and the Node >>> > > coordinator >>> > > > > > > (this >>> > > > > > > > may be returned from the AdminClient itself) as well? This >>> is >>> > > also >>> > > > > for >>> > > > > > > > information consistency with the old client (note that >>> > > protocol_type >>> > > > > was >>> > > > > > > > called assignment_strategy there). >>> > > > > > > > >>> > > > > > > > 3. With 1) / 2) above, maybe we can rename >>> > > "ConsumerGroupListing" to >>> > > > > > > > "ConsumerGroupSummary" and make "ConsumerGroupDescription" >>> an >>> > > > > extended >>> > > > > > > > class of the former with the additional fields? >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > Guozhang >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > On Tue, Nov 7, 2017 at 2:13 AM, Jorge Esteban Quilcate >>> Otoya < >>> > > > > > > > quilcate.jo...@gmail.com> wrote: >>> > > > > > > > >>> > > > > > > >> Hi Tom, >>> > > > > > > >> >>> > > > > > > >> 1. You're right. I've updated the KIP accordingly. >>> > > > > > > >> 2. Yes, I have add it to keep consistency, but I'd like to >>> > know >>> > > what >>> > > > > > > others >>> > > > > > > >> think about this too. >>> > > > > > > >> >>> > > > > > > >> Cheers, >>> > > > > > > >> Jorge. >>> > > > > > > >> >>> > > > > > > >> El mar., 7 nov. 2017 a las 9:29, Tom Bentley (< >>> > > > > t.j.bent...@gmail.com>) >>> > > > > > > >> escribió: >>> > > > > > > >> >>> > > > > > > >>> Hi again Jorge, >>> > > > > > > >>> >>> > > > > > > >>> A couple of minor points: >>> > > > > > > >>> >>> > > > > > > >>> 1. ConsumerGroupDescription has the member `name`, but >>> > > everywhere >>> > > > > else >>> > > > > > > >> that >>> > > > > > > >>> I've seen the term "group id" is used, so perhaps >>> calling it >>> > > "id" >>> > > > > or >>> > > > > > > >>> "groupId" would be more consistent. >>> > > > > > > >>> 2. I think you've added ConsumerGroupListing for >>> consistency >>> > > with >>> > > > > > > >>> TopicListing. For topics it makes sense because at well >>> as >>> > the >>> > > name >>> > > > > > > there >>> > > > > > > >>> is whether the topic is internal. For consumer groups, >>> though >>> > > > > there is >>> > > > > > > >> just >>> > > > > > > >>> the name and having a separate ConsumerGroupListing seems >>> > like >>> > > it >>> > > > > > > doesn't >>> > > > > > > >>> add very much, and would mostly get in the way when >>> using the >>> > > API. >>> > > > > I >>> > > > > > > >> would >>> > > > > > > >>> be interested in what others thought about this. >>> > > > > > > >>> >>> > > > > > > >>> Cheers, >>> > > > > > > >>> >>> > > > > > > >>> Tom >>> > > > > > > >>> >>> > > > > > > >>> On 6 November 2017 at 22:16, Jorge Esteban Quilcate >>> Otoya < >>> > > > > > > >>> quilcate.jo...@gmail.com> wrote: >>> > > > > > > >>> >>> > > > > > > >>>> Thanks for the feedback! >>> > > > > > > >>>> >>> > > > > > > >>>> @Ted Yu: Links added. >>> > > > > > > >>>> >>> > > > > > > >>>> KIP updated. Changes: >>> > > > > > > >>>> >>> > > > > > > >>>> * `#listConsumerGroups(ListConsumerGroupsOptions >>> options)` >>> > > added >>> > > > > to >>> > > > > > > >> the >>> > > > > > > >>>> API. >>> > > > > > > >>>> * `DescribeConsumerGroupResult` and >>> > `ConsumerGroupDescription` >>> > > > > classes >>> > > > > > > >>>> described. >>> > > > > > > >>>> >>> > > > > > > >>>> Cheers, >>> > > > > > > >>>> Jorge. >>> > > > > > > >>>> >>> > > > > > > >>>> >>> > > > > > > >>>> >>> > > > > > > >>>> >>> > > > > > > >>>> El lun., 6 nov. 2017 a las 20:28, Guozhang Wang (< >>> > > > > wangg...@gmail.com >>> > > > > > > >) >>> > > > > > > >>>> escribió: >>> > > > > > > >>>> >>> > > > > > > >>>>> Hi Matthias, >>> > > > > > > >>>>> >>> > > > > > > >>>>> You meant "list groups" I think? >>> > > > > > > >>>>> >>> > > > > > > >>>>> Guozhang >>> > > > > > > >>>>> >>> > > > > > > >>>>> On Mon, Nov 6, 2017 at 11:17 AM, Matthias J. Sax < >>> > > > > > > >>> matth...@confluent.io> >>> > > > > > > >>>>> wrote: >>> > > > > > > >>>>> >>> > > > > > > >>>>>> The main goal of this KIP is to enable decoupling >>> > > > > StreamsResetter >>> > > > > > > >>> from >>> > > > > > > >>>>>> core module. For this case (ie, using AdminClient >>> within >>> > > > > > > >>>>>> StreamsResetter) we get the group.id from the user as >>> > > command >>> > > > > line >>> > > > > > > >>>>>> argument. Thus, I think the KIP is useful without >>> > "describe >>> > > > > group" >>> > > > > > > >>>>>> command to. >>> > > > > > > >>>>>> >>> > > > > > > >>>>>> I am happy to include "describe group" command in the >>> KIP. >>> > > Just >>> > > > > > > >> want >>> > > > > > > >>> to >>> > > > > > > >>>>>> point out, that there is no reason to insist on it >>> IMHO. >>> > > > > > > >>>>>> >>> > > > > > > >>>>>> >>> > > > > > > >>>>>> -Matthias >>> > > > > > > >>>>>> >>> > > > > > > >>>>>> On 11/6/17 7:06 PM, Guozhang Wang wrote: >>> > > > > > > >>>>>>> A quick question: I think we do not yet have the >>> `list >>> > > consumer >>> > > > > > > >>>> groups` >>> > > > > > > >>>>>>> func as in the old AdminClient. Without this >>> `describe >>> > > group` >>> > > > > > > >> given >>> > > > > > > >>>> the >>> > > > > > > >>>>>>> group id would not be very useful. Could you include >>> this >>> > > as >>> > > > > well >>> > > > > > > >>> in >>> > > > > > > >>>>> your >>> > > > > > > >>>>>>> KIP? More specifically, you can look at >>> > > > > > > >> kafka.admin.AdminClientfor >>> > > > > > > >>>> more >>> > > > > > > >>>>>>> details on the APIs. >>> > > > > > > >>>>>>> >>> > > > > > > >>>>>>> >>> > > > > > > >>>>>>> Guozhang >>> > > > > > > >>>>>>> >>> > > > > > > >>>>>>> On Mon, Nov 6, 2017 at 7:22 AM, Ted Yu < >>> > > yuzhih...@gmail.com> >>> > > > > > > >>> wrote: >>> > > > > > > >>>>>>> >>> > > > > > > >>>>>>>> Please fill out Discussion thread and JIRA fields. >>> > > > > > > >>>>>>>> >>> > > > > > > >>>>>>>> Thanks >>> > > > > > > >>>>>>>> >>> > > > > > > >>>>>>>> On Mon, Nov 6, 2017 at 2:02 AM, Tom Bentley < >>> > > > > > > >>> t.j.bent...@gmail.com> >>> > > > > > > >>>>>> wrote: >>> > > > > > > >>>>>>>> >>> > > > > > > >>>>>>>>> Hi Jorge, >>> > > > > > > >>>>>>>>> >>> > > > > > > >>>>>>>>> Thanks for the KIP. A few initial comments: >>> > > > > > > >>>>>>>>> >>> > > > > > > >>>>>>>>> 1. The AdminClient doesn't have any API like >>> > > > > > > >>> `listConsumerGroups()` >>> > > > > > > >>>>>>>>> currently, so in general how does a client know the >>> > > group ids >>> > > > > > > >> it >>> > > > > > > >>> is >>> > > > > > > >>>>>>>>> interested in? >>> > > > > > > >>>>>>>>> 2. Could you fill in the API of >>> > > DescribeConsumerGroupResult, >>> > > > > > > >> just >>> > > > > > > >>>> so >>> > > > > > > >>>>>>>>> everyone knows exactly what being proposed. >>> > > > > > > >>>>>>>>> 3. Can you describe the ConsumerGroupDescription >>> class? >>> > > > > > > >>>>>>>>> 4. Probably worth mentioning that this will use >>> > > > > > > >>>>>>>>> DescribeGroupsRequest/Response, and also >>> enumerating >>> > the >>> > > > > error >>> > > > > > > >>>> codes >>> > > > > > > >>>>>>>> that >>> > > > > > > >>>>>>>>> can return (or, equivalently, enumerate the >>> exceptions >>> > > throw >>> > > > > > > >> from >>> > > > > > > >>>> the >>> > > > > > > >>>>>>>>> futures obtained from the >>> DescribeConsumerGroupResult). >>> > > > > > > >>>>>>>>> >>> > > > > > > >>>>>>>>> Cheers, >>> > > > > > > >>>>>>>>> >>> > > > > > > >>>>>>>>> Tom >>> > > > > > > >>>>>>>>> >>> > > > > > > >>>>>>>>> On 6 November 2017 at 08:19, Jorge Esteban Quilcate >>> > > Otoya < >>> > > > > > > >>>>>>>>> quilcate.jo...@gmail.com> wrote: >>> > > > > > > >>>>>>>>> >>> > > > > > > >>>>>>>>>> Hi everyone, >>> > > > > > > >>>>>>>>>> >>> > > > > > > >>>>>>>>>> I would like to start a discussion on KIP-222 [1] >>> > based >>> > > on >>> > > > > > > >> issue >>> > > > > > > >>>>> [2]. >>> > > > > > > >>>>>>>>>> >>> > > > > > > >>>>>>>>>> Looking forward to feedback. >>> > > > > > > >>>>>>>>>> >>> > > > > > > >>>>>>>>>> [1] >>> > > > > > > >>>>>>>>>> >>> https://cwiki.apache.org/confluence/pages/viewpage. >>> > > > > > > >>>>>>>>> action?pageId=74686265 >>> > > > > > > >>>>>>>>>> [2] >>> https://issues.apache.org/jira/browse/KAFKA-6058 >>> > > > > > > >>>>>>>>>> >>> > > > > > > >>>>>>>>>> Cheers, >>> > > > > > > >>>>>>>>>> Jorge. >>> > > > > > > >>>>>>>>>> >>> > > > > > > >>>>>>>>> >>> > > > > > > >>>>>>>> >>> > > > > > > >>>>>>> >>> > > > > > > >>>>>>> >>> > > > > > > >>>>>>> >>> > > > > > > >>>>>> >>> > > > > > > >>>>>> >>> > > > > > > >>>>> >>> > > > > > > >>>>> >>> > > > > > > >>>>> -- >>> > > > > > > >>>>> -- Guozhang >>> > > > > > > >>>>> >>> > > > > > > >>>> >>> > > > > > > >>> >>> > > > > > > >> >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > >>> > > >>> > >>> >>