Hi Colin,
Thanks for the feedback!
I understand your concerns (I was thinking about making this improvement
in a way that's fully backward compatible, with similar regex syntax as in
some of the existing tools).
In any case, If there is not enough interest in the user community I'll
rest my case.
Thanks!
--Vahid
From: Colin McCabe <cmcc...@apache.org>
To: us...@kafka.apache.org
Date: 08/06/2018 12:44 PM
Subject: Re: ConsumerGroupCommand tool improvement?
On Mon, Aug 6, 2018, at 12:04, Vahid S Hashemian wrote:
> Hi Colin,
>
> Thanks for considering the idea and sharing your feedback.
>
> The improvements I proposed can be achieved, to some extend, using the
> AdminClient API and the Consumer Group CLI tool. But they won't fully
> support the proposal.
>
> For example,
> Regular expressions are not supported on the groups
> Topic / Client filtering is not supported across all groups
>
> So the reason for proposing the idea was to see if other Kafka users are
> also interested in some of these features so we can remove the burden of
> them writing custom code around existing consumer group features, and
make
> those features built into Kafka Consumer Group Command and AdminClient
> API.
Hmm. If you're writing Java code that calls the APIs, though, this is
easy, right? You can filter the groups in the cluster with a regular
expression with just a single line of code in Java 8.
> adminClient.listConsumerGroups().all().stream().
> filter(listing -> listing.groupId().matches(myRegex)).
> collect(Collectors.toList());
An option to filter ListConsumerGroups by a regular expression might
actually be less easy-to-use for most users than this simple filter, since
users would have to read the JavaDoc for our APIs.
Maybe there are some use-cases where it makes sense to add regex support
to the command-line tools, though.
I guess the reason why I am pushing back on this is that regular
expressions add a lot of complexity to the API and make it harder for us
to meet our backwards compatibility guarantees. For example, Java regular
expressions changed slightly between Java 7 and Java 8
best,
Colin
>
> Thanks again!
> --Vahid
>
>
>
> From: Colin McCabe <cmcc...@apache.org>
> To: us...@kafka.apache.org
> Date: 08/03/2018 04:16 PM
> Subject: Re: ConsumerGroupCommand tool improvement?
>
>
>
> Hi Vahid,
>
> Interesting idea.
>
> It seems like if you're using the AdminClient APIs programmatically, you
> can just do the filtering yourself in a more flexible way than what we
> could provide.
>
> On the other hand, if you're using the ./bin/consumer-groups.sh
> command-line tool, why not use grep or a similar tool to filter the
> output? Maybe there is some additional functionality in supporting
> regexes in the command-line tool, but it also seems like it might be
kind
> of complex as well. Do you have some examples where having regex
support
> int the tool would be much easier than the traditional way of piping the
> output to grep, awk, and sed?
>
> best,
> Colin
>
>
> On Thu, Aug 2, 2018, at 14:23, Vahid S Hashemian wrote:
> > Hi all,
> >
> > A requirement has been raised by a colleague and I wanted to see if
> there
> > is any interest in the community in adding the functionality to Apache
> > Kafka.
> >
> > ConsumerGroupCommand tool in describe ('--describe' or '--describe
> > --offsets') mode currently lists all topics the group has consumed
from
> > and all consumers with assigned partitions for a single group.
> > The idea is to allow filtering of topics, consumers (client ids), and
> even
> > groups using regular expressions. This will allow the tool to handle
use
>
> > cases such as
> > What's the status of a particular consumer (or consumers) in all the
> > groups they are consuming from? (for example to check if they are
> lagging
> > behind in all groups)
> > What consumer groups are consuming from a topic (or topics) and what's
> the
> > lag for each group?
> > Limit the existing result to the topics/consumers of interest (for
> groups
> > with several topics/consumers)
> > ...
> >
> > This would potentially lead to enhancing the AdminClient API as well.
> >
> > If the community also sees a value in this, I could start drafting a
> KIP.
> >
> > Thanks for your feedback.
> > --Vahid
> >
>
>
>
>
>
