I see the consumer group much like the username in relational database access credentials. We routinely use the consumer group name as the means of identifying the consumer for operational things, such as alerting based on consumer group lag, autoscaling based on lag, and tooling around manipulating the consumer group offset for a particular service. The consumer group allows us to know, operationally, which offsets we are observing or manipulating, and ideally we would like to limit the set of consumer groups possible.
In practice, I regularly see people simply append an incrementing integer to the end of their consumer group (cg, cg1, cg2, cg1234) when they want to reset their application, INSTEAD of following offset reset or kafka-streams application reset procedures. Sure, it would be nice to get everyone to follow recommended procedures, but people do what they CAN do, not what they're supposed to do. This has significant impact on the brokers and surrounding tooling (orphaned internal topics with indefinite retention, falsely-firing lag monitoring, broken auto-scaling), and instead of us building out layers of supportive tooling to isolate ourselves from it, why not simply set up ACLs to enforce which consumer groups an application can and cannot use? Does this need a KIP? Or is a bug report simply enough? I am unsure how compatibility would work, so I am leaning towards a KIP... Thanks Adam On Wed, Aug 21, 2019 at 5:59 PM Colin McCabe <cmcc...@apache.org> wrote: > I think it's worth considering separating out the permissions needed to > create a consumer group from the permissions needed to join one. We > distinguish these permissions for topics, and people generally find it > useful. We could start checking CREATE on GROUP, perhaps? It might be > hard to do in a compatible way. > > cheers, > Colin > > > On Wed, Aug 21, 2019, at 12:05, Adam Bellemare wrote: > > +users mailing list > > > > David, > > > > I don't think I really understand your email. Are you saying that this > can > > already be achieved only using the READ ACL? > > > > Thanks > > Adam > > > > > > > > On Wed, Aug 21, 2019 at 3:58 AM David Jacot <dja...@confluent.io> wrote: > > > > > Hello, > > > > > > It would be better to ask such question on the user mailing list. > > > > > > The reason is that the group is created automatically when a consumer > > > joins it. It is not created explicitly so it can be restricted. > > > > > > In your case, you could setup a ACL to authorize the application to > only > > > use the group you have defined. It would prevent the application from > > > creating new groups. (READ Acl on Group resource with a specific name). > > > > > > Best, > > > David > > > > > > On Mon, Aug 19, 2019 at 9:01 PM Adam Bellemare < > adam.bellem...@gmail.com> > > > wrote: > > > > > > > Hi All > > > > > > > > I am looking through the Confluent docs and core Kafka docs and > don't see > > > > an ACL for group creation: > > > > > https://docs.confluent.io/current/kafka/authorization.html#acl-format > > > > and > > > > https://kafka.apache.org/documentation/#security_authz > > > > > > > > My scenario is simple: We use the consumer group as the means of > > > > identifying a single application, including tooling for managing > > > > application resets, offset management, lag monitoring, etc. We often > have > > > > situations where someone resets their consumer group by appending an > > > > incremented integer ("cg" to "cg1"), but it throws the rest of the > > > > monitoring and management tooling out of whack. > > > > > > > > Is there a reason why we do not have ACL-based CREATE restrictions > to a > > > > particular consumer group? I am willing to do the work to implement > this > > > > and test it out, but I wanted to validate that there isn't a reason > I am > > > > missing. > > > > > > > > Thanks > > > > Adam > > > > > > > > > >
