Hello, I am trying to come up with a good security approach for a Kafka project inside our company. I see there's a variety of options available (ACL/RBAC/certificates/...), and I'm hoping someone can suggest a few possibilities. Some points of interest,
- Kafka will be used both by end-users/researchers (inside the company) and by some microservices. The users are on a wide variety of platforms (Windows/Linux, JVM/C++/Python/R/Javascript/MATLAB/...). The microservices are probably JVM/Python. - We'd like to make heavy use of both the Schema Registry and REST proxy, for all users, - Users & groups are in LDAP, - Permissioning would need to be granular and at the level of Kafka (topics, consumer groups, ...), the Schema Registry, the REST Proxy, ... - based on LDAP group membership. - Ideally we'd like to use user/password authentication throughout, which is the most widely supported mechanism (as we use many different platforms). The user is the LDAP username, but the password is an API key that users manage separately. For security reasons we cannot use the LDAP password, as it would be too likely to leak (e.g. in git etc). (We already use the LDAP user + API key combo for authentication in other places). For Kafka, I've played around with an approach similar to https://github.com/navikt/kafka-plain-saslserver-2-ad, i.e. writing classes that extend org.apache.kafka.common.security.auth.AuthenticateCallbackHandler and kafka.security.auth.Authorizer for providing our own rules, which works perfectly fine. In this case, - the authenticator checks the username against our own database of (user -> [apiKeys]), - the authoriser meaningfully overrides only the "authorize(..)" function (leaving all the *Acl ones as no-ops) and leverages LDAP (in particular group membership) to decide upon authorization, using an additional database encoding group permissions. This is a bit hacky - and more a proof-of-concept than anything else, but it does seem to cover our needs, as far as Kafka goes. However, I believe this is only available for Kafka itself, and a similar approach wouldn't extend to the REST proxy and the Schema Registry (and Connectors and the Control Center). Ideally, we want a centralized auth service. In that context, I've been looking at Role-Based Access Control. I think it would suit the majority of our purposes, except for the fact that it seems to authenticate using LDAP only (using LDAP user/pass for authentication), which we cannot do. I was wondering, - Is it possible to extend the RBAC with a custom authenticator, that maps (user+passw) -> bool? We could probably still use LDAP directly for authorisation. If not, though - is a similar class extension possible for authorisation as well? If I understood correctly, this would need to be done inside the Metadata Service. - An alternative could be to run a separate LDAP server, identical to our company-central one, but where the user passwords are set to their API keys. That would be quite a faff, though. If this is not possible - what is it that makes our use case different from the norm? Is it that, - generally people are OK with passing LDAP passwords? - generally "external" users don't need the schema registry or REST proxy (i.e. they are only used by internal microservices), so they can be firewalled off? - generally all enterprise consumers are on the JVM, making Kerberos auth more plausible than in our case? - ... ? Note that I do not have a ton of experience with securing Kafka, so the above may well contain numerous mistaken/out-of-date assumptions. In any case, thanks for any suggestions. Best, -Joris.
