I am working with a 3rd party kafka provider and realized that I could use
the projected service account token that EKS (AWS's Elastic Kubernetes
Service) automatically mounts in a pod with a specified service account (a
JWT managed by the AWS OIDC provider
<https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html>),
to authenticate my services to the Kafka cluster. It seemed like a pretty
great solution for short-lived auto-rotated auth tokens (yay security).
This is of course after I registered the OIDC provider with our Kafka
service provider. I noticed kafka added this OIDC support
<https://github.com/apache/kafka/commit/7b379539a5533179b0c86adb352548ac1aa82006>
along with a FileTokenRetriever in the Kafka java client some time back.
The AccessTokenRetriverFactory
<https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internals/secured/AccessTokenRetrieverFactory.java#L63-L64>
creates
a FileTokenRetriever if the token endpoint URL is a filepath, e.g.
file://some/path/to/token.

So I created a service-account in kubernetes, and EKS mounts an
auto-rotated token at /var/run/secrets/
eks.amazonaws.com/serviceaccount/token, which I can then use as the
filepath for the token retriever, however, the default implementation only
reads the file contents once, so after some time, the token will go stale
and as far as I can tell, is never re-read.

I then wrote my own implementation of AuthenticateCallbackHandler that is
almost the same as the OAuthBearerLoginCallbackHandler but replaces the use
of the token retriever factory with my own RefreshingFileTokenRetriever
that reads/caches/re-reads the specified file.

I am wondering if anyone else has this use-case, basically having workload
identities that could have auto-rotated credentials mounted as files that
could be periodically read in and used as the OIDC token. This seems like
it's already a pattern used by the kubernetes client itself:

- k8s client BearerAuthRoundTripper
<https://github.com/kubernetes/client-go/blob/da6e2946e51b907f353947a24a91ae91393df734/transport/round_trippers.go#L290-L309>
- example of datadog-agent using it
<https://github.com/goodspark/datadog-agent/blob/75af92223f9d582e6d8b36fea6624a18a7477160/pkg/util/kubernetes/kubelet/kubelet_client.go#L91-L94>.
This is authenticating with the kubernetes API, of course, but as far as I
can tell, it's using the same kind of mechanism any OIDC provider/consumer
would use, right?

Anyway, I am not a whiz when it comes to oauth really. I'm a bit new to it.
However, when I figured out I could have a form of short-lived credentials,
that my application didn't necessarily have to manage the rotation of, and
getting all the benefits of a "workload identity" role, rather than doling
out long-lived key/secrets for each service with different scopes, etc. I
thought this was amazing and would be a great benefit for the Kafka client.

I don't think this would be very difficult to implement. And there is prior
art in the form of the kubernetes client itself. Would this be something
worth adding as an internal component?

Also, am I even thinking about this the right way? Again, I have little
experience in OIDC, etc. I have used AWS IAM for a long time though, and
want the same role-based workload identity. This seemed to work well so
far, but I'm really not even sure it's the "correct" way to do things. I
could only imagine the k8s api is pretty well vetted. Anyway, Would love
any feedback or linking me to someone with good OIDC knowledge. For now
I'll be plumbing in static Vault secrets that aren't rotated regularly 😬

Thanks,

- Creed Haymond

Reply via email to