Hello Kafka community,
I've encountered an issue with OAuth authentication in Kafka when running
on a system that goes to sleep/hibernates. I believe I've identified a flaw
in the token refresh mechanism that affects reliability in certain
environments.
When using OAuth authentication between brokers and controllers, the token
refresh mechanism fails after system sleep/hibernation, causing all
authentication to fail until the service is restarted.
I observed this on my Confluent Platform setup running on a MacBook:
- OAuth token was set to refresh at 18:31:29
- System went to sleep at 18:19
- System woke up at 18:53, after the tokens had expired at 18:42
- No refresh login attempt occurred after wakeup
- All authentication failed with expired tokens
After reviewing the ExpiringCredentialRefreshingLogin class code, I can see
the issue stems from how the refresh thread sleeps until the next scheduled
refresh time:
log.info("[Principal={}]: Expiring credential re-login sleeping until:
{}", principalLogText(),
new Date(nextRefreshMs));
time.sleep(nextRefreshMs - nowMs);
When the system goes to sleep, this thread's execution is suspended. Upon
waking, the thread simply continues its sleep operation without any
awareness that a significant amount of time may have passed. There's no
mechanism to detect that the planned refresh window has been missed due to
system suspension.
I understand that hibernating a Kafka cluster isn't a common production
scenario, and this issue might not affect many users in production
environments. However, I believe this vulnerability in the token refresh
mechanism could be problematic in certain scenarios like development
environments, containerized setups, or any situation where process
suspension might occur.
Do you consider this behavior a bug that should be addressed? And would you
recommend creating a KIP for this issue?
I'm asking because while this might be a niche case for production, the
authentication failure is particularly frustrating in development
environments as it requires a full cluster restart to resolve.
Thanks for your help,
Adrien
