CefBoud opened a new pull request, #17987: URL: https://github.com/apache/kafka/pull/17987
# Introduction This PR proposes a change to add support for SSL hot reloading. # Motivation SSL certificates are typically short-lived, and while Kafka brokers support dynamic SSL certificate reloads via dynamic configuration, other components still require disruptive restarts to apply updated certificates. This change aims to eliminate such disruptions by allowing SSL certificates to be reloaded dynamically when changes occur. The goal is to improve system uptime and reduce operational overhead. # Description Inspired by [Spring Boot's SSL hot reload feature](https://docs.spring.io/spring-boot/reference/features/ssl.html#features.ssl.reloading), this implementation triggers an `SslFactory` reconfiguration whenever the keystore or truststore files are modified. Java's `WatchService` API from `java.nio` is used for file changes monitoring. A new **`ssl.hot.reload` configuration** is introduced. It can be used by any SslClient (brokers, consumers, producers). All `SslFactories` within a single JVM share the same `WatcherService`. To prevent redundant reconfigurations, a `Quiet Period` is enforced across all watched files, ensuring that related changes do not trigger multiple reconfigurations. # Testing Instruction 1. Generate a keystore and truststore. 2. Append the following to `config/kraft/reconfig-server.properties` : ``` security.protocol=SSL ssl.key.password=<password> ssl.keystore.location=/path/to/keystore.p12 ssl.keystore.password=<password> ssl.truststore.location=/path/to/ca.p12 ssl.truststore.password=<password> # new option ssl.hot.reload=true ``` 3. Build the code in this branch `./gradlew jar`. 4. Start the broker `bin/kafka-server-start.sh config/kraft/reconfig-server.properties`. 5. Replace the `PLAINTEXT` listener with `SSL` then restart the broker. ``` listeners=SSL://:9092,CONTROLLER://:9093 inter.broker.listener.name=SSL advertised.listeners=CONTROLLER://localhost:9093,SSL://localhost:9092 ``` 6. Create a `producer.properties` file: ``` security.protocol=SSL ssl.key.password=<password> ssl.keystore.location=/path/to/keystore.p12 ssl.keystore.password=<password> ssl.truststore.location=/path/to/ca.p12 ssl.truststore.password=<password> ssl.endpoint.identification.algorithm= # new option ssl.hot.reload=true ``` 7. let's increase logging level for `tool` to `INFO` so changes are reflected in logs: ``` sed -i 's/WARN/INFO/g' ./config/tools-log4j.properties ``` 8. Start a console producer: ``` ./bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test1 --producer.config path/to/producer.properties ``` 9. Let's simulate a certificate renewal by moving the keystore to a different location then bringing it back. ``` mv /path/to/keystore.p12 /other/path/to/keystore.p12 && mv /other/path/to/keystore.p12 /path/to/keystore.p12``` After 30 seconds Producer logs: ``` [2024-11-29 12:39:47,712] INFO Created new CLIENT SSL engine builder with keystore java.security.KeyStore@3f7ebffe truststore java.security.KeyStore@3a500477 (org.apache.kafka.common.security.ssl.SslFactory) ``` Broker logs: ``` [2024-11-29 12:39:47,392] INFO Created new SERVER SSL engine builder with keystore java.security.KeyStore@385167fc truststore java.security.KeyStore@16ebfb9 (org.apache.kafka.common.security.ssl.SslFactory) [2024-11-29 12:39:47,412] INFO Created new CLIENT SSL engine builder with keystore java.security.KeyStore@1fb5e6a0 truststore java.security.KeyStore@14181670 (org.apache.kafka.common.security.ssl.SslFactory) [2024-11-29 12:39:47,444] INFO Created new SERVER SSL engine builder with keystore java.security.KeyStore@2d931fe9 truststore java.security.KeyStore@4924c819 (org.apache.kafka.common.security.ssl.SslFactory) [2024-11-29 12:39:47,461] INFO Created new CLIENT SSL engine builder with keystore java.security.KeyStore@5400441c truststore java.security.KeyStore@62c51d22 (org.apache.kafka.common.security.ssl.SslFactory) [2024-11-29 12:39:47,486] INFO Created new SERVER SSL engine builder with keystore java.security.KeyStore@3897543e truststore java.security.KeyStore@6e61d794 (org.apache.kafka.common.security.ssl.SslFactory) ``` ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
