GitHub user oh0873 created a discussion: Server/engine version coupling after #7374 (Kyuubi 1.12) (InternalSecurityAccessor IV change)
### Background We run Kyuubi with internal security enabled (`kyuubi.internal.security.enabled=true`, formerly `kyuubi.engine.security.enabled`). In our deployment, the Kyuubi server and the Spark engine are upgraded on different cadences: - **Kyuubi server** — deployed as a standalone service (we are moving to 1.12.0) - **Spark engine** — `kyuubi-spark-sql-engine_2.12` is baked into our Spark container images, which lag the server upgrade cycle ### Problem After upgrading the Kyuubi server to **1.12.0**, sessions fail when the engine still runs an older `kyuubi-spark-sql-engine` jar (1.11.1 or earlier) from the Spark image. We traced this to [#7374](https://github.com/apache/kyuubi/pull/7374), which changed the on-the-wire format of internal access tokens in `InternalSecurityAccessor`: | | Before (#7374) | After (#7374) | |---|---|---| | IV | Static all-zero IV, fixed at cipher init | Random IV per encryption | | Token payload | `hex(ciphertext)` | `hex(IV \|\| ciphertext)` | The server issues tokens via `InternalSecurityAccessor.issueToken()` when connecting to an engine ([`KyuubiSessionImpl`](https://github.com/apache/kyuubi/blob/master/kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiSessionImpl.scala)). The engine validates them via `EngineSecureAuthenticationProviderImpl` using the same class from `kyuubi-common`. Because the encrypt/decrypt wire format changed, a **1.12 server cannot authenticate to an engine built from a pre-#7374 `kyuubi-common` jar**, even when both sides share the same secret and cipher configuration. We understand the security motivation for the IV fix. The operational impact is that this change effectively requires **lockstep upgrades of server and engine jars** whenever internal security is enabled — which is also required for REST API HA mode per [internal secure access docs](https://kyuubi.readthedocs.io/en/master/security/internal_secure_access.html). ### Impact on us - We cannot roll out a Kyuubi server upgrade without simultaneously rebuilding and redeploying every Spark image that carries the engine jar. - For large fleets, that coupling makes minor/patch server upgrades costly and slow. - We did not find this requirement documented in upgrade or compatibility guidance. ### Questions 1. **What is the recommended rollout strategy** for operators who decouple server and Spark image lifecycles? For example: upgrade all engine images first, then server — or the reverse? 2. **Would the project consider a transitional compatibility approach**, such as: - decrypt-time fallback (try IV-prefixed format, then legacy zero-IV format), or - a config flag to opt into legacy token format during a rolling migration window? /cc @aajisaka @turboFei GitHub link: https://github.com/apache/kyuubi/discussions/7745 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
