wangzhigang1999 opened a new issue, #7529: URL: https://github.com/apache/kyuubi/issues/7529
### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues. ### Describe the feature Umbrella tracking the security-hardening and isolation follow-ups deferred while landing the Data Agent engine ([#7379](https://github.com/apache/kyuubi/issues/7379), KPIP-[7373](https://github.com/apache/kyuubi/discussions/7373)). These were explicitly punted during review of the 1/4–4/4 series (last PR #7431) so the feature could land. Sub-tasks are described inline below; they are not split into separate issues for now. **Root cause that ties most of these together.** `DataAgentProcessBuilder extends ProcBuilder`, so the engine is launched as a **local child process on the Kyuubi server host** — there is no k8s/YARN remote-launch path today. The engine therefore shares the server's ambient environment: its Kerberos credential cache (the proxy super-user TGT), its network identity, and its filesystem. That sharing is exactly why Kerberos is currently banned and why credential handling is constrained. Giving the engine its own isolated runtime with its own identity is the keystone that unblocks the rest. ### Sub-tasks - [ ] **① Isolated runtime with its own identity (keystone).** Give `DATA_AGENT` a remote/isolated launch path so the engine runs in its own container with a distinct identity instead of as a local child of the server. Kubernetes pod **first**; YARN deferred as a later step (don't bundle it). Includes an engine image (or Kyuubi image + data-agent classpath), wiring memory / java-options / extra-classpath into the remote spec, liveness/readiness + log retrieval, and keeping the local-process path as a dev/standalone fallback. Non-trivial — effectively brings a new engine type to parity on remote submission/imaging/probing; expect to split the implementation into (a) k8s launch + image, (b) lifecycle/probe/log wiring, (c) YARN (later). **Unblocks ② and ③.** - [ ] **② Kerberos support via the engine's own login + proxyuser impersonation** *(depends on ①)*. Lift the temporary Kerberos ban and replace it with real impersonation: the engine logs in with its own principal/keytab (no reliance on the server's ticket cache) and proxies the session user via `hadoop.proxyuser` ACLs, then re-enables the ZK-service-discovery and direct back-connection URLs under Kerberos. Two standing code TODOs mark the sites to remove: `kyuubi-server/.../engine/EngineRef.scala` (`deriveDataAgentJdbcUrl`, fails closed when Kerberos is enabled) and `externals/kyuubi-data-agent-engine/.../provider/chatcompletion/ChatCompletionProvider.java` (rejects a Kerberos JDBC URL). Add regression coverage for the impersonation path. - [ ] **③ Credential propagation without embedding secrets in the URL** *(design depends on ①)*. Let users supply datasource credentials via dedicated config and/or UI inputs with an explicit `propagateCredential` opt-in, instead of stuffing `user`/`password` into `kyuubi.engine.data.agent.jdbc.url` (which forces the UI to sanitize them on every surface). Raised in #7431 review. Align with the same shape elsewhere: the JDBC engine's `propagateCredential` is binary, and Trino's `connection.user` silently falls back to `sessionUser`. Keep new credential configs redacted in logs/`toString`. - [ ] **④ Separate (cheaper) summarization model for context compaction** *(independent)*. Let `CompactionMiddleware` use a summarization model distinct from the main agent model, falling back to the main model when unset. Marked by a TODO at `externals/kyuubi-data-agent-engine/.../runtime/middleware/CompactionMiddleware.java`. Low-risk; can proceed any time. - [ ] **⑤ Proper SQL read-only check for `run_select_query`** — tracked separately in #7412 *(orthogonal — SQL-safety guardrail, not isolation/identity)*. Linked here for visibility only. **Dependency order.** ① is the critical path; ② and ③ build on it; ④ and #7412 are independent and can proceed in parallel at any time. ### Motivation The Data Agent engine shipped with two deliberate, documented restrictions — Kerberos is rejected on both reachable paths, and JDBC credentials must be supplied in the URL — both rooted in the engine running co-located with the server under the server's credentials. There are also standing code TODOs marking the Kerberos sites. Tracking these in one umbrella keeps the deferred hardening visible and lets the container-isolation work be sequenced ahead of the items that depend on it. ### Describe the solution See the inline sub-tasks. Intended sequence: isolate the engine runtime and give it its own identity (①), then enable real Kerberos impersonation (②) and a safer credential-propagation model (③) on top of that identity, while ④ and #7412 proceed independently. ### Additional context Origin: Data Agent engine series under #7379 (PRs 1/4–4/4, last #7431). Standing code TODOs: `EngineRef.scala` (`deriveDataAgentJdbcUrl`) and `ChatCompletionProvider.java`; plus the `CompactionMiddleware` summarization-model TODO. ### Are you willing to submit PR? - [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
