turboFei commented on issue #7511: URL: https://github.com/apache/kyuubi/issues/7511#issuecomment-5201150473
Hi @alexandrefimov, Please treat #7510 as background material only, rather than an agreed design baseline. Feel free to lead the design and use #7510 and #6642 as references where useful. There is now an upstream Spark proposal that we should evaluate before implementing a Kyuubi-specific gateway: [SPARK-58455: Spark Connect Gateway — Session Affinity, Auth, and Multi-Tenancy for Spark Connect Servers](https://issues.apache.org/jira/browse/SPARK-58455). The full [SPIP document](https://docs.google.com/document/d/16RNnRJjSDBSlcgeBjVz_Mv-SkqvH33F9tIsR1Ii_P3c/edit?usp=sharing) proposes an official standalone, wire-compatible gateway covering session affinity, operation reattach routing, authentication, multi-tenancy, rate limiting, auditing, backend discovery, and HA. This overlaps substantially with the thin-proxy responsibilities proposed in #7511. One important boundary is that the upstream gateway explicitly does not manage Spark driver lifecycle: it consumes an existing pool of Spark Connect servers and delegates provisioning and scaling to an external system. Therefore, I suggest that the design first evaluate how Kyuubi should integrate with the upstream gateway proposal. A possible boundary is: - The Spark Connect Gateway owns protocol-aware routing, session affinity, authentication, and reattach routing. - Spark Connect servers own `SparkSession`, operation state, plan execution, and plan-level authorization. - Kyuubi owns engine provisioning and lifecycle, tenant resource policies, engine discovery, and integration with the rest of the Kyuubi ecosystem. If a native Kyuubi gateway is still needed, the design should explain what it provides beyond SPARK-58455 and how we can avoid maintaining duplicate implementations of Spark Connect routing and protocol semantics. Since SPARK-58455 is still an open SPIP, the final upstream direction is not settled yet. Please feel free to lead this investigation and propose the Kyuubi-side design. It would also be useful to participate in the upstream discussion so the two designs can evolve together. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
