GitHub user ktricky9 created a discussion: Cluster-level admission control & query queuing in Kyuubi (OperationManager hook approach)
Hi everyone, I’m exploring approaches to introduce cluster-level admission control in Kyuubi for multi-tenant workloads (BI + batch), and wanted to validate the approach and learn from existing production practices. ## Background As far as I understand, Kyuubi currently acts as a SQL gateway that forwards queries to engines immediately. While it provides strong multi-tenancy and engine isolation, it does not seem to have: - global concurrency limits - query admission control - built-in queuing mechanism In bursty workloads (e.g., Tableau dashboards or Airflow jobs), this can lead to: - driver/executor overload - degraded latency for all queries - lack of fairness across tenants ## Current Approach We are experimenting with admission control at the operation layer: - Intercept at `KyuubiOperationManager.newExecuteStatementOperation()` - Wrap `ExecuteStatement` - Acquire a global token (via external store like Redis) before execution - Release token in `afterRun()` This allows enforcing a cluster-wide concurrency limit before queries reach the engine. ## Open Questions 1. How are others handling admission control today? - Spark scheduler (FAIR pools / queues)? - Engine isolation (per user/group)? - External gateway or orchestration layer? - Any custom Kyuubi extensions or patches? 2. Query behavior under saturation: - Current implementation is **fail-fast** (reject when limit reached) - However, for BI workloads, **queueing (blocking execute)** seems more appropriate Has anyone implemented: - query queuing while preserving JDBC semantics? - where clients wait instead of fail? 3. Extension points: - Is there any plan or discussion around making `OperationManager` pluggable via config? - Or introducing a pre-execution admission hook in Kyuubi? ## Goal Ultimately trying to move towards more warehouse-like behavior: - controlled concurrency - fairness across workloads - stable performance under load Would appreciate any insights, prior art, or guidance from folks running Kyuubi at scale. Thanks! GitHub link: https://github.com/apache/kyuubi/discussions/7405 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
