GitHub user jueshali added a comment to the discussion: [FEATURE]Support a
configuration-driven query optimization runtime for Kyuubi Spark SQL extensions
- For ai
# Example
## Original SQL
```sql
INSERT OVERWRITE TABLE dws_user_pay_day PARTITION(dt='20260714')
SELECT
a.user_id,
a.game,
SUM(b.pay_amount) AS total_pay
FROM ods_user_login_day a
JOIN dwd_pay_detail_day b
ON a.user_id = b.user_id
WHERE a.dt = '20260714'
AND b.dt = '20260714'
GROUP BY a.user_id, a.game
```
## Upstream analysis
Historical statistics show:
- duration: `1200s`
- shuffle: `95GB`
- spill disk: `42GB`
- diagnoses: `TASK_SKEW`, `BIG_SPILL_DISK`
AI suggests:
- rebalance before write
- enable skew join
- tune shuffle partitions
- isolate final write stage resources
- send to dual-run validation
## Optimization intent
```json
{
"version": 1,
"source": "AI",
"traceId": "trace-20260714-001",
"dualRunId": "dual-20260714-abc",
"writeStrategy": {
"mode": "rebalance"
},
"runtimeStrategy": {
"shufflePartitions": 128,
"advisoryPartitionSizeInBytes": "64MB",
"skewJoinEnabled": true
},
"resourceStrategy": {
"finalStageResourceIsolationEnabled": true,
"finalStageExecutorCores": 2,
"finalStageExecutorMemory": "4g"
}
}
```
## What Kyuubi does
1. Parse the optimization plan into an internal `OptimizationPlan`
2. Apply runtime configs:
- `spark.sql.shuffle.partitions=128`
- `spark.sql.adaptive.advisoryPartitionSizeInBytes=64MB`
- `spark.sql.adaptive.skewJoin.enabled=true`
3. Apply write optimization:
- rebalance before write
4. Apply final-stage optimization:
- final-stage resource isolation
- custom resource profile
## Execution trace
```json
{
"traceId": "trace-20260714-001",
"dualRunId": "dual-20260714-abc",
"source": "AI",
"appliedActions": [
{"layer": "session-conf", "action": "set shuffle partitions to 128"},
{"layer": "session-conf", "action": "enable skew join"},
{"layer": "post-hoc-resolution", "action": "apply write rebalance"},
{"layer": "query-stage-prep", "action": "enable final-stage resource
isolation"}
],
"partialApplied": false
}
```
## Dual-run result
```json
{
"dualRunId": "dual-20260714-abc",
"resultValidation": "PASS",
"durationImprovement": "18%",
"shuffleReduction": "12%",
"spillDiskReduction": "35%",
"recommendation": "PROMOTE"
}
```
## Summary
- historical statistics + AI generate optimization intent
- Kyuubi executes that intent
- trace records what was applied
- dual-run validates correctness and benefit
GitHub link:
https://github.com/apache/kyuubi/discussions/7563#discussioncomment-17657583
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
