wangzhigang1999 commented on PR #7417: URL: https://github.com/apache/kyuubi/pull/7417#issuecomment-4301550808
# Evidence: runtime under real workload > Supplementary evidence for this PR. The benchmark harness itself is kept on a local branch (`pr2c/...`) and is **not** part of this PR — the numbers below are cited only to show that the runtime shipped here behaves correctly under a non-trivial text-to-SQL workload. ## TL;DR - **End-to-end robustness.** 500/500 BIRD mini-dev questions completed, `gen_ok=100%`, zero framework-level failures (no crashes, no stuck sessions, no rate-limit fallout at concurrency=8). - Accuracy (EX=64.80%, F1=68.72% with `kimi-k2.5`) is included as context — this PR is not a model or accuracy deliverable. ## Setup | | | |---|---| | **Runtime under test** | `ReactAgent` + `ToolResultOffloadMiddleware` + `CompactionMiddleware` (this PR) | | **Tools** | `submit_sql`, `run_select_query`; `max_iterations=30` | | **Model** | `kimi-k2.5` via dashscope OpenAI-compatible endpoint | | **Workload** | BIRD mini-dev, 500 questions / 11 SQLite databases | | **Harness** | internal, on local branch `pr2c/...` (not submitted) | | **Concurrency** | 8 | | **Run** | `benchmark-out/20260423-012636/` | ## Results **Overall**: 500 questions, `gen_ok=100%`, EX=**64.80%**, F1=68.72%, avg 8.9 steps / 43.6k tokens / 42s per question. **By difficulty**: | Difficulty | N | EX | F1 | avg steps | avg tokens | |---|---:|---:|---:|---:|---:| | simple | 148 | 75.00% | 78.33% | 7.6 | 33k | | moderate | 250 | 62.80% | 66.99% | 9.1 | 46k | | challenging | 102 | 54.90% | 59.01% | 10.2 | 53k | **By database** (sorted by EX): | DB | N | EX | F1 | |---|---:|---:|---:| | superhero | 52 | 84.62% | 89.35% | | european_football_2 | 51 | 76.47% | 79.48% | | student_club | 48 | 72.92% | 74.44% | | toxicology | 40 | 72.50% | 73.68% | | codebase_community | 49 | 69.39% | 74.17% | | card_games | 52 | 63.46% | 58.27% | | debit_card_specializing | 30 | 60.00% | 62.78% | | formula_1 | 66 | 54.55% | 62.13% | | thrombosis_prediction | 50 | 54.00% | 59.60% | | financial | 32 | 50.00% | 58.79% | | california_schools | 30 | 43.33% | 54.29% | Cost: ~45 min wall time at concurrency=8, ~21M tokens total. ## What this evidence supports - The runtime completes 500 non-trivial ReAct loops end-to-end without framework-level errors, at concurrency=8, against a live OpenAI-compatible provider. That's the load-bearing claim for this PR. - Difficulty scaling is monotonic (75% → 63% → 55%) — no pathological inversion that would suggest the runtime mishandles longer loops. ## Scope disclaimers - **Not a model benchmark.** Numbers are bound to `kimi-k2.5`; other models will move them. - **Not a leaderboard submission.** Vanilla ReAct with the stock prompt; no schema-ranking, few-shot, or self-consistency. This is a framework viability floor, not SOTA. - **Not a regression gate.** LLM non-determinism is ±1pp run-to-run. ## Follow-up A second run on a Spark execution backend (same harness, same workload, different datasource) is planned as additional validation that the runtime is decoupled from the SQLite-specific path. Results will be appended here when available. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
