dengc367 commented on issue #14308:
URL: https://github.com/apache/gravitino/issues/14308#issuecomment-6514871228
Thanks @mchades. Concretely, using the model from the design doc
(`sales_model`: datasets `orders` and `regions`, metric `Gross_Revenue`,
relationship `orders_to_regions`):
**The agent query.** An agent is asked "revenue by region for October 2026?"
and turns it into:
```json
{"metrics": ["Gross_Revenue"],
"dimensions": ["Region_Name", "Calendar_Month"],
"filters": [{"field": "Calendar_Month", "operator": "EQ", "value":
"2026-10"}],
"limit": 1000}
```
**How it is handled today, with our planner service:**
1. Agent -> planner service (REST).
2. Planner service -> catalog: read the model's current definition -
datasets, relationships, and the per-dialect expressions of the three fields.
3. Resolve on the client side: `Gross_Revenue` -> dataset `orders` +
`SUM(orders.amount)`; `Region_Name` -> `regions.name`; join path `orders ->
regions` on `region_id = id`; aggregate vs dimension; `GROUP BY` =
`Region_Name, Calendar_Month`.
4. Render: assign aliases, quote identifiers, render the literal
`'2026-10'`, assemble Spark SQL. A Trino caller needs a second renderer.
5. Submit to Spark/Trino, return rows.
6. Authorization: the model read is only checked against the caller if the
service forwards the caller's identity; otherwise the service account's
privileges apply.
**The gap is steps 2-3, not 4-5.** Steps 2-3 are model knowledge and
privilege, and every consumer needs exactly the same answers: join path,
metric-to-dataset ownership, aggregate classification, grouping, and the errors
for ambiguous or unjoinable queries. Our agent platform gets them right because
it goes through the planner service. Gravitino's Spark/Trino connectors and any
BI tool have no equivalent, so each one either duplicates this or stays
unsupported - which is how two consumers end up returning different numbers for
the same question. It is also why resolution belongs next to the model and next
to `USE_SEMANTIC_MODEL`.
**What Gravitino returns** for that request (no dialect, no SQL):
```json
{"sources": [{"dataset": "orders", "source": ["catalogA", "dbB",
"order_table"]},
{"dataset": "regions", "source": ["catalogA", "dbB",
"region_table"]}],
"joins": [{"relationship": "orders_to_regions", "type": "INNER"}],
"selections": [{"name": "Gross_Revenue", "role": "METRIC", "aggregate":
true,
"field": {"dataset": "orders",
"expression": {"ANSI_SQL":
"SUM(orders.amount)"}}}],
"groupBy": ["Region_Name", "Calendar_Month"], "limit": 1000,
"semanticModelVersion": 3}
```
**What stays with our planner service, deliberately:**
| Responsibility | Owner after the change |
| --- | --- |
| expression dialect selection, quoting, literals, aliases, statement shape
| planner service |
| join reordering, pushdown, CBO | planner service / its engine |
| submission, execution, result handling, pagination | planner service |
| plan/result caching, cost guardrails | planner service |
| model resolution, join graph, aggregation classification, authorization,
version | Gravitino |
Gravitino resolves; it never renders or executes. The conversion is
mechanical because every decision is already in the plan. Our prototype (a
Spark planner against the old SQL-text SPI, 2.3k lines / 43 tests) spends most
of its code on steps 2-3 plus dialect rendering - the design removes the first
half and leaves the second half with the engine.
**Why not keep it inside the planner service only?** For a single consumer
it works, and that is exactly why we built one. The cost is that the contract
is private: a new consumer either depends on our service or forks the resolver,
and a consumer that already has its own catalog connection (spark-connector
does) still has to route resolution through a second hop. Standardizing the
seam lets each engine keep its renderer while the duplicated part disappears.
If the community prefers Gravitino to stay a document registry, that is a
legitimate call - but then every connector re-implements steps 2-3.
Where I would not argue for it: a deployment with one consumer and one
engine. The case here is several consumers (agents, BI, engine connectors),
more than one engine, and one governed model.
Full contract and the deterministic resolution rules:
https://github.com/dengc367/gravitino/blob/dc1397478dee242bae743d5b3c088c1df42eef2c/design-docs/gravitino-semantic-query-plan-contract.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
