mchades commented on code in PR #14038: URL: https://github.com/apache/gravitino/pull/14038#discussion_r4375999659
########## design-docs/gravitino-schema-metrics-discovery-design.md: ########## @@ -0,0 +1,407 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Schema-level Semantic Model Metrics Discovery in Gravitino + +## Background + +Metrics in Gravitino are defined within Semantic Models. To browse or select metrics across a +schema, clients currently list the Semantic Models, load each model, and extract its metrics. + +This requires a separate request to load each model and returns the full definitions, including +datasets and relationships that metric discovery does not need. Clients must also implement the +collection logic themselves. The number of requests and volume of metadata transferred increase +as more models are added or their definitions grow. + +## Goals + +1. **Discovery.** Provide schema-level listing of metric identifiers and complete definitions through + REST, Java, Python, and MCP, without requiring clients to load each Semantic Model separately. +2. **Identity.** Keep internal identities stable for metrics retained during Semantic Model updates. +3. **Consistency.** Keep discovered metric definitions consistent with the current committed + definition of each Semantic Model. +4. **Visibility.** Apply the same visibility rules as the existing Semantic Model listing. +5. **Integration.** Both listing modes emit pre, success, and failure events through Gravitino's + existing listener framework. + +## Non-Goals + +1. **Independent metric management.** Standalone APIs for creating, updating, deleting, or renaming + metrics; metric-level ownership and privileges. +2. **Public identity and history.** Exposing internal metric IDs or revision identifiers, or querying + historical metric definitions. + +## Proposal + +### API Contract + +#### Java API + +The `SemanticModelCatalog` interface adds separate methods for identifiers and complete definitions: + +```java +default NameIdentifier[] listMetrics(Namespace namespace) throws NoSuchSchemaException { + throw new UnsupportedOperationException("Metric listing is not supported"); +} + +default SemanticModelMetric[] listMetricInfos(Namespace namespace) throws NoSuchSchemaException { + throw new UnsupportedOperationException("Metric info listing is not supported"); +} +``` + +The built-in implementation and Java client override these defaults to support metric discovery. + +`listMetrics` returns one identifier per metric. A full identifier has a four-level namespace +`[metalake, catalog, schema, semanticModel]` and the metric name as its `name`. Including the containing +model distinguishes metrics with the same name in different models. Catalog-scoped clients follow +the existing convention: callers pass `Namespace.of(schema)`, and returned identifiers contain the +namespace `[schema, semanticModel]` and the metric name. REST uses full identifiers. + +`listMetricInfos` returns one `SemanticModelMetric` per metric: + +```java +/** A metric's namespace and complete definition returned by discovery. */ +public interface SemanticModelMetric { + + /** + * Returns the metric namespace, whose last level is the containing Semantic Model. + * + * @return The metric namespace. + */ + Namespace namespace(); + + /** + * Returns the complete metric definition. + * + * @return The metric definition. + */ + Metric metric(); +} +``` + +`namespace()` uses the same full or catalog-relative namespace as the identifiers from `listMetrics`. +In this response type, the metric name is stored only in the existing `Metric` definition. +Callers can reconstruct its identifier with `NameIdentifier.of(info.namespace(), info.metric().name())`. +`SemanticModelMetricDTO` contains a `namespace` array and an unchanged `MetricDTO` in `metric`. + +#### REST API + +The schema-level discovery route sits alongside `semantic-models`: + +```http +GET /api/metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/semantic-model-metrics +``` + +The `details` query parameter defaults to `false`. With `details=false`, the endpoint implements +`listMetrics` and returns the existing `EntityListResponse`: + +```json +{ + "code": 0, + "identifiers": [ + { + "namespace": ["example", "sales", "mart", "sales_model"], + "name": "total_revenue" + } + ] +} +``` + +With `details=true`, it implements `listMetricInfos` and returns a `MetricInfoListResponse` containing +a `metrics` array of `SemanticModelMetricDTO` objects: + +```json +{ + "code": 0, + "metrics": [ + { + "namespace": ["example", "sales", "mart", "sales_model"], Review Comment: `listTables` returns `[metalake, catalog, schema]` as the namespace and the table name as `name`. Here, `name` is the metric name, so the namespace includes its containing model: `[metalake, catalog, schema, semanticModel]`. This distinguishes same-named metrics in different models. The listing request remains schema-scoped and does not require a model name. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
