This is an automated email from the ASF dual-hosted git repository.
jerryshao pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new ed23d2043f [#12604] feat(core): Add Semantic Model create and load
operations (#12831)
ed23d2043f is described below
commit ed23d2043f3c2498b50daa75578170fd00569862
Author: mchades <[email protected]>
AuthorDate: Thu Sep 24 18:37:40 2026 +0800
[#12604] feat(core): Add Semantic Model create and load operations (#12831)
### What changes were proposed in this pull request?
Add managed Core operations for creating and loading Semantic Models:
- Validate Semantic Model definitions with Java-native,
catalog-independent validation.
- Leave catalog-backed source existence, column, and authorization
checks to the caller-facing authorization layer.
- Build Semantic Model entity and audit metadata and persist them
through the entity store.
- Return persisted Semantic Models on load.
- Add focused unit and JDBC tests.
### Why are the changes needed?
Gravitino Core needs managed create and load behavior for Semantic
Models, including definition validation, metadata construction,
persistence, and dispatcher integration. Source metadata access must be
validated together with the caller's authorization context rather than
through internal catalog dispatchers.
Fix: #12604
### Does this PR introduce _any_ user-facing change?
Yes. It implements the existing Semantic Model create and load
operations in Gravitino Core. No new public API, REST, OpenAPI, or
client surface is introduced.
### How was this patch tested?
- `./gradlew :core:spotlessCheck :core:test --tests
org.apache.gravitino.catalog.TestSemanticModelValidator --tests
org.apache.gravitino.catalog.TestSemanticModelOperationDispatcher
--tests org.apache.gravitino.catalog.TestManagedSemanticModelOperations
-PskipITs`
- `./gradlew :core:check -PskipITs` — 2,356 tests, 0 failures, 0 errors,
2 skipped
- `git diff --check origin/main..HEAD`
---
.../catalog/ManagedSemanticModelOperations.java | 86 ++++++--
.../catalog/SemanticModelOperationDispatcher.java | 7 +-
.../gravitino/catalog/SemanticModelValidator.java | 156 +++++++++++++
.../TestManagedSemanticModelOperations.java | 191 +++++++++++++++-
.../TestManagedSemanticModelOperationsJDBC.java | 125 +++++++++++
.../TestSemanticModelOperationDispatcher.java | 148 +++++++------
.../catalog/TestSemanticModelValidator.java | 241 +++++++++++++++++++++
design-docs/gravitino-semantic-model-design.md | 49 ++---
8 files changed, 871 insertions(+), 132 deletions(-)
diff --git
a/core/src/main/java/org/apache/gravitino/catalog/ManagedSemanticModelOperations.java
b/core/src/main/java/org/apache/gravitino/catalog/ManagedSemanticModelOperations.java
index de6b1f819d..4fc4491797 100644
---
a/core/src/main/java/org/apache/gravitino/catalog/ManagedSemanticModelOperations.java
+++
b/core/src/main/java/org/apache/gravitino/catalog/ManagedSemanticModelOperations.java
@@ -18,58 +18,73 @@
*/
package org.apache.gravitino.catalog;
+import com.google.common.base.Preconditions;
+import java.io.IOException;
+import java.time.Instant;
import java.util.Map;
+import java.util.function.BiConsumer;
import javax.annotation.Nullable;
+import org.apache.gravitino.Entity;
+import org.apache.gravitino.EntityAlreadyExistsException;
import org.apache.gravitino.EntityStore;
import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.Namespace;
import org.apache.gravitino.exceptions.IllegalSemanticModelException;
+import org.apache.gravitino.exceptions.NoSuchEntityException;
import org.apache.gravitino.exceptions.NoSuchSchemaException;
import org.apache.gravitino.exceptions.NoSuchSemanticModelException;
import org.apache.gravitino.exceptions.SemanticModelAlreadyExistsException;
+import org.apache.gravitino.meta.AuditInfo;
+import org.apache.gravitino.meta.SemanticModelEntity;
import org.apache.gravitino.semantic.SemanticModel;
import org.apache.gravitino.semantic.SemanticModelCatalog;
import org.apache.gravitino.semantic.SemanticModelChange;
import org.apache.gravitino.semantic.SemanticModelDefinition;
import org.apache.gravitino.storage.IdGenerator;
+import org.apache.gravitino.utils.PrincipalUtils;
-/**
- * Provides storage-level Semantic Model operations backed by Gravitino's
{@link EntityStore}.
- *
- * <p>The framework establishes the managed-operation boundary. Semantic Model
entity persistence
- * and lifecycle implementations will be added in a follow-up change.
- */
+/** EntityStore-backed operations for Gravitino-managed Semantic Models. */
public class ManagedSemanticModelOperations implements SemanticModelCatalog {
- @SuppressWarnings("UnusedVariable")
private final EntityStore store;
-
- @SuppressWarnings("UnusedVariable")
private final IdGenerator idGenerator;
+ private final BiConsumer<NameIdentifier, SemanticModelDefinition>
writeValidator;
/**
* Creates managed Semantic Model operations.
*
* @param store The EntityStore used for persistence.
* @param idGenerator The stable entity ID generator.
+ * @param writeValidator The definition validator for write operations.
*/
- public ManagedSemanticModelOperations(EntityStore store, IdGenerator
idGenerator) {
+ public ManagedSemanticModelOperations(
+ EntityStore store,
+ IdGenerator idGenerator,
+ BiConsumer<NameIdentifier, SemanticModelDefinition> writeValidator) {
+ Preconditions.checkArgument(store != null, "EntityStore must not be null");
+ Preconditions.checkArgument(idGenerator != null, "IdGenerator must not be
null");
+ Preconditions.checkArgument(writeValidator != null, "Write validator must
not be null");
this.store = store;
this.idGenerator = idGenerator;
+ this.writeValidator = writeValidator;
}
@Override
public NameIdentifier[] listSemanticModels(Namespace namespace) throws
NoSuchSchemaException {
- // TODO: Implement when SemanticModelEntity is available.
+ // TODO: Implement in the Semantic Model list/alter/drop capability.
throw new UnsupportedOperationException(
- "listSemanticModels: SemanticModelEntity is not yet implemented");
+ "listSemanticModels: list/alter/drop capability is not implemented");
}
@Override
public SemanticModel loadSemanticModel(NameIdentifier ident) throws
NoSuchSemanticModelException {
- // TODO: Implement when SemanticModelEntity is available.
- throw new UnsupportedOperationException(
- "loadSemanticModel: SemanticModelEntity is not yet implemented");
+ try {
+ return store.get(ident, Entity.EntityType.SEMANTIC_MODEL,
SemanticModelEntity.class);
+ } catch (NoSuchEntityException e) {
+ throw new NoSuchSemanticModelException(e, "Semantic Model %s does not
exist", ident);
+ } catch (IOException e) {
+ throw new RuntimeException("Failed to load Semantic Model " + ident, e);
+ }
}
@Override
@@ -80,24 +95,51 @@ public class ManagedSemanticModelOperations implements
SemanticModelCatalog {
Map<String, String> properties)
throws NoSuchSchemaException, SemanticModelAlreadyExistsException,
IllegalSemanticModelException {
- // TODO: Implement when SemanticModelEntity is available.
- throw new UnsupportedOperationException(
- "createSemanticModel: SemanticModelEntity is not yet implemented");
+ Preconditions.checkArgument(properties != null, "Properties must not be
null");
+
+ writeValidator.accept(ident, definition);
+
+ Instant now = Instant.now();
+ SemanticModelEntity entity =
+ SemanticModelEntity.builder()
+ .withId(idGenerator.nextId())
+ .withName(ident.name())
+ .withNamespace(ident.namespace())
+ .withComment(comment)
+ .withDefinition(definition)
+ .withProperties(properties)
+ .withAuditInfo(
+ AuditInfo.builder()
+ .withCreator(PrincipalUtils.getCurrentUserName())
+ .withCreateTime(now)
+ .build())
+ .build();
+
+ try {
+ store.put(entity, false /* overwrite */);
+ return entity;
+ } catch (NoSuchEntityException e) {
+ throw new NoSuchSchemaException(e, "Schema %s does not exist",
ident.namespace());
+ } catch (EntityAlreadyExistsException e) {
+ throw new SemanticModelAlreadyExistsException(e, "Semantic Model %s
already exists", ident);
+ } catch (IOException e) {
+ throw new RuntimeException("Failed to create Semantic Model " + ident,
e);
+ }
}
@Override
public SemanticModel alterSemanticModel(NameIdentifier ident,
SemanticModelChange... changes)
throws NoSuchSemanticModelException, SemanticModelAlreadyExistsException,
IllegalSemanticModelException {
- // TODO: Implement when SemanticModelEntity is available.
+ // TODO: Implement in the Semantic Model list/alter/drop capability.
throw new UnsupportedOperationException(
- "alterSemanticModel: SemanticModelEntity is not yet implemented");
+ "alterSemanticModel: list/alter/drop capability is not implemented");
}
@Override
public boolean dropSemanticModel(NameIdentifier ident) {
- // TODO: Implement when SemanticModelEntity is available.
+ // TODO: Implement in the Semantic Model list/alter/drop capability.
throw new UnsupportedOperationException(
- "dropSemanticModel: SemanticModelEntity is not yet implemented");
+ "dropSemanticModel: list/alter/drop capability is not implemented");
}
}
diff --git
a/core/src/main/java/org/apache/gravitino/catalog/SemanticModelOperationDispatcher.java
b/core/src/main/java/org/apache/gravitino/catalog/SemanticModelOperationDispatcher.java
index 8388310f2d..1be27b1100 100644
---
a/core/src/main/java/org/apache/gravitino/catalog/SemanticModelOperationDispatcher.java
+++
b/core/src/main/java/org/apache/gravitino/catalog/SemanticModelOperationDispatcher.java
@@ -61,7 +61,11 @@ public class SemanticModelOperationDispatcher extends
OperationDispatcher
super(catalogManager, store, idGenerator, secretManager);
this.catalogManager = catalogManager;
this.schemaDispatcher = schemaDispatcher;
- this.managedOperations = new ManagedSemanticModelOperations(store,
idGenerator);
+ this.managedOperations =
+ new ManagedSemanticModelOperations(
+ store,
+ idGenerator,
+ (ident, definition) ->
SemanticModelValidator.validateDefinition(definition));
}
@Override
@@ -89,7 +93,6 @@ public class SemanticModelOperationDispatcher extends
OperationDispatcher
Map<String, String> properties)
throws NoSuchSchemaException, SemanticModelAlreadyExistsException,
IllegalSemanticModelException {
- Preconditions.checkArgument(definition != null, "Definition must not be
null");
Preconditions.checkArgument(properties != null, "Properties must not be
null");
checkRelationalCatalog(ident.namespace());
NameIdentifier schemaIdent = schemaIdentifier(ident);
diff --git
a/core/src/main/java/org/apache/gravitino/catalog/SemanticModelValidator.java
b/core/src/main/java/org/apache/gravitino/catalog/SemanticModelValidator.java
new file mode 100644
index 0000000000..211585b6ae
--- /dev/null
+++
b/core/src/main/java/org/apache/gravitino/catalog/SemanticModelValidator.java
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog;
+
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.gravitino.NameIdentifier;
+import org.apache.gravitino.exceptions.IllegalSemanticModelException;
+import org.apache.gravitino.semantic.Dataset;
+import org.apache.gravitino.semantic.Field;
+import org.apache.gravitino.semantic.Metric;
+import org.apache.gravitino.semantic.Relationship;
+import org.apache.gravitino.semantic.SemanticModelDefinition;
+
+/**
+ * Validates model-level constraints for Semantic Model writes.
+ *
+ * <p>At implementation time, Apache Ossie commit {@code
88e0011148283302c9a04cd0287e00e0b9d87354},
+ * whose core specification version is {@code 0.2.0.dev0}, did not publish a
reusable Java SDK or
+ * general-purpose Java validator artifact. The Java schema validation in that
upstream tree was
+ * converter-specific. Gravitino therefore implements the applicable
model-level rules directly in
+ * Java. If a future Ossie release publishes a compatible Java SDK or
validator, Gravitino should
+ * evaluate replacing this implementation with that upstream library.
+ *
+ * <p>The API value-object builders enforce value-level invariants, including
required values, array
+ * element validity, relationship column shapes, and expression dialect
uniqueness. This validator
+ * handles constraints that span value objects: name uniqueness, relationship
endpoint resolution,
+ * and Semantic Model source identifier shape.
+ *
+ * <p>Validation is deterministic and performs no catalog I/O. Catalog-backed
source validation,
+ * including existence, columns, and authorization, must be completed by the
caller before this
+ * validator is invoked. SQL expression semantics, transitive View semantics,
and query engine
+ * compatibility are outside this validator's scope.
+ */
+final class SemanticModelValidator {
+
+ private SemanticModelValidator() {}
+
+ // TODO(#12594): Validate source existence, columns, and authorization in
the caller before
+ // invoking this definition-only validator.
+ static void validateDefinition(@Nullable SemanticModelDefinition definition)
{
+ if (definition == null) {
+ throw invalid("$", "definition must not be null");
+ }
+
+ Dataset[] datasets = definition.datasets();
+ Map<String, String> datasetNames = new HashMap<>();
+ for (int index = 0; index < datasets.length; index++) {
+ validateDataset(datasets[index], "datasets[" + index + "]",
datasetNames);
+ }
+
+ validateRelationships(definition.relationships(), datasetNames);
+ validateMetrics(definition.metrics());
+ }
+
+ private static void validateDataset(
+ Dataset dataset, String path, Map<String, String> datasetNames) {
+ String namePath = path + ".name";
+ validateUniqueName(dataset.name(), namePath, "dataset", datasetNames);
+ validateSource(dataset.source(), path + ".source");
+ validateFields(dataset.fields(), path + ".fields");
+ }
+
+ private static void validateSource(NameIdentifier source, String path) {
+ if (source.namespace().length() != 2) {
+ throw invalid(
+ path, "must contain exactly catalog.schema.name, but was '" +
source.toString() + "'");
+ }
+ }
+
+ private static void validateFields(@Nullable Field[] fields, String path) {
+ if (fields == null) {
+ return;
+ }
+
+ Map<String, String> fieldNames = new HashMap<>();
+ for (int index = 0; index < fields.length; index++) {
+ String fieldPath = path + "[" + index + "]";
+ Field field = fields[index];
+ String namePath = fieldPath + ".name";
+ validateUniqueName(field.name(), namePath, "field", fieldNames);
+ }
+ }
+
+ private static void validateRelationships(
+ @Nullable Relationship[] relationships, Map<String, String>
datasetNames) {
+ if (relationships == null) {
+ return;
+ }
+
+ Map<String, String> relationshipNames = new HashMap<>();
+ for (int index = 0; index < relationships.length; index++) {
+ String path = "relationships[" + index + "]";
+ Relationship relationship = relationships[index];
+ String namePath = path + ".name";
+ validateUniqueName(relationship.name(), namePath, "relationship",
relationshipNames);
+ validateEndpoint(relationship.from(), path + ".from", datasetNames);
+ validateEndpoint(relationship.to(), path + ".to", datasetNames);
+ }
+ }
+
+ private static void validateEndpoint(
+ String endpoint, String path, Map<String, String> datasetNames) {
+ if (!datasetNames.containsKey(endpoint)) {
+ throw invalid(
+ path,
+ "unknown dataset '"
+ + endpoint
+ + "'; relationship endpoints must reference datasets in the same
model");
+ }
+ }
+
+ private static void validateMetrics(@Nullable Metric[] metrics) {
+ if (metrics == null) {
+ return;
+ }
+
+ Map<String, String> metricNames = new HashMap<>();
+ for (int index = 0; index < metrics.length; index++) {
+ String path = "metrics[" + index + "]";
+ Metric metric = metrics[index];
+ String namePath = path + ".name";
+ validateUniqueName(metric.name(), namePath, "metric", metricNames);
+ }
+ }
+
+ private static void validateUniqueName(
+ String name, String path, String memberType, Map<String, String> names) {
+ String firstPath = names.putIfAbsent(name, path);
+ if (firstPath != null) {
+ throw invalid(
+ path, "duplicate " + memberType + " name '" + name + "'; first
declared at " + firstPath);
+ }
+ }
+
+ private static IllegalSemanticModelException invalid(String path, String
detail) {
+ return new IllegalSemanticModelException("%s: %s", path, detail);
+ }
+}
diff --git
a/core/src/test/java/org/apache/gravitino/catalog/TestManagedSemanticModelOperations.java
b/core/src/test/java/org/apache/gravitino/catalog/TestManagedSemanticModelOperations.java
index 2375316ce8..acabf3ec38 100644
---
a/core/src/test/java/org/apache/gravitino/catalog/TestManagedSemanticModelOperations.java
+++
b/core/src/test/java/org/apache/gravitino/catalog/TestManagedSemanticModelOperations.java
@@ -18,39 +18,210 @@
*/
package org.apache.gravitino.catalog;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertSame;
import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.inOrder;
import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verifyNoInteractions;
+import static org.mockito.Mockito.when;
+import java.io.IOException;
import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.gravitino.Entity;
+import org.apache.gravitino.EntityAlreadyExistsException;
import org.apache.gravitino.EntityStore;
import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.Namespace;
+import org.apache.gravitino.exceptions.IllegalSemanticModelException;
+import org.apache.gravitino.exceptions.NoSuchEntityException;
+import org.apache.gravitino.exceptions.NoSuchSchemaException;
+import org.apache.gravitino.exceptions.NoSuchSemanticModelException;
+import org.apache.gravitino.exceptions.SemanticModelAlreadyExistsException;
+import org.apache.gravitino.meta.AuditInfo;
+import org.apache.gravitino.meta.SemanticModelEntity;
+import org.apache.gravitino.semantic.Dataset;
+import org.apache.gravitino.semantic.SemanticModel;
import org.apache.gravitino.semantic.SemanticModelChange;
import org.apache.gravitino.semantic.SemanticModelDefinition;
import org.apache.gravitino.storage.IdGenerator;
+import org.apache.gravitino.storage.RandomIdGenerator;
+import
org.apache.gravitino.storage.memory.TestMemoryEntityStore.InMemoryEntityStore;
import org.junit.jupiter.api.Test;
+import org.mockito.InOrder;
public class TestManagedSemanticModelOperations {
private static final Namespace NAMESPACE = Namespace.of("metalake",
"catalog", "schema");
- private static final NameIdentifier IDENT = NameIdentifier.of(NAMESPACE,
"model");
+ private static final NameIdentifier IDENT = NameIdentifier.of(NAMESPACE,
"sales_model");
- private final ManagedSemanticModelOperations operations =
- new ManagedSemanticModelOperations(mock(EntityStore.class),
mock(IdGenerator.class));
+ @Test
+ public void testCreateThenLoadFromMemoryStore() {
+ InMemoryEntityStore store = new InMemoryEntityStore();
+ ManagedSemanticModelOperations operations =
+ new ManagedSemanticModelOperations(
+ store, new RandomIdGenerator(), (ident, definition) -> {});
+
+ SemanticModel created =
+ operations.createSemanticModel(
+ IDENT, "Sales model", definition("orders"), Map.of("domain",
"sales"));
+ SemanticModel loaded = operations.loadSemanticModel(IDENT);
+
+ assertSame(created, loaded);
+ assertEquals("sales_model", loaded.name());
+ assertEquals("Sales model", loaded.comment());
+ assertEquals(definition("orders"), loaded.definition());
+ assertEquals(Map.of("domain", "sales"), loaded.properties());
+ assertNotNull(loaded.auditInfo().creator());
+ assertNotNull(loaded.auditInfo().createTime());
+ }
+
+ @Test
+ public void testValidationAndPersistenceOrder() throws IOException {
+ EntityStore store = mock(EntityStore.class);
+ IdGenerator idGenerator = mock(IdGenerator.class);
+ @SuppressWarnings("unchecked")
+ BiConsumer<NameIdentifier, SemanticModelDefinition> writeValidator =
mock(BiConsumer.class);
+ when(idGenerator.nextId()).thenReturn(7L);
+ ManagedSemanticModelOperations operations =
+ new ManagedSemanticModelOperations(store, idGenerator, writeValidator);
+ SemanticModelDefinition definition = definition("orders");
+
+ SemanticModel created =
+ operations.createSemanticModel(IDENT, null, definition, Map.of("key",
"value"));
+
+ InOrder order = inOrder(writeValidator, idGenerator, store);
+ order.verify(writeValidator).accept(IDENT, definition);
+ order.verify(idGenerator).nextId();
+ order.verify(store).put(any(SemanticModelEntity.class), eq(false));
+ assertEquals(7L, ((SemanticModelEntity) created).id());
+ }
+
+ @Test
+ public void testWriteValidationPrecedesEntityConstructionAndPersistence() {
+ @SuppressWarnings("unchecked")
+ BiConsumer<NameIdentifier, SemanticModelDefinition> writeValidator =
mock(BiConsumer.class);
+ IllegalSemanticModelException failure =
+ new IllegalSemanticModelException("Semantic Model definition is
invalid");
+ doThrow(failure).when(writeValidator).accept(eq(IDENT),
any(SemanticModelDefinition.class));
+ EntityStore store = mock(EntityStore.class);
+ IdGenerator idGenerator = mock(IdGenerator.class);
+ ManagedSemanticModelOperations operations =
+ new ManagedSemanticModelOperations(store, idGenerator, writeValidator);
+
+ assertSame(
+ failure,
+ assertThrows(
+ IllegalSemanticModelException.class,
+ () -> operations.createSemanticModel(IDENT, null,
definition("orders"), Map.of())));
+ verifyNoInteractions(idGenerator, store);
+ }
+
+ @Test
+ public void testLoadDoesNotRunAnyValidation() throws IOException {
+ EntityStore store = mock(EntityStore.class);
+ @SuppressWarnings("unchecked")
+ BiConsumer<NameIdentifier, SemanticModelDefinition> writeValidator =
mock(BiConsumer.class);
+ SemanticModelEntity entity = entity();
+ when(store.get(IDENT, Entity.EntityType.SEMANTIC_MODEL,
SemanticModelEntity.class))
+ .thenReturn(entity);
+ ManagedSemanticModelOperations operations =
+ new ManagedSemanticModelOperations(store, mock(IdGenerator.class),
writeValidator);
+
+ assertSame(entity, operations.loadSemanticModel(IDENT));
+
+ verifyNoInteractions(writeValidator);
+ }
+
+ @Test
+ public void testCreateExceptionMapping() throws IOException {
+ assertCreateFailure(new NoSuchEntityException("Missing parent"),
NoSuchSchemaException.class);
+ assertCreateFailure(
+ new EntityAlreadyExistsException("Already exists"),
+ SemanticModelAlreadyExistsException.class);
+
+ IOException ioFailure = new IOException("Write failed");
+ RuntimeException wrapped = assertCreateFailure(ioFailure,
RuntimeException.class);
+ assertSame(ioFailure, wrapped.getCause());
+ }
+
+ @Test
+ public void testLoadExceptionMapping() throws IOException {
+ EntityStore store = mock(EntityStore.class);
+ ManagedSemanticModelOperations operations =
+ new ManagedSemanticModelOperations(
+ store, mock(IdGenerator.class), (ident, definition) -> {});
+
+ when(store.get(IDENT, Entity.EntityType.SEMANTIC_MODEL,
SemanticModelEntity.class))
+ .thenThrow(new NoSuchEntityException("Missing model"));
+ assertThrows(NoSuchSemanticModelException.class, () ->
operations.loadSemanticModel(IDENT));
+
+ IOException ioFailure = new IOException("Read failed");
+ doThrow(ioFailure)
+ .when(store)
+ .get(IDENT, Entity.EntityType.SEMANTIC_MODEL,
SemanticModelEntity.class);
+ RuntimeException wrapped =
+ assertThrows(RuntimeException.class, () ->
operations.loadSemanticModel(IDENT));
+ assertSame(ioFailure, wrapped.getCause());
+ }
@Test
- public void testOperationsRemainUnsupportedUntilEntityPersistenceIsAdded() {
- SemanticModelDefinition definition = mock(SemanticModelDefinition.class);
+ public void testRemainingCapabilitiesAreExplicitlyUnsupported() {
+ ManagedSemanticModelOperations operations =
+ new ManagedSemanticModelOperations(
+ mock(EntityStore.class), mock(IdGenerator.class), (ident,
definition) -> {});
assertThrows(
UnsupportedOperationException.class, () ->
operations.listSemanticModels(NAMESPACE));
- assertThrows(UnsupportedOperationException.class, () ->
operations.loadSemanticModel(IDENT));
- assertThrows(
- UnsupportedOperationException.class,
- () -> operations.createSemanticModel(IDENT, null, definition,
Map.of()));
assertThrows(
UnsupportedOperationException.class,
- () -> operations.alterSemanticModel(IDENT,
SemanticModelChange.updateComment("updated")));
+ () ->
+ operations.alterSemanticModel(
+ IDENT, SemanticModelChange.updateComment("Not implemented")));
assertThrows(UnsupportedOperationException.class, () ->
operations.dropSemanticModel(IDENT));
}
+
+ private static RuntimeException assertCreateFailure(
+ Exception failure, Class<? extends RuntimeException> expectedType)
throws IOException {
+ EntityStore store = mock(EntityStore.class);
+ doThrow(failure).when(store).put(any(SemanticModelEntity.class),
eq(false));
+ ManagedSemanticModelOperations operations =
+ new ManagedSemanticModelOperations(store, () -> 1L, (ident,
definition) -> {});
+
+ RuntimeException thrown =
+ assertThrows(
+ RuntimeException.class,
+ () -> operations.createSemanticModel(IDENT, null,
definition("orders"), Map.of()));
+ assertInstanceOf(expectedType, thrown);
+ return thrown;
+ }
+
+ private static SemanticModelEntity entity() {
+ return SemanticModelEntity.builder()
+ .withId(1L)
+ .withName(IDENT.name())
+ .withNamespace(IDENT.namespace())
+ .withDefinition(definition("orders"))
+ .withAuditInfo(AuditInfo.EMPTY)
+ .build();
+ }
+
+ private static SemanticModelDefinition definition(String datasetName) {
+ return SemanticModelDefinition.builder()
+ .withDatasets(new Dataset[] {dataset(datasetName)})
+ .build();
+ }
+
+ private static Dataset dataset(String name) {
+ return Dataset.builder()
+ .withName(name)
+ .withSource(NameIdentifier.of("sales", "mart", name))
+ .build();
+ }
}
diff --git
a/core/src/test/java/org/apache/gravitino/catalog/TestManagedSemanticModelOperationsJDBC.java
b/core/src/test/java/org/apache/gravitino/catalog/TestManagedSemanticModelOperationsJDBC.java
new file mode 100644
index 0000000000..c6b10f7c31
--- /dev/null
+++
b/core/src/test/java/org/apache/gravitino/catalog/TestManagedSemanticModelOperationsJDBC.java
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.commons.lang3.reflect.FieldUtils;
+import org.apache.gravitino.Config;
+import org.apache.gravitino.Configs;
+import org.apache.gravitino.Entity;
+import org.apache.gravitino.GravitinoEnv;
+import org.apache.gravitino.NameIdentifier;
+import org.apache.gravitino.Namespace;
+import org.apache.gravitino.cache.NoOpsCache;
+import org.apache.gravitino.semantic.CustomExtension;
+import org.apache.gravitino.semantic.Dataset;
+import org.apache.gravitino.semantic.Metric;
+import org.apache.gravitino.semantic.Relationship;
+import org.apache.gravitino.semantic.SemanticModel;
+import org.apache.gravitino.semantic.SemanticModelDefinition;
+import org.apache.gravitino.storage.RandomIdGenerator;
+import org.apache.gravitino.storage.relational.RelationalEntityStore;
+import org.apache.gravitino.storage.relational.TestJDBCBackend;
+import org.apache.gravitino.utils.NamespaceUtil;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+
+/** Verifies managed create and load through a real relational persistence
backend. */
+public class TestManagedSemanticModelOperationsJDBC extends TestJDBCBackend {
+
+ private final AtomicInteger writeValidationCount = new AtomicInteger();
+
+ private Config previousConfig;
+ private NameIdentifier modelIdent;
+ private ManagedSemanticModelOperations operations;
+
+ @BeforeAll
+ public void captureEnvironmentConfig() {
+ // BackendTestExtension replaces GravitinoEnv.config before each backend
invocation.
+ previousConfig = GravitinoEnv.getInstance().config();
+ }
+
+ @BeforeEach
+ public void prepareManagedOperations() throws IOException,
IllegalAccessException {
+ writeValidationCount.set(0);
+ String metalake = "managed_semantic_model_metalake";
+ String catalog = "managed_semantic_model_catalog";
+ String schema = "managed_semantic_model_schema";
+ createAndInsertMakeLake(metalake);
+ createAndInsertCatalog(metalake, catalog);
+ createAndInsertSchema(metalake, catalog, schema);
+
+ Namespace namespace = NamespaceUtil.ofSemanticModel(metalake, catalog,
schema);
+ modelIdent = NameIdentifier.of(namespace, "sales_model");
+
+ Config config = new Config(false) {};
+ config.set(Configs.CACHE_ENABLED, false);
+ RelationalEntityStore store = new RelationalEntityStore();
+ FieldUtils.writeField(store, "backend", backend, true);
+ FieldUtils.writeField(store, "cache", new NoOpsCache(config), true);
+ operations =
+ new ManagedSemanticModelOperations(
+ store,
+ RandomIdGenerator.INSTANCE,
+ (ident, definition) -> writeValidationCount.incrementAndGet());
+ }
+
+ @AfterEach
+ public void restoreEnvironmentConfig() throws IllegalAccessException {
+ FieldUtils.writeField(GravitinoEnv.getInstance(), "config",
previousConfig, true);
+ }
+
+ @TestTemplate
+ public void testCreateThenLoadRoundTrip() throws IOException {
+ Dataset dataset =
+ Dataset.builder()
+ .withName("orders")
+ .withSource(NameIdentifier.of("source_catalog", "source_schema",
"orders"))
+ .withPrimaryKey(new String[0])
+ .withUniqueKeys(new String[0][])
+ .withCustomExtensions(new CustomExtension[0])
+ .build();
+ SemanticModelDefinition definition =
+ SemanticModelDefinition.builder()
+ .withDatasets(new Dataset[] {dataset})
+ .withRelationships(new Relationship[0])
+ .withMetrics((Metric[]) null)
+ .withCustomExtensions(new CustomExtension[0])
+ .build();
+
+ SemanticModel created =
+ operations.createSemanticModel(
+ modelIdent, "Persisted model", definition, Map.of("domain",
"sales"));
+ SemanticModel loaded = operations.loadSemanticModel(modelIdent);
+
+ assertNotSame(created, loaded);
+ assertEquals(created, loaded);
+ assertEquals(definition, loaded.definition());
+ assertEquals(1, writeValidationCount.get());
+ assertTrue(backend.exists(modelIdent, Entity.EntityType.SEMANTIC_MODEL));
+ }
+}
diff --git
a/core/src/test/java/org/apache/gravitino/catalog/TestSemanticModelOperationDispatcher.java
b/core/src/test/java/org/apache/gravitino/catalog/TestSemanticModelOperationDispatcher.java
index fc7d0c7ac7..9d0c2aa53d 100644
---
a/core/src/test/java/org/apache/gravitino/catalog/TestSemanticModelOperationDispatcher.java
+++
b/core/src/test/java/org/apache/gravitino/catalog/TestSemanticModelOperationDispatcher.java
@@ -21,13 +21,13 @@ package org.apache.gravitino.catalog;
import static org.apache.gravitino.Configs.TREE_LOCK_CLEAN_INTERVAL;
import static org.apache.gravitino.Configs.TREE_LOCK_MAX_NODE_IN_MEMORY;
import static org.apache.gravitino.Configs.TREE_LOCK_MIN_NODE_IN_MEMORY;
+import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertSame;
import static org.junit.jupiter.api.Assertions.assertThrows;
-import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.Mockito.doReturn;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
-import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
@@ -35,19 +35,21 @@ import java.util.Map;
import org.apache.commons.lang3.reflect.FieldUtils;
import org.apache.gravitino.Catalog;
import org.apache.gravitino.Config;
-import org.apache.gravitino.EntityStore;
import org.apache.gravitino.GravitinoEnv;
import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.Namespace;
import org.apache.gravitino.Schema;
+import org.apache.gravitino.exceptions.IllegalSemanticModelException;
import org.apache.gravitino.exceptions.NoSuchSchemaException;
-import org.apache.gravitino.exceptions.NoSuchSemanticModelException;
import org.apache.gravitino.lock.LockManager;
import org.apache.gravitino.secret.SecretManager;
import org.apache.gravitino.semantic.Dataset;
+import org.apache.gravitino.semantic.Relationship;
+import org.apache.gravitino.semantic.SemanticModel;
import org.apache.gravitino.semantic.SemanticModelChange;
import org.apache.gravitino.semantic.SemanticModelDefinition;
-import org.apache.gravitino.storage.IdGenerator;
+import org.apache.gravitino.storage.RandomIdGenerator;
+import
org.apache.gravitino.storage.memory.TestMemoryEntityStore.InMemoryEntityStore;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
@@ -55,13 +57,16 @@ import org.junit.jupiter.api.Test;
public class TestSemanticModelOperationDispatcher {
private static final String METALAKE = "metalake";
- private static final NameIdentifier CATALOG_IDENT =
NameIdentifier.of(METALAKE, "catalog");
- private static final Namespace NAMESPACE = Namespace.of(METALAKE, "catalog",
"schema");
+ private static final NameIdentifier METADATA_CATALOG_IDENT =
+ NameIdentifier.of(METALAKE, "metadata_catalog");
+ private static final Namespace NAMESPACE =
+ Namespace.of(METALAKE, "metadata_catalog", "semantic_schema");
private static final NameIdentifier SCHEMA_IDENT =
NameIdentifier.of(NAMESPACE.levels());
private static final NameIdentifier MODEL_IDENT =
NameIdentifier.of(NAMESPACE, "sales_model");
private CatalogManager catalogManager;
private SchemaDispatcher schemaDispatcher;
+ private InMemoryEntityStore store;
private SemanticModelOperationDispatcher dispatcher;
@BeforeAll
@@ -77,10 +82,11 @@ public class TestSemanticModelOperationDispatcher {
public void setUp() throws Exception {
catalogManager = mock(CatalogManager.class);
schemaDispatcher = mock(SchemaDispatcher.class);
+ store = new InMemoryEntityStore();
Catalog catalog = mock(Catalog.class);
when(catalog.type()).thenReturn(Catalog.Type.RELATIONAL);
- when(catalogManager.loadCatalog(CATALOG_IDENT)).thenReturn(catalog);
+
when(catalogManager.loadCatalog(METADATA_CATALOG_IDENT)).thenReturn(catalog);
when(schemaDispatcher.loadSchema(SCHEMA_IDENT)).thenReturn(mock(Schema.class));
when(schemaDispatcher.schemaExists(SCHEMA_IDENT)).thenReturn(true);
@@ -88,97 +94,97 @@ public class TestSemanticModelOperationDispatcher {
new SemanticModelOperationDispatcher(
catalogManager,
schemaDispatcher,
- mock(EntityStore.class),
- mock(IdGenerator.class),
+ store,
+ new RandomIdGenerator(),
mock(SecretManager.class));
}
@Test
- public void testFrameworkValidatesParentAndDelegatesToManagedOperations() {
- UnsupportedOperationException listFailure =
- assertThrows(
- UnsupportedOperationException.class, () ->
dispatcher.listSemanticModels(NAMESPACE));
- assertTrue(listFailure.getMessage().startsWith("listSemanticModels:"));
-
- UnsupportedOperationException createFailure =
- assertThrows(
- UnsupportedOperationException.class,
- () -> dispatcher.createSemanticModel(MODEL_IDENT, null,
definition(), Map.of()));
- assertTrue(createFailure.getMessage().startsWith("createSemanticModel:"));
-
- UnsupportedOperationException loadFailure =
- assertThrows(
- UnsupportedOperationException.class, () ->
dispatcher.loadSemanticModel(MODEL_IDENT));
- assertTrue(loadFailure.getMessage().startsWith("loadSemanticModel:"));
-
- UnsupportedOperationException alterFailure =
- assertThrows(
- UnsupportedOperationException.class,
- () ->
- dispatcher.alterSemanticModel(
- MODEL_IDENT,
SemanticModelChange.updateComment("updated")));
- assertTrue(alterFailure.getMessage().startsWith("alterSemanticModel:"));
-
- UnsupportedOperationException dropFailure =
- assertThrows(
- UnsupportedOperationException.class, () ->
dispatcher.dropSemanticModel(MODEL_IDENT));
- assertTrue(dropFailure.getMessage().startsWith("dropSemanticModel:"));
-
- verify(schemaDispatcher, times(2)).loadSchema(SCHEMA_IDENT);
- verify(schemaDispatcher, times(3)).schemaExists(SCHEMA_IDENT);
+ public void testCreateThenLoadWithoutCatalogBackedSourceValidation() {
+ SemanticModel created =
+ dispatcher.createSemanticModel(MODEL_IDENT, "Sales",
validDefinition(), Map.of());
+
+ assertEquals("sales_model", created.name());
+ assertEquals(2, created.definition().datasets().length);
+ assertSame(created, dispatcher.loadSemanticModel(MODEL_IDENT));
}
@Test
- public void testMissingSchemaPreservesTypedResults() {
+ public void testSchemaFailureRemainsTyped() {
when(schemaDispatcher.loadSchema(SCHEMA_IDENT))
.thenThrow(new NoSuchSchemaException("Schema does not exist"));
- when(schemaDispatcher.schemaExists(SCHEMA_IDENT)).thenReturn(false);
-
- assertThrows(NoSuchSchemaException.class, () ->
dispatcher.listSemanticModels(NAMESPACE));
assertThrows(
NoSuchSchemaException.class,
- () -> dispatcher.createSemanticModel(MODEL_IDENT, null, definition(),
Map.of()));
- assertThrows(
- NoSuchSemanticModelException.class, () ->
dispatcher.loadSemanticModel(MODEL_IDENT));
- assertThrows(
- NoSuchSemanticModelException.class,
- () ->
- dispatcher.alterSemanticModel(
- MODEL_IDENT, SemanticModelChange.updateComment("updated")));
- assertFalse(dispatcher.dropSemanticModel(MODEL_IDENT));
+ () -> dispatcher.createSemanticModel(MODEL_IDENT, null,
validDefinition(), Map.of()));
+ assertFalse(dispatcher.semanticModelExists(MODEL_IDENT));
}
@Test
public void testNonRelationalCatalogIsRejectedBeforeSchemaLookup() {
Catalog catalog = mock(Catalog.class);
when(catalog.type()).thenReturn(Catalog.Type.FILESET);
- when(catalogManager.loadCatalog(CATALOG_IDENT)).thenReturn(catalog);
+
when(catalogManager.loadCatalog(METADATA_CATALOG_IDENT)).thenReturn(catalog);
- UnsupportedOperationException failure =
- assertThrows(
- UnsupportedOperationException.class, () ->
dispatcher.listSemanticModels(NAMESPACE));
-
- assertTrue(failure.getMessage().contains("does not support Semantic Model
operations"));
+ assertThrows(
+ UnsupportedOperationException.class, () ->
dispatcher.listSemanticModels(NAMESPACE));
verify(schemaDispatcher, never()).loadSchema(SCHEMA_IDENT);
}
@Test
- public void testInvalidInputsAreRejectedBeforeCatalogLookup() {
+ public void testRemainingManagedCapabilitiesStayUnsupported() {
assertThrows(
- IllegalArgumentException.class,
+ UnsupportedOperationException.class, () ->
dispatcher.listSemanticModels(NAMESPACE));
+ assertThrows(
+ UnsupportedOperationException.class,
+ () ->
+ dispatcher.alterSemanticModel(
+ MODEL_IDENT, SemanticModelChange.updateComment("Not
implemented")));
+ assertThrows(
+ UnsupportedOperationException.class, () ->
dispatcher.dropSemanticModel(MODEL_IDENT));
+ }
+
+ @Test
+ public void testNullDefinitionIsRejectedByValidator() {
+ assertThrows(
+ IllegalSemanticModelException.class,
() -> dispatcher.createSemanticModel(MODEL_IDENT, null, null,
Map.of()));
+ assertFalse(dispatcher.semanticModelExists(MODEL_IDENT));
+ }
+
+ @Test
+ public void testNullPropertiesAreRejectedBeforeCatalogLookup() {
assertThrows(
IllegalArgumentException.class,
- () -> dispatcher.createSemanticModel(MODEL_IDENT, null, definition(),
null));
- verify(catalogManager, never()).loadCatalog(CATALOG_IDENT);
+ () -> dispatcher.createSemanticModel(MODEL_IDENT, null,
validDefinition(), null));
+ verify(catalogManager, never()).loadCatalog(METADATA_CATALOG_IDENT);
}
- private static SemanticModelDefinition definition() {
- Dataset dataset =
- Dataset.builder()
- .withName("orders")
- .withSource(NameIdentifier.of("sales", "mart", "orders"))
+ private static SemanticModelDefinition validDefinition() {
+ Dataset orders =
+ dataset("orders", "orders", new String[] {"order_id"}, new String[][]
{{"customer_id"}});
+ Dataset customers =
+ dataset("customers", "customers", new String[] {"customer_id"}, new
String[0][]);
+ Relationship relationship =
+ Relationship.builder()
+ .withName("orders_to_customers")
+ .withFrom("orders")
+ .withTo("customers")
+ .withFromColumns(new String[] {"customer_id"})
+ .withToColumns(new String[] {"customer_id"})
.build();
- return SemanticModelDefinition.builder().withDatasets(new Dataset[]
{dataset}).build();
+ return SemanticModelDefinition.builder()
+ .withDatasets(new Dataset[] {orders, customers})
+ .withRelationships(new Relationship[] {relationship})
+ .build();
+ }
+
+ private static Dataset dataset(
+ String name, String source, String[] primaryKey, String[][] uniqueKeys) {
+ return Dataset.builder()
+ .withName(name)
+ .withSource(NameIdentifier.of("sales", "mart", source))
+ .withPrimaryKey(primaryKey)
+ .withUniqueKeys(uniqueKeys)
+ .build();
}
}
diff --git
a/core/src/test/java/org/apache/gravitino/catalog/TestSemanticModelValidator.java
b/core/src/test/java/org/apache/gravitino/catalog/TestSemanticModelValidator.java
new file mode 100644
index 0000000000..cff181337a
--- /dev/null
+++
b/core/src/test/java/org/apache/gravitino/catalog/TestSemanticModelValidator.java
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+import javax.annotation.Nullable;
+import org.apache.gravitino.NameIdentifier;
+import org.apache.gravitino.exceptions.IllegalSemanticModelException;
+import org.apache.gravitino.semantic.CustomExtension;
+import org.apache.gravitino.semantic.Dataset;
+import org.apache.gravitino.semantic.DialectExpression;
+import org.apache.gravitino.semantic.Dialects;
+import org.apache.gravitino.semantic.Expression;
+import org.apache.gravitino.semantic.Field;
+import org.apache.gravitino.semantic.Metric;
+import org.apache.gravitino.semantic.Relationship;
+import org.apache.gravitino.semantic.SemanticModelDefinition;
+import org.junit.jupiter.api.Test;
+
+public class TestSemanticModelValidator {
+
+ @Test
+ public void testValidCompleteDefinitionWithoutExternalResolution() {
+ CustomExtension extension = extension();
+ Dataset orders =
+ Dataset.builder()
+ .withName("orders")
+ .withSource(NameIdentifier.of("missing_catalog", "missing_schema",
"orders"))
+ .withPrimaryKey(new String[] {"order_id"})
+ .withUniqueKeys(new String[][] {{"order_id"}, {"external_id",
"source"}})
+ .withFields(
+ new Field[] {
+ field("id", expression("order_id")), field("amount",
expression("amount"))
+ })
+ .withCustomExtensions(new CustomExtension[] {extension})
+ .build();
+ Dataset customers =
+ Dataset.builder()
+ .withName("customers")
+ .withSource(NameIdentifier.of("missing_catalog", "missing_schema",
"customers"))
+ .withFields(new Field[] {field("id", expression("customer_id"))})
+ .build();
+ Relationship relationship =
+ relationship(
+ "orders_to_customers",
+ "orders",
+ "customers",
+ new String[] {"customer_id", "tenant_id"},
+ new String[] {"id", "tenant_id"});
+ Metric metric =
+ Metric.builder()
+ .withName("total_revenue")
+ .withExpression(
+ multiDialectExpression(
+ "SUM(orders.amount) /* text is intentionally uninterpreted
*/",
+ "SUM(orders.amount)"))
+ .withCustomExtensions(new CustomExtension[] {extension})
+ .build();
+
+ SemanticModelDefinition definition =
+ SemanticModelDefinition.builder()
+ .withDatasets(new Dataset[] {orders, customers})
+ .withRelationships(new Relationship[] {relationship})
+ .withMetrics(new Metric[] {metric})
+ .withCustomExtensions(new CustomExtension[] {extension})
+ .build();
+
+ assertDoesNotThrow(() ->
SemanticModelValidator.validateDefinition(definition));
+ }
+
+ @Test
+ public void testDefinitionAndDatasetConstraints() {
+ assertInvalid(null, "$: definition must not be null");
+
+ assertInvalid(
+ definition(dataset("orders"), dataset("orders")),
+ "datasets[1].name: duplicate dataset name 'orders'; first declared at
datasets[0].name");
+
+ Dataset shortSource =
+ Dataset.builder()
+ .withName("orders")
+ .withSource(NameIdentifier.of("sales", "orders"))
+ .build();
+ assertInvalid(
+ definition(shortSource),
+ "datasets[0].source: must contain exactly catalog.schema.name, but was
'sales.orders'");
+ }
+
+ @Test
+ public void testFieldNamesAreUniquePerDataset() {
+ Dataset duplicateFields =
+ Dataset.builder()
+ .withName("orders")
+ .withSource(NameIdentifier.of("sales", "mart", "orders"))
+ .withFields(
+ new Field[] {field("id", expression("id")), field("id",
expression("order_id"))})
+ .build();
+
+ assertInvalid(
+ definition(duplicateFields),
+ "datasets[0].fields[1].name: duplicate field name 'id'; first declared
at "
+ + "datasets[0].fields[0].name");
+
+ assertDoesNotThrow(
+ () ->
+ SemanticModelValidator.validateDefinition(
+ definition(datasetWithField("orders", "id"),
datasetWithField("customers", "id"))));
+ }
+
+ @Test
+ public void testRelationshipNamesAndEndpoints() {
+ Dataset orders = dataset("orders");
+ Dataset customers = dataset("customers");
+ Relationship relationship = relationship("by_customer", "orders",
"customers");
+
+ assertInvalid(
+ definition(
+ new Dataset[] {orders, customers},
+ new Relationship[] {relationship, relationship},
+ null),
+ "relationships[1].name: duplicate relationship name 'by_customer';
first declared at "
+ + "relationships[0].name");
+ assertInvalid(
+ definition(
+ new Dataset[] {orders},
+ new Relationship[] {relationship("missing", "orders",
"customers")},
+ null),
+ "relationships[0].to: unknown dataset 'customers'; relationship
endpoints must reference "
+ + "datasets in the same model");
+ }
+
+ @Test
+ public void testMetricNames() {
+ Metric revenue = metric("revenue", expression("SUM(amount)"));
+ assertInvalid(
+ definition(new Dataset[] {dataset("orders")}, null, new Metric[]
{revenue, revenue}),
+ "metrics[1].name: duplicate metric name 'revenue'; first declared at
metrics[0].name");
+ }
+
+ private static void assertInvalid(
+ @Nullable SemanticModelDefinition definition, String expectedMessage) {
+ IllegalSemanticModelException exception =
+ assertThrows(
+ IllegalSemanticModelException.class,
+ () -> SemanticModelValidator.validateDefinition(definition));
+ assertEquals(expectedMessage, exception.getMessage());
+ }
+
+ private static SemanticModelDefinition definition(Dataset... datasets) {
+ return SemanticModelDefinition.builder().withDatasets(datasets).build();
+ }
+
+ private static SemanticModelDefinition definition(
+ Dataset[] datasets, @Nullable Relationship[] relationships, @Nullable
Metric[] metrics) {
+ return SemanticModelDefinition.builder()
+ .withDatasets(datasets)
+ .withRelationships(relationships)
+ .withMetrics(metrics)
+ .build();
+ }
+
+ private static Dataset dataset(String name) {
+ return Dataset.builder()
+ .withName(name)
+ .withSource(NameIdentifier.of("sales", "mart", name))
+ .build();
+ }
+
+ private static Dataset datasetWithField(String name, String fieldName) {
+ return Dataset.builder()
+ .withName(name)
+ .withSource(NameIdentifier.of("sales", "mart", name))
+ .withFields(new Field[] {field(fieldName, expression(fieldName))})
+ .build();
+ }
+
+ private static Field field(String name, Expression expression) {
+ return Field.builder().withName(name).withExpression(expression).build();
+ }
+
+ private static Metric metric(String name, Expression expression) {
+ return Metric.builder().withName(name).withExpression(expression).build();
+ }
+
+ private static Relationship relationship(String name, String from, String
to) {
+ return relationship(name, from, to, new String[] {"customer_id"}, new
String[] {"id"});
+ }
+
+ private static Relationship relationship(
+ String name, String from, String to, String[] fromColumns, String[]
toColumns) {
+ return Relationship.builder()
+ .withName(name)
+ .withFrom(from)
+ .withTo(to)
+ .withFromColumns(fromColumns)
+ .withToColumns(toColumns)
+ .build();
+ }
+
+ private static Expression expression(String value) {
+ return Expression.builder()
+ .withDialects(new DialectExpression[] {dialect(Dialects.ANSI_SQL,
value)})
+ .build();
+ }
+
+ private static Expression multiDialectExpression(String ansi, String
bigQuery) {
+ return Expression.builder()
+ .withDialects(
+ new DialectExpression[] {
+ dialect(Dialects.ANSI_SQL, ansi), dialect(Dialects.BIGQUERY,
bigQuery)
+ })
+ .build();
+ }
+
+ private static DialectExpression dialect(String dialect, String expression) {
+ return
DialectExpression.builder().withDialect(dialect).withExpression(expression).build();
+ }
+
+ private static CustomExtension extension() {
+ return
CustomExtension.builder().withVendorName("example").withData("{}").build();
+ }
+}
diff --git a/design-docs/gravitino-semantic-model-design.md
b/design-docs/gravitino-semantic-model-design.md
index a65069ef21..44abd65927 100644
--- a/design-docs/gravitino-semantic-model-design.md
+++ b/design-docs/gravitino-semantic-model-design.md
@@ -269,10 +269,10 @@ DataType = "String" | "Integer" | "Decimal" | "Float" |
"Boolean"
Create and alter validate the complete candidate object before persistence.
Validation is atomic:
if any check fails, no change is persisted.
-1. **Contract validation.** Check required fields, collection cardinality,
enum values, string
- constraints, and the pinned Ossie structure.
-2. **Model-local validation.** Check name uniqueness, relationship endpoints,
referenced fields,
- key shapes, and dialect uniqueness without consulting a catalog.
+1. **Contract validation.** Check required fields, collection cardinality,
typed alternatives, string
+ constraints, and the Java contract derived from the pinned Ossie structure.
+2. **Model-local validation.** Check name uniqueness, relationship endpoints,
key shapes, and dialect
+ uniqueness without consulting a catalog.
3. **Catalog validation.** Resolve every Dataset source and validate
explicitly named source
columns. Validation uses the caller's authorization context so it does not
disclose metadata the
caller cannot access.
@@ -293,33 +293,23 @@ engine and remain outside the metadata write path.
#### Ossie Compatibility
-The public API is a Gravitino contract derived from Ossie, not a stored YAML
or JSON document. For
-compatibility validation, the server projects a candidate object into an
in-memory Ossie document:
-
-```yaml
-version: 0.2.0.dev0
-semantic_model:
- - name: sales_model
- description: Governed sales definitions
- datasets:
- - name: orders
- source: sales.mart.orders
-```
-
-The projection maps the entity name to `SemanticModel.name`, `comment` to
`description`, and encodes
-each source `NameIdentifier` using the reversible source grammar above. The
request path supplies the
-metalake and is not serialized into Ossie.
+The public API is a Gravitino contract derived from Ossie, not a stored YAML
or JSON document. The
+runtime write path validates the Java object model directly; it does not
project a candidate into an
+Ossie document or invoke JSON Schema validation. At implementation time,
Apache Ossie commit
+`88e0011148283302c9a04cd0287e00e0b9d87354` did not publish a reusable Java SDK
or general-purpose
+Java validator; its Java schema validation was converter-specific. Gravitino
therefore implements
+the applicable structural and model-local rules directly in Java, followed by
catalog-backed source
+checks. SQL expression validation is not carried into the metadata write path
because execution
+semantics and engine compatibility are outside its scope.
-The request path does not invoke the upstream Python validator or create an
intermediate YAML
-string. The implementation validates the in-memory projection against a
Gravitino profile derived
-from the bundled copy of the pinned JSON Schema, then runs Gravitino-specific
model-local and
-catalog checks. The profile preserves the pinned structural constraints while
treating dialect
-identifiers as open non-empty strings. Upstream validation tools are used only
in compatibility
-fixtures that contain Ossie-defined dialects.
+Core contract tests validate the Java value model directly, including both
`AIContext` variants,
+name uniqueness, relationship references and column cardinality, and absent
versus explicitly empty
+arrays. Gravitino's runtime contract deliberately accepts any non-empty custom
dialect identifier and
+preserves it exactly, even when the dialect is outside the pinned upstream
schema.
The entity carries no per-model Ossie specification version. Each Gravitino
release pins one exact
upstream schema commit and defines the supported read and write contract.
Updating that schema
-requires a Gravitino code change, compatibility fixtures, and storage read
tests. If a future
+requires a Gravitino code change, native validator tests, and storage read
tests. If a future
incompatible Ossie contract must coexist with the current one, explicit
per-entity versioning
requires a separate design; it should not be inferred from `custom_extensions`.
@@ -354,6 +344,11 @@ Supported `SemanticModelChange` operations are:
- `replaceDefinition`: Atomically replaces AI context, datasets,
relationships, metrics, and custom
extensions.
+Create and `replaceDefinition` run complete Java definition and catalog-backed
source validation
+before persistence. A properties-only alter does not revalidate the definition
or visit sources.
+Rename and comment changes also do not visit sources; rename continues through
the existing
+identifier normalization contract. List, load, and drop do not run write
validation.
+
Each alter request applies all changes atomically to the current Semantic
Model. Rename retains the
stable entity ID. Owner, tag, and policy changes use their existing governance
stores and remain
outside `SemanticModelChange`.
