This is an automated email from the ASF dual-hosted git repository.
yuqi1129 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 3ec114aa28 [#13568] feat(iceberg): Add experimental Iceberg REST
auxiliary service (#13569)
3ec114aa28 is described below
commit 3ec114aa28aee7cfc1672b5f1ba24993df25b511
Author: roryqi <[email protected]>
AuthorDate: Fri Oct 16 21:23:28 2026 +0800
[#13568] feat(iceberg): Add experimental Iceberg REST auxiliary service
(#13569)
### What changes were proposed in this pull request?
- Add the `iceberg-rest-experimental-server` module.
- Package the published `com.datastrato:experimental-iceberg-*` 1.13.0
artifacts under a separate classpath.
- Keep the packaged experimental server's shared Caffeine and Jackson
dependencies aligned with Gravitino's pinned versions.
- Add the experimental module to the Gravitino binary distribution and
update its legal notices.
- Add configuration documentation and log the effective Iceberg version
and code source at startup.
- Run the experimental authorization suite in a dedicated test task with
the experimental Iceberg runtime and Spark 4.1/Scala 2.13. Keep the
embedded Gravitino server on its javax Jersey/HK2 stack and disable the
incompatible Spark UI servlet during this suite.
### Why are the changes needed?
Developers need to test Iceberg read-restriction work against the
Datastrato Iceberg distribution without replacing the regular Iceberg
dependency set.
The regular and experimental variants reuse the `iceberg-rest` auxiliary
service name and configuration keys. Users select one variant through
`gravitino.iceberg-rest.classpath`; the two variants cannot run side by
side.
Fix: #13568
### Does this PR introduce _any_ user-facing change?
Yes. The binary distribution adds the `iceberg-rest-experimental-server`
directory. To use it, configure:
```properties
gravitino.auxService.names = iceberg-rest
gravitino.iceberg-rest.classpath = iceberg-rest-experimental-server/libs,
iceberg-rest-experimental-server/conf
```
### How was this patch tested?
- `./gradlew :iceberg:iceberg-rest-server:experimentalTest -PskipTests
-PskipDockerTests=false -PtestMode=embedded`
- `./gradlew :iceberg:iceberg-rest-server:test -PskipITs`
- `./gradlew :iceberg:iceberg-rest-server:test -PskipTests
-PskipDockerTests=false -PtestMode=embedded --tests
'org.apache.gravitino.iceberg.integration.test.IcebergTableAuthorizationIT.testLoadTable'`
- `./gradlew :iceberg:iceberg-rest-server:compileTestJava
:iceberg:iceberg-rest-experimental-server:copyDepends`
- `./gradlew :iceberg:iceberg-rest-server:spotlessCheck
:iceberg:iceberg-rest-experimental-server:spotlessCheck`
- Verified that the experimental test ran 16 tests with no failures on
Spark 4.1.3 and loaded the pinned Datastrato Iceberg commit from
`com.datastrato:experimental-iceberg-api`.
- Verified that the packaged experimental dependencies contain only
`experimental-iceberg-*` 1.13.0 artifacts, Caffeine 2.9.3, and Jackson
2.15.2 for the affected shared libraries.
---
.../workflows/backend-integration-test-action.yml | 2 +-
NOTICE.bin | 10 +-
build.gradle.kts | 1 +
.../client-python/gravitino/utils/http_client.py | 2 +-
.../tests/unittests/test_http_client_headers.py | 21 ++++
conf/gravitino.conf.template | 2 +
docs/iceberg-rest-service.md | 24 ++++-
gradle/libs.versions.toml | 3 +
.../build.gradle.kts | 93 ++++++++++++++++++
iceberg/iceberg-rest-server/build.gradle.kts | 106 +++++++++++++++++++-
.../org/apache/gravitino/iceberg/RESTService.java | 7 ++
.../integration/test/IcebergAuthorizationIT.java | 9 +-
.../test/IcebergExperimentalAuthorizationIT.java | 107 +++++++++++++++++++++
.../test/IcebergTableAuthorizationIT.java | 13 ++-
.../integration/test/util/TestDatabaseName.java | 3 +
settings.gradle.kts | 1 +
16 files changed, 392 insertions(+), 12 deletions(-)
diff --git a/.github/workflows/backend-integration-test-action.yml
b/.github/workflows/backend-integration-test-action.yml
index b5de5e641e..d5cf1bd57e 100644
--- a/.github/workflows/backend-integration-test-action.yml
+++ b/.github/workflows/backend-integration-test-action.yml
@@ -72,7 +72,7 @@ jobs:
EXCLUDE_CONTRIB_TESTS="$EXCLUDE_CONTRIB_TESTS -x
:catalogs-contrib:$module:test"
done
- ./gradlew test -PskipTests -PtestMode=${{ inputs.test-mode }}
-PjdbcBackend=${{ inputs.backend }} -PskipDockerTests=false -PskipWeb=true \
+ ./gradlew test :iceberg:iceberg-rest-server:experimentalTest
-PskipTests -PtestMode=${{ inputs.test-mode }} -PjdbcBackend=${{ inputs.backend
}} -PskipDockerTests=false -PskipWeb=true \
-x :web:web:test -x :web:integration-test:test -x :web-v2:web:test
-x :web-v2:integration-test:test -x :clients:client-python:test \
-x :flink-connector:flink-common:test \
-x :flink-connector:flink-1.19:test -x
:flink-connector:flink-runtime-1.19:test \
diff --git a/NOTICE.bin b/NOTICE.bin
index a400cf9f48..dedddb7fc6 100644
--- a/NOTICE.bin
+++ b/NOTICE.bin
@@ -31,7 +31,7 @@ AWS EventStream for Java
Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
Apache Iceberg
-Copyright 2017-2022 The Apache Software Foundation
+Copyright 2017-2026 The Apache Software Foundation
Apache Kafka
Copyright 2011-2023 The Apache Software Foundation
@@ -603,6 +603,14 @@ This project includes code from Kite, developed at
Cloudera, Inc. with
the following copyright notice:
Copyright 2013 Cloudera Inc.
+This project includes code from Presto, developed at Facebook, with the
+following copyright notice:
+ Copyright 2016 Facebook and contributors.
+
+This project includes code from Delta Lake, developed at The Delta Lake
Project,
+with the following copyright notice:
+ Copyright (2021) The Delta Lake Project Authors.
+
Apache Ivy (TM)
Copyright 2007-2014 The Apache Software Foundation
diff --git a/build.gradle.kts b/build.gradle.kts
index ea7cd68e2b..c2daee27e3 100644
--- a/build.gradle.kts
+++ b/build.gradle.kts
@@ -1183,6 +1183,7 @@ tasks {
"copyJobsLib",
":authorizations:copyLibAndConfig",
":iceberg:iceberg-rest-server:copyLibAndConfigs",
+ ":iceberg:iceberg-rest-server-experiment:copyLibAndConfigs",
":lance:lance-rest-server:copyLibAndConfigs",
":maintenance:optimizer:copyLibAndConfigs",
":plugins:idp-basic:copyLibAndConfigs",
diff --git a/clients/client-python/gravitino/utils/http_client.py
b/clients/client-python/gravitino/utils/http_client.py
index a0caada82d..b9cac49891 100644
--- a/clients/client-python/gravitino/utils/http_client.py
+++ b/clients/client-python/gravitino/utils/http_client.py
@@ -172,7 +172,7 @@ class HTTPClient:
if isinstance(data, (str, bytes)):
request_data = data.encode("utf-8") if isinstance(data, str) else
data
- self._update_headers(self.JSON_HEADER)
+ request_headers = self._update_headers(self.JSON_HEADER)
elif data:
request_data = urlencode(data.to_dict()).encode()
request_headers = self._update_headers(self.FORMDATA_HEADER)
diff --git a/clients/client-python/tests/unittests/test_http_client_headers.py
b/clients/client-python/tests/unittests/test_http_client_headers.py
index e5ace7862a..65882cc5f2 100644
--- a/clients/client-python/tests/unittests/test_http_client_headers.py
+++ b/clients/client-python/tests/unittests/test_http_client_headers.py
@@ -123,6 +123,27 @@ class TestHTTPClientHeaders(unittest.TestCase):
self.assertEqual("configured/type", configured_headers["Content-Type"])
self.assertEqual(configured_headers, client.request_headers)
+ def test_raw_data_uses_request_local_headers(self):
+ client = HTTPClient(
+ "http://localhost:8090";, request_headers={"X-Configured":
"configured"}
+ )
+
+ with patch("gravitino.utils.http_client.build_opener") as build_opener:
+ requests = self._capture_requests(
+ build_opener, [self._success_response(),
self._success_response()]
+ )
+ client.post("/string", data="raw", headers={"X-Per-Request":
"string"})
+ client.post("/bytes", data=b"raw")
+
+ self.assertEqual("raw", requests[0].data.decode("utf-8"))
+ self.assertEqual("string", self._header_value(requests[0],
"X-Per-Request"))
+ self.assertIsNone(self._header_value(requests[1], "X-Per-Request"))
+ for request in requests:
+ self.assertEqual("configured", self._header_value(request,
"X-Configured"))
+ self.assertEqual(
+ "application/json", self._header_value(request, "Content-Type")
+ )
+
def test_failed_request_headers_do_not_leak(self):
client = HTTPClient("http://localhost:8090";)
error_body = (
diff --git a/conf/gravitino.conf.template b/conf/gravitino.conf.template
index af8a6c0651..ef26e8a0b5 100644
--- a/conf/gravitino.conf.template
+++ b/conf/gravitino.conf.template
@@ -111,6 +111,8 @@ gravitino.authorization.serviceAdmins = anonymous
gravitino.auxService.names = iceberg-rest,lance-rest
# Iceberg REST service classpath
gravitino.iceberg-rest.classpath = iceberg-rest-server/libs,
iceberg-rest-server/conf
+# To use the experimental Iceberg dependencies, replace the classpath above
with:
+# gravitino.iceberg-rest.classpath = iceberg-rest-server-experiment/libs,
iceberg-rest-server-experiment/conf
# Iceberg REST service host
gravitino.iceberg-rest.host = 0.0.0.0
# Iceberg REST service http port
diff --git a/docs/iceberg-rest-service.md b/docs/iceberg-rest-service.md
index 098c6a7feb..d259b71ff6 100644
--- a/docs/iceberg-rest-service.md
+++ b/docs/iceberg-rest-service.md
@@ -102,14 +102,30 @@ The server-level
`gravitino.fetchFile.blockUnsafeRemoteUri` configuration contro
#### Auxiliary Service
-| Configuration item | Description
| Default value | Required |
-|------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|
-| `gravitino.auxService.names` | The auxiliary service name of the
Gravitino Iceberg REST catalog service. Use **`iceberg-rest`**.
| (none) | Yes |
-| `gravitino.iceberg-rest.classpath` | The classpath of the Gravitino Iceberg
REST catalog service; includes the directory containing jars and configuration.
It supports both absolute and relative paths, for example,
`iceberg-rest-server/libs, iceberg-rest-server/conf` | (none) | Yes
|
+| Configuration item | Description
| Default value | Required |
+|------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|
+| `gravitino.auxService.names` | The auxiliary service name of the
Gravitino Iceberg REST catalog service. Use **`iceberg-rest`**.
| (none) | Yes
|
+| `gravitino.iceberg-rest.classpath` | The classpath of the Gravitino Iceberg
REST catalog service. Use `iceberg-rest-server/libs, iceberg-rest-server/conf`
for the regular Iceberg dependencies or `iceberg-rest-server-experiment/libs,
iceberg-rest-server-experiment/conf` for the experimental dependencies. It
supports both absolute and relative paths. | (none) | Yes |
These settings apply only to `gravitino.conf`.
Do not add them to the standalone server configuration.
+The auxiliary service name and all Iceberg REST configuration keys remain the
same when using the
+experimental dependencies. Only select the experimental classpath:
+
+```properties
+gravitino.auxService.names = iceberg-rest
+gravitino.iceberg-rest.classpath = iceberg-rest-server-experiment/libs,
iceberg-rest-server-experiment/conf
+```
+
+The selected directory is prepended to the Iceberg REST auxiliary service
classpath. Only one of
+the regular and experimental directories can be active at a time. The
selection takes effect when
+Iceberg classes are absent from the Gravitino server's base classpath; the
binary distribution is
+packaged this way. The service logs the loaded Iceberg version and code source
during startup so
+the effective runtime can be verified. The experimental directory uses the
Datastrato Iceberg
+distribution published under the `com.datastrato` Maven group. Its version is
pinned by
+`datastrato-iceberg` in `gradle/libs.versions.toml`, independently of the
regular Iceberg version.
+
#### HTTP Server
| Configuration item | Description
| Default value
| Required |
diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml
index 9b2f9a9168..589b91b10d 100644
--- a/gradle/libs.versions.toml
+++ b/gradle/libs.versions.toml
@@ -71,6 +71,8 @@ commons-configuration1 = "1.6"
commons-dbcp2 = "2.11.0"
caffeine = "2.9.3"
iceberg = '1.11.0'
+datastrato-iceberg = "1.13.0"
+experimental-iceberg-git-commit = "e6118db078e09d960600c3c302e0f220457cb388"
# Last Iceberg release compatible with Spark 3.3.
iceberg4spark33 = "1.8.1"
iceberg4spark35 = "1.11.0"
@@ -82,6 +84,7 @@ spark33 = "3.3.4"
spark34 = "3.4.3"
spark35 = "3.5.3"
spark40 = "4.0.3"
+spark41 = "4.1.3"
kyuubi4spark = "1.12.0"
kyuubi4authz = "1.10.2"
trino = '435'
diff --git a/iceberg/iceberg-rest-server-experiment/build.gradle.kts
b/iceberg/iceberg-rest-server-experiment/build.gradle.kts
new file mode 100644
index 0000000000..b4be172ce8
--- /dev/null
+++ b/iceberg/iceberg-rest-server-experiment/build.gradle.kts
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+description = "iceberg-rest-server-experiment"
+
+plugins {
+ `maven-publish`
+ id("java")
+ id("idea")
+}
+
+val datastratoIcebergVersion: String = libs.versions.datastrato.iceberg.get()
+val caffeineVersion: String = libs.versions.caffeine.get()
+val jacksonVersion: String = libs.versions.jackson.get()
+
+configurations.configureEach {
+ resolutionStrategy {
+ force(
+ "com.github.ben-manes.caffeine:caffeine:$caffeineVersion",
+ "com.fasterxml.jackson.core:jackson-annotations:$jacksonVersion",
+ "com.fasterxml.jackson.core:jackson-core:$jacksonVersion",
+ "com.fasterxml.jackson.core:jackson-databind:$jacksonVersion",
+ "com.fasterxml.jackson.datatype:jackson-datatype-jdk8:$jacksonVersion",
+ "com.fasterxml.jackson.datatype:jackson-datatype-jsr310:$jacksonVersion"
+ )
+ eachDependency {
+ if (
+ requested.name.startsWith("iceberg-") &&
+ (requested.group == "org.apache.iceberg" || requested.group ==
"com.datastrato")
+ ) {
+
useTarget("com.datastrato:experimental-${requested.name}:$datastratoIcebergVersion")
+ because("Use the Datastrato Iceberg distribution in the experimental
auxiliary service")
+ }
+ }
+ }
+}
+
+dependencies {
+ implementation(project(":iceberg:iceberg-rest-server"))
+ implementation(libs.bundles.iceberg)
+}
+
+tasks {
+ val copyDepends by registering(Sync::class) {
+ dependsOn(":iceberg:iceberg-rest-server:copyDepends")
+ from(configurations.runtimeClasspath)
+ into(layout.buildDirectory.dir("dependencies"))
+ }
+
+ val copyLibs by registering(Sync::class) {
+ dependsOn(copyDepends, "build")
+ from(jar)
+ from(copyDepends)
+ into("$rootDir/distribution/package/iceberg-rest-server-experiment/libs")
+ }
+
+ val copyConfigs by registering(Copy::class) {
+ from(project(":iceberg:iceberg-rest-server").file("src/main/resources"))
+ into("$rootDir/distribution/package/iceberg-rest-server-experiment/conf")
+
+ include("core-site.xml.template")
+ include("hdfs-site.xml.template")
+
+ rename { original ->
+ if (original.endsWith(".template")) {
+ original.replace(".template", "")
+ } else {
+ original
+ }
+ }
+
+ fileMode = 0b111101101
+ }
+
+ register("copyLibAndConfigs") {
+ dependsOn(copyLibs, copyConfigs)
+ }
+}
diff --git a/iceberg/iceberg-rest-server/build.gradle.kts
b/iceberg/iceberg-rest-server/build.gradle.kts
index 20a8abe8a9..9bc8637480 100644
--- a/iceberg/iceberg-rest-server/build.gradle.kts
+++ b/iceberg/iceberg-rest-server/build.gradle.kts
@@ -16,6 +16,8 @@
* specific language governing permissions and limitations
* under the License.
*/
+import org.gradle.api.attributes.Attribute
+
description = "iceberg-rest-service"
plugins {
@@ -27,8 +29,75 @@ plugins {
val scalaVersion: String = project.properties["scalaVersion"] as? String ?:
extra["defaultScalaVersion"].toString()
val sparkVersion: String = libs.versions.spark34.get()
val sparkMajorVersion: String = sparkVersion.substringBeforeLast(".")
+val experimentalScalaVersion = "2.13"
+val experimentalSparkVersion: String = libs.versions.spark41.get()
+val experimentalSparkMajorVersion: String =
experimentalSparkVersion.substringBeforeLast(".")
val icebergVersion: String = libs.versions.iceberg.get()
+val datastratoIcebergVersion: String = libs.versions.datastrato.iceberg.get()
+val datastratoIcebergGitCommit: String =
libs.versions.experimental.iceberg.git.commit.get()
val scalaCollectionCompatVersion: String =
libs.versions.scala.collection.compat.get()
+val caffeineVersion: String = libs.versions.caffeine.get()
+val jerseyVersion: String = libs.versions.jersey.get()
+val javaxWsRsVersion: String = libs.versions.javax.ws.rs.api.get()
+val hk2Version = "2.6.1"
+val jakartaValidationVersion = "2.0.2"
+
+val testRuntimeClasspath = configurations.testRuntimeClasspath.get()
+val experimentalTestRuntimeClasspath by configurations.creating {
+ isCanBeConsumed = false
+ isCanBeResolved = true
+ extendsFrom(testRuntimeClasspath)
+ attributes {
+ testRuntimeClasspath.attributes.keySet().forEach { key ->
+ @Suppress("UNCHECKED_CAST")
+ val typedKey = key as Attribute<Any>
+ attribute(typedKey,
testRuntimeClasspath.attributes.getAttribute(typedKey)!!)
+ }
+ }
+ resolutionStrategy {
+ force(
+ "com.github.ben-manes.caffeine:caffeine:$caffeineVersion",
+ "javax.ws.rs:javax.ws.rs-api:$javaxWsRsVersion",
+ "org.glassfish.jersey.core:jersey-server:$jerseyVersion",
+ "org.glassfish.jersey.core:jersey-common:$jerseyVersion",
+ "org.glassfish.jersey.core:jersey-client:$jerseyVersion",
+
"org.glassfish.jersey.containers:jersey-container-servlet:$jerseyVersion",
+
"org.glassfish.jersey.containers:jersey-container-servlet-core:$jerseyVersion",
+
"org.glassfish.jersey.containers:jersey-container-jetty-http:$jerseyVersion",
+ "org.glassfish.jersey.inject:jersey-hk2:$jerseyVersion",
+ "org.glassfish.hk2:hk2-api:$hk2Version",
+ "org.glassfish.hk2:hk2-locator:$hk2Version",
+ "org.glassfish.hk2:hk2-utils:$hk2Version",
+ "org.glassfish.hk2.external:aopalliance-repackaged:$hk2Version",
+ "org.glassfish.hk2.external:jakarta.inject:$hk2Version",
+ "jakarta.validation:jakarta.validation-api:$jakartaValidationVersion"
+ )
+ eachDependency {
+ if (requested.group == "org.apache.spark") {
+ val experimentalSparkModule = requested.name.replace("_$scalaVersion",
"_$experimentalScalaVersion")
+
useTarget("${requested.group}:$experimentalSparkModule:$experimentalSparkVersion")
+ because("The experimental Iceberg Spark runtime requires Spark 4.1
with Scala 2.13")
+ } else if (
+ requested.group == "org.scala-lang.modules" &&
requested.name.endsWith("_$scalaVersion")
+ ) {
+ val experimentalScalaModule = requested.name.replace("_$scalaVersion",
"_$experimentalScalaVersion")
+
useTarget("${requested.group}:$experimentalScalaModule:${requested.version}")
+ because("Spark 4.1 uses Scala 2.13")
+ } else if (requested.name.startsWith("iceberg-spark-runtime-")) {
+ useTarget(
+
"com.datastrato:experimental-iceberg-spark-runtime-${experimentalSparkMajorVersion}_$experimentalScalaVersion:$datastratoIcebergVersion"
+ )
+ because("Run experimental Iceberg integration tests with the matching
Spark runtime")
+ } else if (
+ requested.name.startsWith("iceberg-") &&
+ (requested.group == "org.apache.iceberg" || requested.group ==
"com.datastrato")
+ ) {
+
useTarget("com.datastrato:experimental-${requested.name}:$datastratoIcebergVersion")
+ because("Run experimental Iceberg integration tests with the
Datastrato distribution")
+ }
+ }
+ }
+}
dependencies {
implementation(project(":api"))
@@ -98,6 +167,7 @@ dependencies {
}
testImplementation(libs.awaitility)
+ testImplementation(libs.javax.ws.rs.api)
// At runtime use the SDK supplied by iceberg-aws-bundle, keeping its S3 and
STS versions aligned.
testCompileOnly(libs.aws.s3)
testImplementation(libs.h2db)
@@ -199,13 +269,47 @@ tasks {
}
}
+val experimentalTest by tasks.registering(Test::class) {
+ description = "Runs Iceberg REST tests that require the experimental Iceberg
runtime"
+ group = "verification"
+ testClassesDirs = sourceSets.test.get().output.classesDirs
+ classpath =
+ sourceSets.main.get().output +
+ sourceSets.test.get().output +
+ experimentalTestRuntimeClasspath
+ systemProperty("datastratoIcebergGitCommit", datastratoIcebergGitCommit)
+ systemProperty("spark.ui.enabled", "false")
+ setIncludes(
+ setOf(
+ "**/IcebergExperimentalAuthorizationIT.class"
+ )
+ )
+ filter {
+ includeTestsMatching(
+
"org.apache.gravitino.iceberg.integration.test.IcebergExperimentalAuthorizationIT"
+ )
+ }
+ mustRunAfter(tasks.test)
+ dependsOn(
+ tasks.testClasses,
+ tasks.jar,
+ ":catalogs:catalog-lakehouse-iceberg:jar",
+ ":iceberg:iceberg-rest-server-experiment:copyDepends",
+ ":iceberg:iceberg-rest-server-experiment:jar"
+ )
+}
+
tasks.test {
val skipITs = project.hasProperty("skipITs")
+ exclude("**/IcebergExperimentalAuthorizationIT.class")
if (skipITs) {
// Exclude integration tests
exclude("**/integration/test/**")
} else {
- dependsOn(tasks.jar)
+ dependsOn(
+ tasks.jar,
+ ":catalogs:catalog-lakehouse-iceberg:jar"
+ )
}
}
diff --git
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/RESTService.java
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/RESTService.java
index 1786ba3e0e..99077475d9 100644
---
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/RESTService.java
+++
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/RESTService.java
@@ -19,6 +19,7 @@
package org.apache.gravitino.iceberg;
import com.google.common.collect.Lists;
+import java.security.CodeSource;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
@@ -65,6 +66,7 @@ import org.apache.gravitino.server.web.JettyServerConfig;
import org.apache.gravitino.server.web.OutOfMemoryErrorListener;
import org.apache.gravitino.server.web.RequestContextFilter;
import
org.apache.gravitino.server.web.filter.IcebergRESTAuthInterceptionService;
+import org.apache.iceberg.IcebergBuild;
import org.glassfish.hk2.api.InterceptionService;
import org.glassfish.hk2.utilities.binding.AbstractBinder;
import org.glassfish.jersey.jackson.JacksonFeature;
@@ -268,6 +270,11 @@ public class RESTService implements
GravitinoAuxiliaryService {
@Override
public void serviceInit(Map<String, String> properties, boolean auxMode) {
this.auxMode = auxMode;
+ CodeSource icebergCodeSource =
IcebergBuild.class.getProtectionDomain().getCodeSource();
+ LOG.info(
+ "Iceberg REST service runtime: {}, loaded from {}",
+ IcebergBuild.fullVersion(),
+ icebergCodeSource == null ? "unknown" :
icebergCodeSource.getLocation());
IcebergConfig icebergConfig = new IcebergConfig(properties);
initServer(icebergConfig);
LOG.info("Iceberg REST service init. Running in {} mode", auxMode ?
"auxiliary" : "standalone");
diff --git
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/IcebergAuthorizationIT.java
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/IcebergAuthorizationIT.java
index bd69a06659..aba2d127aa 100644
---
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/IcebergAuthorizationIT.java
+++
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/IcebergAuthorizationIT.java
@@ -95,7 +95,7 @@ public class IcebergAuthorizationIT extends BaseIT {
@BeforeAll
@Override
public void startIntegrationTest() throws Exception {
-
containerSuite.startPostgreSQLContainer(TestDatabaseName.PG_ICEBERG_AUTHZ_IT);
+ containerSuite.startPostgreSQLContainer(getTestDatabaseName());
startGravitinoServerWithIcebergREST();
initMetalakeAndCatalog();
initSparkEnv();
@@ -219,6 +219,11 @@ public class IcebergAuthorizationIT extends BaseIT {
return null;
}
+ /** Returns the PostgreSQL database used by this authorization test suite. */
+ protected TestDatabaseName getTestDatabaseName() {
+ return TestDatabaseName.PG_ICEBERG_AUTHZ_IT;
+ }
+
void revokeUserRoles() {
List<String> roles =
metalakeClientWithAllPrivilege.getUser(NORMAL_USER).roles();
if (roles.size() > 0) {
@@ -341,7 +346,7 @@ public class IcebergAuthorizationIT extends BaseIT {
}
private String getPGUri() {
- return
containerSuite.getPostgreSQLContainer().getJdbcUrl(TestDatabaseName.PG_ICEBERG_AUTHZ_IT);
+ return
containerSuite.getPostgreSQLContainer().getJdbcUrl(getTestDatabaseName());
}
private String getPGUser() {
diff --git
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/IcebergExperimentalAuthorizationIT.java
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/IcebergExperimentalAuthorizationIT.java
new file mode 100644
index 0000000000..95de77ca7a
--- /dev/null
+++
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/IcebergExperimentalAuthorizationIT.java
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.iceberg.integration.test;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import org.apache.gravitino.integration.test.util.ITUtils;
+import org.apache.gravitino.integration.test.util.TestDatabaseName;
+import org.apache.iceberg.IcebergBuild;
+import org.apache.spark.sql.AnalysisException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.function.Executable;
+
+/**
+ * Integration test for Spark authorization through the Iceberg REST service
loaded from the
+ * experimental classpath.
+ *
+ * <p>The inherited tests exercise table creation, loading, updates, deletion,
listing, renaming,
+ * nested namespaces, and table privilege enforcement against the experimental
Iceberg artifacts.
+ */
+@Tag("gravitino-docker-test")
+public class IcebergExperimentalAuthorizationIT extends
IcebergTableAuthorizationIT {
+
+ private static final String EXPERIMENTAL_SERVER_DIRECTORY =
"iceberg-rest-server-experiment";
+
+ @BeforeAll
+ @Override
+ public void startIntegrationTest() throws Exception {
+ Assertions.assertEquals(
+ System.getProperty("datastratoIcebergGitCommit"),
+ IcebergBuild.gitCommitId(),
+ "The experimental integration test must run with the Datastrato
Iceberg runtime");
+
+ String rootDirectory = System.getenv("GRAVITINO_ROOT_DIR");
+ String experimentalServerHome =
+ Paths.get(rootDirectory, "iceberg",
EXPERIMENTAL_SERVER_DIRECTORY).toString();
+ customConfigs.put(
+ "gravitino.iceberg-rest.classpath",
+ String.join(
+ ",",
+ Paths.get(experimentalServerHome, "build", "libs").toString(),
+ Paths.get(experimentalServerHome, "build",
"dependencies").toString()));
+
+ Path serverLog = getServerLog();
+ long logOffset = Files.exists(serverLog) ? Files.size(serverLog) : 0;
+ super.startIntegrationTest();
+
+ String startupLog = readLogFrom(serverLog, logOffset);
+ Assertions.assertTrue(
+ startupLog.contains(System.getProperty("datastratoIcebergGitCommit")),
+ "The Iceberg REST service must load the Datastrato Iceberg runtime");
+ if
(ITUtils.DEPLOY_TEST_MODE.equals(System.getProperty(ITUtils.TEST_MODE))) {
+ Assertions.assertTrue(
+ startupLog.contains(EXPERIMENTAL_SERVER_DIRECTORY),
+ "The deployed Iceberg REST service must load its configured
experimental classpath");
+ }
+ }
+
+ @Override
+ protected TestDatabaseName getTestDatabaseName() {
+ return TestDatabaseName.PG_ICEBERG_EXPERIMENTAL_AUTHZ_IT;
+ }
+
+ /**
+ * Spark 4.1 wraps a missing table in a subclass of {@link
AnalysisException}, unlike Spark 3.4.
+ *
+ * @param executable the table load operation
+ */
+ @Override
+ protected void assertNoSuchTableException(Executable executable) {
+ Assertions.assertThrows(AnalysisException.class, executable);
+ }
+
+ private Path getServerLog() {
+ if
(ITUtils.DEPLOY_TEST_MODE.equals(System.getProperty(ITUtils.TEST_MODE))) {
+ return Paths.get(System.getenv("GRAVITINO_HOME"), "logs",
"gravitino-server.log");
+ }
+ return Paths.get(System.getProperty("gravitino.log.path"));
+ }
+
+ private String readLogFrom(Path logFile, long offset) throws IOException {
+ byte[] logContent = Files.readAllBytes(logFile);
+ int start = logContent.length >= offset ? Math.toIntExact(offset) : 0;
+ return new String(logContent, start, logContent.length - start,
StandardCharsets.UTF_8);
+ }
+}
diff --git
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/IcebergTableAuthorizationIT.java
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/IcebergTableAuthorizationIT.java
index 0d19211fc7..bc2d4c12af 100644
---
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/IcebergTableAuthorizationIT.java
+++
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/IcebergTableAuthorizationIT.java
@@ -45,6 +45,7 @@ import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.function.Executable;
@Tag("gravitino-docker-test")
public class IcebergTableAuthorizationIT extends IcebergAuthorizationIT {
@@ -57,6 +58,15 @@ public class IcebergTableAuthorizationIT extends
IcebergAuthorizationIT {
catalogClientWithAllPrivilege.asSchemas().createSchema(SCHEMA_NAME,
"test", new HashMap<>());
}
+ /**
+ * Verifies the exception thrown when an authorized user loads a table that
does not exist.
+ *
+ * @param executable the table load operation
+ */
+ protected void assertNoSuchTableException(Executable executable) {
+ Assertions.assertThrowsExactly(AnalysisException.class, executable);
+ }
+
@BeforeEach
void revokePrivilege() {
revokeUserRoles();
@@ -272,8 +282,7 @@ public class IcebergTableAuthorizationIT extends
IcebergAuthorizationIT {
setSchemaOwner(NORMAL_USER);
Assertions.assertDoesNotThrow(() -> sql("DESC TABLE %s", tableName));
// With ownership: non-existent table returns 404 (AnalysisException)
- Assertions.assertThrowsExactly(
- AnalysisException.class, () -> sql("DESC TABLE %s_not_exist",
tableName));
+ assertNoSuchTableException(() -> sql("DESC TABLE %s_not_exist",
tableName));
setSchemaOwner(SUPER_USER);
Assertions.assertThrowsExactly(ForbiddenException.class, () -> sql("DESC
TABLE %s", tableName));
diff --git
a/integration-test-common/src/test/java/org/apache/gravitino/integration/test/util/TestDatabaseName.java
b/integration-test-common/src/test/java/org/apache/gravitino/integration/test/util/TestDatabaseName.java
index e127c0bf6f..52f1c8c4f7 100644
---
a/integration-test-common/src/test/java/org/apache/gravitino/integration/test/util/TestDatabaseName.java
+++
b/integration-test-common/src/test/java/org/apache/gravitino/integration/test/util/TestDatabaseName.java
@@ -133,6 +133,9 @@ public enum TestDatabaseName {
},
PG_ICEBERG_AUTHZ_IT,
+ /** Represents the PostgreSQL database for experimental Iceberg
authorization tests. */
+ PG_ICEBERG_EXPERIMENTAL_AUTHZ_IT,
+
PG_ICEBERG_ASYNC_CLEANUP_IT,
CLICKHOUSE_CLICKHOUSE_ABSTRACT_IT,
diff --git a/settings.gradle.kts b/settings.gradle.kts
index 511cdd025d..4d15b84397 100644
--- a/settings.gradle.kts
+++ b/settings.gradle.kts
@@ -63,6 +63,7 @@ include(
// in-tree for reference; see clients/filesystem-fuse/README.md for details.
include("iceberg:iceberg-common")
include("iceberg:iceberg-rest-server")
+include("iceberg:iceberg-rest-server-experiment")
include("iceberg:iceberg-rest-trino-it")
include("lance:lance-common")
include("lance:lance-rest-server")
