Re: [PR] feat(oceanbase-catalog): Support schema operations for OceanBase JDBC catalog. [gravitino]
mchades commented on PR #5013: URL: https://github.com/apache/gravitino/pull/5013#issuecomment-2408316508 > > LGTM. Thanks for your contributions and sorry for my later response. > > I think the next PR should be `support simple table operations for Oceanbase catalog` > > I think it might be a better choice to implement the `org.apache.gravitino.catalog.oceanbase.converter` package first. Both schema and table operations depend on it, and we can test it while testing schema and table operations.What do you think? ok, plz go ahead -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5057] Added first part of CLI code [gravitino]
justinmclean commented on PR #5058: URL: https://github.com/apache/gravitino/pull/5058#issuecomment-2408393991 @shaofengshi @diqiu50 @yuqi1129 @mchades if you could review agian that would be appreciated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [#5059] Add commands to create, delete and modify metalakes, catalogs and schema. [gravitino]
justinmclean opened a new pull request, #5117: URL: https://github.com/apache/gravitino/pull/5117 ### What changes were proposed in this pull request? Add commands to create, delete and modify metalakes, catalogs and schema. ### Why are the changes needed? stage 2 of the CLI. Fix: #5059 ### Does this PR introduce _any_ user-facing change? Add new CLI features. ### How was this patch tested? Tests pass locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4951] improvement(test): Reduce fields shares between different IT to make AbstractIT more independent. [gravitino]
yuqi1129 commented on code in PR #4996:
URL: https://github.com/apache/gravitino/pull/4996#discussion_r1796601197
##
trino-connector/integration-test/src/test/java/org/apache/gravitino/trino/connector/integration/test/TrinoQueryITBase.java:
##
@@ -68,14 +68,17 @@ public class TrinoQueryITBase {
protected static final String metalakeName = "test";
protected static GravitinoMetalake metalake;
- private static void setEnv() throws Exception {
+ private static AbstractIT abstractIT;
+
+ private void setEnv() throws Exception {
+abstractIT = new AbstractIT();
Review Comment:
I have created a new class `BaseIT` that merely inherit `AbstractIT`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] [Bug report] 'ndbinfo','performance_schema','FEDERATED' engines are not supported in Gravitino MySQL catalog [gravitino]
yuqi1129 commented on issue #5100: URL: https://github.com/apache/gravitino/issues/5100#issuecomment-2406965454 @danhuawang Starting from MySQL 5.5, the FEDERATED engine is installed by default but is not enabled, so you need to enable the `FEDERATED` first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4952] feat(hudi-catalog): add implementation of HMSBackend for Hudi catalog [gravitino]
jerryshao merged PR #4942: URL: https://github.com/apache/gravitino/pull/4942 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Subtask] Support read operations of HMSBackend for Hudi catalog [gravitino]
jerryshao closed issue #4952: [Subtask] Support read operations of HMSBackend for Hudi catalog URL: https://github.com/apache/gravitino/issues/4952 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(gravitino) branch main updated: [#4952] feat(hudi-catalog): add implementation of HMSBackend for Hudi catalog (#4942)
This is an automated email from the ASF dual-hosted git repository.
jshao pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new fdb07abfb [#4952] feat(hudi-catalog): add implementation of HMSBackend
for Hudi catalog (#4942)
fdb07abfb is described below
commit fdb07abfba0676a03c7378ef1f7382471050ad38
Author: mchades
AuthorDate: Fri Oct 11 17:04:44 2024 +0800
[#4952] feat(hudi-catalog): add implementation of HMSBackend for Hudi
catalog (#4942)
### What changes were proposed in this pull request?
support read operations for Hudi catalog HMS backend
### Why are the changes needed?
Fix: #4952
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
UTs added
---
catalogs/catalog-lakehouse-hudi/build.gradle.kts | 99 +++
.../lakehouse/hudi/HudiCatalogOperations.java | 5 +-
.../hudi/HudiCatalogPropertiesMetadata.java| 33 +++-
.../catalog/lakehouse/hudi/HudiColumn.java | 61 +++
.../catalog/lakehouse/hudi/HudiSchema.java | 5 +
.../hudi/HudiSchemaPropertiesMetadata.java | 17 +-
.../catalog/lakehouse/hudi/HudiTable.java | 5 +
.../hudi/HudiTablePropertiesMetadata.java | 36 +++-
.../lakehouse/hudi/backend/HudiCatalogBackend.java | 10 +-
.../lakehouse/hudi/backend/hms/HudiHMSBackend.java | 3 +-
.../hudi/backend/hms/HudiHMSBackendOps.java| 150 -
.../lakehouse/hudi/backend/hms/HudiHMSSchema.java | 20 ++-
.../lakehouse/hudi/backend/hms/HudiHMSTable.java | 30 +++-
.../catalog/lakehouse/hudi/utils/CatalogUtils.java | 18 +-
.../catalog/lakehouse/hudi/TestHudiCatalog.java| 62 +++
.../lakehouse/hudi/TestHudiCatalogOperations.java | 127 ++
.../catalog/lakehouse/hudi/TestHudiSchema.java}| 24 +--
.../catalog/lakehouse/hudi/TestHudiTable.java} | 28 +---
.../hudi/backend/hms/TestHudiHMSBackend.java} | 26 ++-
.../hudi/backend/hms/TestHudiHMSBackendOps.java| 184 +
.../lakehouse/hudi/ops/InMemoryBackendOps.java}| 80 +++--
.../lakehouse/hudi/utils/TestCatalogUtils.java}| 22 ++-
.../hive/converter/HiveTableConverter.java | 2 +-
23 files changed, 935 insertions(+), 112 deletions(-)
diff --git a/catalogs/catalog-lakehouse-hudi/build.gradle.kts
b/catalogs/catalog-lakehouse-hudi/build.gradle.kts
index 78ff2f8b1..eef90f029 100644
--- a/catalogs/catalog-lakehouse-hudi/build.gradle.kts
+++ b/catalogs/catalog-lakehouse-hudi/build.gradle.kts
@@ -24,18 +24,117 @@ plugins {
id("idea")
}
+val scalaVersion: String = project.properties["scalaVersion"] as? String ?:
extra["defaultScalaVersion"].toString()
+val fullSparkVersion: String = libs.versions.spark34.get()
+val sparkVersion = fullSparkVersion.split(".").take(2).joinToString(".")
+
dependencies {
implementation(project(":api")) {
exclude(group = "*")
}
+ implementation(project(":common")) {
+exclude(group = "*")
+ }
+ implementation(project(":catalogs:hive-metastore-common"))
implementation(project(":core")) {
exclude(group = "*")
}
implementation(libs.guava)
+ implementation(libs.hive2.exec) {
+artifact {
+ classifier = "core"
+}
+exclude("com.google.code.findbugs", "jsr305")
+exclude("com.google.protobuf")
+exclude("org.apache.avro")
+exclude("org.apache.ant")
+exclude("org.apache.calcite")
+exclude("org.apache.calcite.avatica")
+exclude("org.apache.curator")
+exclude("org.apache.derby")
+exclude("org.apache.hadoop", "hadoop-yarn-server-resourcemanager")
+exclude("org.apache.hive", "hive-llap-tez")
+exclude("org.apache.hive", "hive-vector-code-gen")
+exclude("org.apache.ivy")
+exclude("org.apache.logging.log4j")
+exclude("org.apache.zookeeper")
+exclude("org.codehaus.groovy", "groovy-all")
+exclude("org.datanucleus", "datanucleus-core")
+exclude("org.eclipse.jetty.aggregate", "jetty-all")
+exclude("org.eclipse.jetty.orbit", "javax.servlet")
+exclude("org.openjdk.jol")
+exclude("org.pentaho")
+exclude("org.slf4j")
+ }
implementation(libs.hive2.metastore) {
+exclude("ant")
+exclude("co.cask.tephra")
+exclude("com.github.joshelser")
+exclude("com.google.code.findbugs", "jsr305")
+exclude("com.google.code.findbugs", "sr305")
+exclude("com.tdunning", "json")
+exclude("com.zaxxer", "HikariCP")
+exclude("io.dropwizard.metricss")
+exclude("javax.transaction", "transaction-api")
+exclude("org.apache.ant")
+exclude("org.apache.avro")
+exclude("org.apache.curator")
+exclude("org.apache.derby")
+exclude("org.apache.hadoop", "hadoop-yarn-server-resourcemanager")
+exclude("org.apache.hbase")
+exclude("org.apache.logging.log4j")
+exclude("org.apache.par
[PR] [#5101] improvement(docs): Fix the document error about access control [gravitino]
jerqi opened a new pull request, #5102: URL: https://github.com/apache/gravitino/pull/5102 ### What changes were proposed in this pull request? Fix the document error about access control ### Why are the changes needed? Fix: #5101 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Just docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4867] feat(core): Add storage support for columns in Gravitino [gravitino]
jerryshao commented on code in PR #5078: URL: https://github.com/apache/gravitino/pull/5078#discussion_r1796518199 ## scripts/mysql/upgrade-0.6.0-to-0.7.0-mysql.sql: ## @@ -0,0 +1,41 @@ +-- +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file-- +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"). You may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. +-- +CREATE TABLE IF NOT EXISTS `table_column_version_info` ( +`id` BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT COMMENT 'auto increment id', Review Comment: column_id will be duplicated when we update the column multiple times, so it cannot be a primary key. You can check the current column table design, it uses MVCC to manage the column changes, so for one column, it will have multiple same column_id records with different fields in this table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4867] feat(core): Add storage support for columns in Gravitino [gravitino]
jerryshao commented on code in PR #5078: URL: https://github.com/apache/gravitino/pull/5078#discussion_r1796521933 ## scripts/mysql/upgrade-0.6.0-to-0.7.0-mysql.sql: ## @@ -0,0 +1,41 @@ +-- +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file-- +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"). You may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. +-- +CREATE TABLE IF NOT EXISTS `table_column_version_info` ( +`id` BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT COMMENT 'auto increment id', +`metalake_id` BIGINT(20) UNSIGNED NOT NULL COMMENT 'metalake id', +`catalog_id` BIGINT(20) UNSIGNED NOT NULL COMMENT 'catalog id', +`schema_id` BIGINT(20) UNSIGNED NOT NULL COMMENT 'schema id', +`table_id` BIGINT(20) UNSIGNED NOT NULL COMMENT 'table id', +`table_version` INT UNSIGNED NOT NULL COMMENT 'table version', +`column_id` BIGINT(20) UNSIGNED NOT NULL COMMENT 'column id', +`column_name` VARCHAR(128) NOT NULL COMMENT 'column name', +`column_type` VARCHAR(128) NOT NULL COMMENT 'column type', +`column_comment` VARCHAR(256) DEFAULT '' COMMENT 'column comment', +`column_nullable` TINYINT(1) NOT NULL DEFAULT 1 COMMENT 'column nullable, 0 is not nullable, 1 is nullable', +`column_auto_increment` TINYINT(1) NOT NULL DEFAULT 0 COMMENT 'column auto increment, 0 is not auto increment, 1 is auto increment', +`column_default_value` VARCHAR(256) DEFAULT NULL COMMENT 'column default value', +`column_op_type` TINYINT(1) NOT NULL COMMENT 'column operation type, 1 is create, 2 is update, 3 is delete', Review Comment: Yeah, it can be. but I think this might be more efficient and I clearly define it in Enum, so from my side the current implementation is acceptable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4940] feat(lakehouse-paimon): Support OSS filesystem for Paimon catalog. [gravitino]
yuqi1129 commented on code in PR #4941:
URL: https://github.com/apache/gravitino/pull/4941#discussion_r1796527212
##
catalogs/catalog-lakehouse-paimon/src/test/java/org/apache/gravitino/catalog/lakehouse/paimon/integration/test/CatalogPaimonOSSIT.java:
##
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.catalog.lakehouse.paimon.integration.test;
+
+import com.google.common.collect.Maps;
+import java.util.Map;
+import
org.apache.gravitino.catalog.lakehouse.paimon.PaimonCatalogPropertiesMetadata;
+import org.apache.gravitino.integration.test.util.DownloaderUtils;
+import org.apache.gravitino.integration.test.util.ITUtils;
+import org.apache.gravitino.storage.OSSProperties;
+import org.apache.spark.sql.SparkSession;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+
+@Tag("gravitino-docker-test")
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+@Disabled(
+"You need to specify the real OSS bucket name, access key, secret key and
endpoint to run this test")
+public class CatalogPaimonOSSIT extends CatalogPaimonBaseIT {
+
+ private static final String OSS_BUCKET_NAME = "YOUR_BUCKET";
+ private static final String accessKey = "YOUR_ACCESS_KEY";
+ private static final String secretKey = "YOUR_SECRET_KEY";
+ private static final String endpoint = "OSS_ENDPOINT";
+ private static final String warehouse = "oss://" + OSS_BUCKET_NAME +
"/paimon-test";
+
+ private static final String PAIMON_OSS_JAR_URL =
+
"https://repo1.maven.org/maven2/org/apache/paimon/paimon-oss/0.8.0/paimon-oss-0.8.0.jar";;
+
+ @Override
+ protected Map initPaimonCatalogProperties() {
+
+Map catalogProperties = Maps.newHashMap();
+catalogProperties.put("key1", "val1");
+catalogProperties.put("key2", "val2");
+
+TYPE = "filesystem";
+
+WAREHOUSE = warehouse;
+
catalogProperties.put(PaimonCatalogPropertiesMetadata.GRAVITINO_CATALOG_BACKEND,
TYPE);
+catalogProperties.put(PaimonCatalogPropertiesMetadata.WAREHOUSE,
WAREHOUSE);
+catalogProperties.put(OSSProperties.GRAVITINO_OSS_ACCESS_KEY_ID,
accessKey);
+catalogProperties.put(OSSProperties.GRAVITINO_OSS_ACCESS_KEY_SECRET,
secretKey);
+catalogProperties.put(OSSProperties.GRAVITINO_OSS_ENDPOINT, endpoint);
+
+// Need to download the OSS dependency in the deploy mode.
+downloadOSSDependency();
+
+return catalogProperties;
+ }
+
+ private void downloadOSSDependency() {
+String gravitinoHome = System.getenv("GRAVITINO_HOME");
+try {
+ if (!ITUtils.EMBEDDED_TEST_MODE.equals(testMode)) {
+String serverPath = ITUtils.joinPath(gravitinoHome, "libs");
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#4657] Improvement (docs): Add documentation for the Gravitino-Trino connector running on AWS S3. [gravitino]
yuqi1129 commented on PR #4658: URL: https://github.com/apache/gravitino/pull/4658#issuecomment-2406740079 Generally LGTM, @FANNG1 , do you have any further comments? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5101] improvement(docs): Fix the document error about access control [gravitino]
jerryshao merged PR #5102: URL: https://github.com/apache/gravitino/pull/5102 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Improvement] Fix the document error about access control [gravitino]
jerryshao closed issue #5101: [Improvement] Fix the document error about access control URL: https://github.com/apache/gravitino/issues/5101 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Improvement] Fix the document error about access control [gravitino]
jerryshao closed issue #5101: [Improvement] Fix the document error about access control URL: https://github.com/apache/gravitino/issues/5101 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(gravitino) branch main updated: [#5101] improvement(docs): Fix the document error about access control (#5102)
This is an automated email from the ASF dual-hosted git repository.
jshao pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new df2ce81e8 [#5101] improvement(docs): Fix the document error about
access control (#5102)
df2ce81e8 is described below
commit df2ce81e857851e234021e49d49c94bad702519c
Author: roryqi
AuthorDate: Fri Oct 11 15:42:34 2024 +0800
[#5101] improvement(docs): Fix the document error about access control
(#5102)
### What changes were proposed in this pull request?
Fix the document error about access control
### Why are the changes needed?
Fix: #5101
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Just docs.
---
docs/docker-image-details.md| 4 ++--
docs/security/access-control.md | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/docs/docker-image-details.md b/docs/docker-image-details.md
index c7942625b..cad304657 100644
--- a/docs/docker-image-details.md
+++ b/docs/docker-image-details.md
@@ -14,7 +14,7 @@ You can deploy the service with the Gravitino Docker image.
Container startup commands
```shell
-docker run --rm -d -p 8090:8090 -p 9001:9001 apache/gravitino:0.6.0-incubating
+docker run --rm -d -p 8090:8090 -p 9001:9001 apache/gravitino:0.6.1-incubating
```
Changelog
@@ -51,7 +51,7 @@ You can deploy the standalone Gravitino Iceberg REST server
with the Docker imag
Container startup commands
```shell
-docker run --rm -d -p 9001:9001 apache/gravitino-iceberg-rest:0.6.0-incubating
+docker run --rm -d -p 9001:9001 apache/gravitino-iceberg-rest:0.6.1-incubating
```
Changelog
diff --git a/docs/security/access-control.md b/docs/security/access-control.md
index 9d98c0a37..fb8bde8cb 100644
--- a/docs/security/access-control.md
+++ b/docs/security/access-control.md
@@ -662,7 +662,7 @@ MetadataObject table = ...
Role role = client.grantPrivilegesToRole("role1", table,
Lists.newArrayList(Privileges.SelectTable.allow()));
```
-
+
### Revoke privileges from a role
[PR] [#5101] improvement(docs): Fix the document error about access control [gravitino]
github-actions[bot] opened a new pull request, #5104: URL: https://github.com/apache/gravitino/pull/5104 ### What changes were proposed in this pull request? Fix the document error about access control ### Why are the changes needed? Fix: #5101 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Just docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(gravitino) branch cherry-pick-branch-0.6-df2ce81e857851e234021e49d49c94bad702519c created (now 072582199)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch cherry-pick-branch-0.6-df2ce81e857851e234021e49d49c94bad702519c in repository https://gitbox.apache.org/repos/asf/gravitino.git at 072582199 [#5101] improvement(docs): Fix the document error about access control (#5102) No new revisions were added by this update.
Re: [PR] [#5101] improvement(docs): Fix the document error about access control [gravitino]
jerryshao closed pull request #5104: [#5101] improvement(docs): Fix the document error about access control URL: https://github.com/apache/gravitino/pull/5104 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4951] improvement(test): Reduce fields shares between different IT to make AbstractIT more independent. [gravitino]
mchades commented on code in PR #4996:
URL: https://github.com/apache/gravitino/pull/4996#discussion_r1796557198
##
trino-connector/integration-test/src/test/java/org/apache/gravitino/trino/connector/integration/test/TrinoQueryITBase.java:
##
@@ -68,14 +68,17 @@ public class TrinoQueryITBase {
protected static final String metalakeName = "test";
protected static GravitinoMetalake metalake;
- private static void setEnv() throws Exception {
+ private static AbstractIT abstractIT;
+
+ private void setEnv() throws Exception {
+abstractIT = new AbstractIT();
Review Comment:
Is it possible for us to standardize the usage of `AbstractIT` class? Just
like its name, we set it as an abstract class that requires inheritance to use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5019] feat: (hadoop-catalog): Add a framework to support multi-storage in a pluginized manner for fileset catalog [gravitino]
jerryshao commented on PR #5020: URL: https://github.com/apache/gravitino/pull/5020#issuecomment-2406752097 @xloya can you please also help to review this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5019] feat: (hadoop-catalog): Add a framework to support multi-storage in a pluginized manner for fileset catalog [gravitino]
jerryshao commented on code in PR #5020:
URL: https://github.com/apache/gravitino/pull/5020#discussion_r1796568717
##
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/FileSystemProvider.java:
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.catalog.hadoop;
+
+import java.io.IOException;
+import java.util.Map;
+import org.apache.hadoop.fs.FileSystem;
+
+public interface FileSystemProvider {
+
+ /**
+ * Get the FileSystem instance according to the configuration and the path.
Review Comment:
Please update the javadoc here, `path` is not valid anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5101] improvement(docs): Fix the document error about access control [gravitino]
yuqi1129 commented on code in PR #5102: URL: https://github.com/apache/gravitino/pull/5102#discussion_r1796531401 ## docs/docker-image-details.md: ## @@ -14,7 +14,7 @@ You can deploy the service with the Gravitino Docker image. Container startup commands ```shell -docker run --rm -d -p 8090:8090 -p 9001:9001 apache/gravitino:0.6.0-incubating +docker run --rm -d -p 8090:8090 -p 9001:9001 apache/gravitino:0.6.1-incubating Review Comment: Another place also need to be changed https://github.com/user-attachments/assets/2ba21296-44cd-44ba-9546-0bf531c5789b";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5101] improvement(docs): Fix the document error about access control [gravitino]
jerqi commented on code in PR #5102: URL: https://github.com/apache/gravitino/pull/5102#discussion_r1796542631 ## docs/docker-image-details.md: ## @@ -14,7 +14,7 @@ You can deploy the service with the Gravitino Docker image. Container startup commands ```shell -docker run --rm -d -p 8090:8090 -p 9001:9001 apache/gravitino:0.6.0-incubating +docker run --rm -d -p 8090:8090 -p 9001:9001 apache/gravitino:0.6.1-incubating Review Comment: OK. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [MINOR] fix(docs): update javaDoc link from 0.6.0-incubating to 0.7.0-incubating [gravitino]
mchades opened a new pull request, #5103: URL: https://github.com/apache/gravitino/pull/5103 ### What changes were proposed in this pull request? update javaDoc link from 0.6.0-incubating to 0.7.0-incubating ### Why are the changes needed? Fix version info ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? no need -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4867] feat(core): Add storage support for columns in Gravitino [gravitino]
jerryshao commented on code in PR #5078:
URL: https://github.com/apache/gravitino/pull/5078#discussion_r1796548589
##
core/src/main/java/org/apache/gravitino/storage/relational/service/TableMetaService.java:
##
@@ -144,30 +153,47 @@ public TableEntity
updateTable(
CommonMetaService.getInstance().getParentEntityIdByNamespace(identifier.namespace());
TablePO oldTablePO = getTablePOBySchemaIdAndName(schemaId, tableName);
-TableEntity oldTableEntity = POConverters.fromTablePO(oldTablePO,
identifier.namespace());
-TableEntity newEntity = (TableEntity) updater.apply((E) oldTableEntity);
+List oldTableColumns =
+TableColumnMetaService.getInstance()
+.getColumnsByTableIdAndVersion(oldTablePO.getTableId(),
oldTablePO.getCurrentVersion());
+TableEntity oldTableEntity =
+POConverters.fromTableAndColumnPOs(oldTablePO, oldTableColumns,
identifier.namespace());
+
+TableEntity newTableEntity = (TableEntity) updater.apply((E)
oldTableEntity);
Preconditions.checkArgument(
-Objects.equals(oldTableEntity.id(), newEntity.id()),
+Objects.equals(oldTableEntity.id(), newTableEntity.id()),
"The updated table entity id: %s should be same with the table entity
id before: %s",
-newEntity.id(),
+newTableEntity.id(),
oldTableEntity.id());
-Integer updateResult;
+boolean isColumnChanged =
+TableColumnMetaService.getInstance().isColumnUpdated(oldTableEntity,
newTableEntity);
+TablePO newTablePO =
+POConverters.updateTablePOWithVersion(oldTablePO, newTableEntity,
isColumnChanged);
+
+final AtomicInteger updateResult = new AtomicInteger(0);
try {
- updateResult =
- SessionUtils.doWithCommitAndFetchResult(
- TableMetaMapper.class,
- mapper ->
- mapper.updateTableMeta(
- POConverters.updateTablePOWithVersion(oldTablePO,
newEntity), oldTablePO));
+ SessionUtils.doMultipleWithCommit(
+ () ->
+ updateResult.set(
Review Comment:
Fixed for table update and delete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] fix(docs): update javaDoc link from 0.6.0-incubating to 0.7.0-incubating [gravitino]
jerryshao merged PR #5103: URL: https://github.com/apache/gravitino/pull/5103 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(gravitino) branch main updated: [MINOR] fix(docs): update javaDoc link from 0.6.0-incubating to 0.7.0-incubating (#5103)
This is an automated email from the ASF dual-hosted git repository.
jshao pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new d956d5d23 [MINOR] fix(docs): update javaDoc link from 0.6.0-incubating
to 0.7.0-incubating (#5103)
d956d5d23 is described below
commit d956d5d238ee9748295e87d887999c2a417c3c91
Author: mchades
AuthorDate: Fri Oct 11 15:38:13 2024 +0800
[MINOR] fix(docs): update javaDoc link from 0.6.0-incubating to
0.7.0-incubating (#5103)
### What changes were proposed in this pull request?
update javaDoc link from 0.6.0-incubating to 0.7.0-incubating
### Why are the changes needed?
Fix version info
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
no need
---
docs/index.md | 4 ++--
docs/manage-relational-metadata-using-gravitino.md | 10 +-
2 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/docs/index.md b/docs/index.md
index e216a6de8..f2d7584e1 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -59,7 +59,7 @@ REST API and the Java SDK. You can use either to manage
metadata. See
Also, you can find the complete REST API definition in
[Gravitino Open API](./api/rest/gravitino-rest-api),
-Java SDK definition in [Gravitino Java
doc](pathname:///docs/0.6.0-incubating/api/java/index.html).
+Java SDK definition in [Gravitino Java
doc](pathname:///docs/0.7.0-incubating-SNAPSHOT/api/java/index.html).
Gravitino provides a web UI to manage the metadata. Visit the web UI in the
browser via `http://:8090`. See [Gravitino web UI](./webui.md) for
details.
@@ -163,7 +163,7 @@ Gravitino provides security configurations for Gravitino,
including HTTPS, authe
### Programming guides
* [Gravitino Open API](./api/rest/gravitino-rest-api): provides the complete
Open API definition of Gravitino.
-* [Gravitino Java doc](pathname:///docs/0.6.0-incubating/api/java/index.html):
provides the Javadoc for the Gravitino API.
+* [Gravitino Java
doc](pathname:///docs/0.7.0-incubating-SNAPSHOT/api/java/index.html): provides
the Javadoc for the Gravitino API.
### Development guides
diff --git a/docs/manage-relational-metadata-using-gravitino.md
b/docs/manage-relational-metadata-using-gravitino.md
index cdef8a680..6226891e2 100644
--- a/docs/manage-relational-metadata-using-gravitino.md
+++ b/docs/manage-relational-metadata-using-gravitino.md
@@ -737,7 +737,7 @@ The following types that Gravitino supports:
| Union | `Types.UnionType.of([type1, type2, ...])`
| `{"type": "union", "types": [type JSON, ...]}`
| Union type, indicates a union of types
|
| UUID | `Types.UUIDType.get()`
| `uuid`
| UUID type, indicates a universally unique identifier
|
-The related java doc is
[here](pathname:///docs/0.6.0-incubating/api/java/org/apache/gravitino/rel/types/Type.html).
+The related java doc is
[here](pathname:///docs/0.7.0-incubating-SNAPSHOT/api/java/org/apache/gravitino/rel/types/Type.html).
# External type
@@ -842,10 +842,10 @@ In addition to the basic settings, Gravitino supports the
following features:
| Feature | Description
| Java
doc
|
|-||-|
-| Table partitioning | Equal to `PARTITION BY` in Apache Hive, It is a
partitioning strategy that is used to split a table into parts based on
partition keys. Some table engine may not support this feature
Re: [PR] [#5019] feat: (hadoop-catalog): Add a framework to support multi-storage in a pluginized manner for fileset catalog [gravitino]
jerryshao commented on code in PR #5020:
URL: https://github.com/apache/gravitino/pull/5020#discussion_r1796584976
##
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/HadoopCatalogOperations.java:
##
@@ -119,32 +135,59 @@ public void initialize(
Map config, CatalogInfo info, HasPropertyMetadata
propertiesMetadata)
throws RuntimeException {
this.propertiesMetadata = propertiesMetadata;
+this.catalogInfo = info;
+
// Initialize Hadoop Configuration.
this.conf = config;
-this.hadoopConf = new Configuration();
-this.catalogInfo = info;
+
+hadoopConf = new Configuration();
Map bypassConfigs =
-config.entrySet().stream()
+conf.entrySet().stream()
.filter(e -> e.getKey().startsWith(CATALOG_BYPASS_PREFIX))
.collect(
Collectors.toMap(
e -> e.getKey().substring(CATALOG_BYPASS_PREFIX.length()),
Map.Entry::getValue));
bypassConfigs.forEach(hadoopConf::set);
+this.bypassConfigs = bypassConfigs;
+
+initPluginFileSystem(config);
+
String catalogLocation =
(String)
propertiesMetadata
.catalogPropertiesMetadata()
.getOrDefault(config,
HadoopCatalogPropertiesMetadata.LOCATION);
-conf.forEach(hadoopConf::set);
-
this.catalogStorageLocation =
StringUtils.isNotBlank(catalogLocation)
? Optional.of(catalogLocation).map(Path::new)
: Optional.empty();
}
+ private void initPluginFileSystem(Map config) {
+String fileSystemProviders =
+(String)
+propertiesMetadata
+.catalogPropertiesMetadata()
+.getOrDefault(config,
HadoopCatalogPropertiesMetadata.FILESYSTEM_PROVIDER);
Review Comment:
providers or provider?
##
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/HadoopCatalogOperations.java:
##
@@ -75,8 +77,12 @@ public class HadoopCatalogOperations implements
CatalogOperations, SupportsSchem
private static final String SCHEMA_DOES_NOT_EXIST_MSG = "Schema %s does not
exist";
private static final String FILESET_DOES_NOT_EXIST_MSG = "Fileset %s does
not exist";
private static final String SLASH = "/";
+ public static final String DEFAULT_FS = "fs.defaultFS";
+ private static final String LOCAL_FILE_SCHEMA = "file";
+ public static final String LOCAL_FILE_PATH = "file:///";
Review Comment:
Move the definition of public field on top of private ones.
##
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/HadoopCatalogOperations.java:
##
@@ -119,32 +135,59 @@ public void initialize(
Map config, CatalogInfo info, HasPropertyMetadata
propertiesMetadata)
throws RuntimeException {
this.propertiesMetadata = propertiesMetadata;
+this.catalogInfo = info;
+
// Initialize Hadoop Configuration.
this.conf = config;
-this.hadoopConf = new Configuration();
-this.catalogInfo = info;
+
+hadoopConf = new Configuration();
Map bypassConfigs =
-config.entrySet().stream()
+conf.entrySet().stream()
.filter(e -> e.getKey().startsWith(CATALOG_BYPASS_PREFIX))
.collect(
Collectors.toMap(
e -> e.getKey().substring(CATALOG_BYPASS_PREFIX.length()),
Map.Entry::getValue));
bypassConfigs.forEach(hadoopConf::set);
+this.bypassConfigs = bypassConfigs;
+
+initPluginFileSystem(config);
Review Comment:
`initializeFileSystemProviders`
##
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/HadoopCatalogOperations.java:
##
@@ -119,32 +135,59 @@ public void initialize(
Map config, CatalogInfo info, HasPropertyMetadata
propertiesMetadata)
throws RuntimeException {
this.propertiesMetadata = propertiesMetadata;
+this.catalogInfo = info;
+
// Initialize Hadoop Configuration.
this.conf = config;
-this.hadoopConf = new Configuration();
-this.catalogInfo = info;
+
+hadoopConf = new Configuration();
Review Comment:
Do we still need this `hadoopConf`?
##
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/HadoopCatalogOperations.java:
##
@@ -742,4 +785,44 @@ private boolean checkSingleFile(Fileset fileset) {
fileset.name());
}
}
+
+ static FileSystem getFileSystem(Path path, Map config)
throws IOException {
+Map newConfig = Maps.newHashMap(config);
+String scheme;
+Path fsPath;
+if (path != null) {
+ scheme = path.toUri().getScheme();
+ if (scheme == null) {
+scheme = LOCAL_FILE_SCHEMA;
Review Comment:
I think if the sheme is null, then it should be the one got from
`DEFAULT_FS`, not the local file system. Besides, your assumption is that user
have to set
Re: [PR] [#5019] feat: (hadoop-catalog): Add a framework to support multi-storage in a pluginized manner for fileset catalog [gravitino]
jerryshao commented on PR #5020: URL: https://github.com/apache/gravitino/pull/5020#issuecomment-2406873415 Can you please do several rounds of self polishing to make the code better? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5019] feat: (hadoop-catalog): Add a framework to support multi-storage in a pluginized manner for fileset catalog [gravitino]
jerryshao commented on code in PR #5020:
URL: https://github.com/apache/gravitino/pull/5020#discussion_r1796603274
##
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/HadoopCatalogOperations.java:
##
@@ -742,4 +785,44 @@ private boolean checkSingleFile(Fileset fileset) {
fileset.name());
}
}
+
+ static FileSystem getFileSystem(Path path, Map config)
throws IOException {
+Map newConfig = Maps.newHashMap(config);
+String scheme;
+Path fsPath;
+if (path != null) {
+ scheme = path.toUri().getScheme();
+ if (scheme == null) {
+scheme = LOCAL_FILE_SCHEMA;
Review Comment:
Please think a bit on how to use it from user side, do not make it hard for
user to use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#4951] improvement(test): Reduce fields shares between different IT to make AbstractIT more independent. [gravitino]
yuqi1129 commented on code in PR #4996:
URL: https://github.com/apache/gravitino/pull/4996#discussion_r1796601197
##
trino-connector/integration-test/src/test/java/org/apache/gravitino/trino/connector/integration/test/TrinoQueryITBase.java:
##
@@ -68,14 +68,17 @@ public class TrinoQueryITBase {
protected static final String metalakeName = "test";
protected static GravitinoMetalake metalake;
- private static void setEnv() throws Exception {
+ private static AbstractIT abstractIT;
+
+ private void setEnv() throws Exception {
+abstractIT = new AbstractIT();
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(gravitino) branch cherry-pick-branch-0.6-df2ce81e857851e234021e49d49c94bad702519c deleted (was 072582199)
This is an automated email from the ASF dual-hosted git repository. jshao pushed a change to branch cherry-pick-branch-0.6-df2ce81e857851e234021e49d49c94bad702519c in repository https://gitbox.apache.org/repos/asf/gravitino.git was 072582199 [#5101] improvement(docs): Fix the document error about access control (#5102) The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
Re: [PR] [#5072]Improvement(docs): Add docs for Trino cascading query [gravitino]
yuqi1129 commented on PR #5012: URL: https://github.com/apache/gravitino/pull/5012#issuecomment-2406895287 @mchades @FANNG1 Would you like to take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5055] support spark authentication with Gravitino [gravitino]
yuqi1129 commented on code in PR #5040:
URL: https://github.com/apache/gravitino/pull/5040#discussion_r1796624990
##
docs/spark-connector/spark-authentication-with-gravitino.md:
##
@@ -0,0 +1,47 @@
+---
+title: "Spark authentication with Gravitino server"
+slug: /spark-connector/spark-authentication
+keyword: spark connector authentication oauth2 kerberos
+license: "This software is licensed under the Apache License version 2."
+---
+
+## Overview
+
+Spark connector supports `simple` `oauth2` and `kerberos` authentication when
accessing Gravitino server.
+
+| Property | Type | Default Value | Description
| Required | Since Version |
+|--||---|-|--|---|
+| spark.sql.gravitino.authType | string | `simple` | The authentication
mechanisms when communicating with Gravitino server, supports `simple`,
`oauth2` and `kerberos`. | No | 0.7.0 |
+
+## Simple mode
+
+In the simple mode, you could specify the username by configuration.
+
+| Property| Type | Default Value | Description
| Required | Since Version |
+|-||---|---|--|---|
+| spark.sql.gravitino.simple.userName | string | None | The user name
to access the Gravitino | No | 0.7.0 |
+
+If you doesn't set `spark.sql.gravitino.simple.userName` explicitly. It will
use the value of environment variable `GRAVITINO_USER` as the user. If the
environment variable `GRAVITINO_USER` isn't set, the client uses the user
logging in the machine.
Review Comment:
So the default value of `spark.sql.gravitino.simple.userName` is not `None`?
##
spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/plugin/GravitinoDriverPlugin.java:
##
@@ -155,4 +162,45 @@ private void registerSqlExtensions(SparkConf conf) {
conf.set(StaticSQLConf.SPARK_SESSION_EXTENSIONS().key(),
extensionString);
}
}
+
+ private static GravitinoClient createGravitinoClient(
+ String uri, String metalake, SparkConf sparkConf) {
+ClientBuilder builder =
GravitinoClient.builder(uri).withMetalake(metalake);
+String authType =
+sparkConf.get(GravitinoSparkConfig.GRAVITINO_AUTH_TYPE,
AuthProperties.SIMPLE_AUTH_TYPE);
+if (AuthProperties.isSimple(authType)) {
+ String username =
sparkConf.get(GravitinoSparkConfig.GRAVITINO_SIMPLE_USER_NAME, null);
+ if (StringUtils.isNotBlank(username)) {
+builder.withSimpleAuth(username.trim());
+ } else {
+builder.withSimpleAuth();
+ }
+} else if (AuthProperties.isOAuth2(authType)) {
+ String oAuthUri =
sparkConf.get(GravitinoSparkConfig.GRAVITINO_OAUTH2_URI, null);
+ String credential =
sparkConf.get(GravitinoSparkConfig.GRAVITINO_OAUTH2_CREDENTIAL, null);
+ String path = sparkConf.get(GravitinoSparkConfig.GRAVITINO_OAUTH2_PATH,
null);
+ String scope =
sparkConf.get(GravitinoSparkConfig.GRAVITINO_OAUTH2_SCOPE, null);
+ DefaultOAuth2TokenProvider oAuth2TokenProvider =
+ DefaultOAuth2TokenProvider.builder()
+ .withUri(oAuthUri)
+ .withCredential(credential)
+ .withPath(path)
+ .withScope(scope)
+ .build();
+ builder.withOAuth(oAuth2TokenProvider);
+} else if (AuthProperties.isKerberos(authType)) {
+ String principal =
sparkConf.get(GravitinoSparkConfig.GRAVITINO_KERBEROS_PRINCIPAL, null);
Review Comment:
The default value `null` is unnecessary since null is NOT a valid value for
`principal`, could you check the validity before creating a Gravitino client?
##
spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/plugin/GravitinoDriverPlugin.java:
##
@@ -155,4 +162,45 @@ private void registerSqlExtensions(SparkConf conf) {
conf.set(StaticSQLConf.SPARK_SESSION_EXTENSIONS().key(),
extensionString);
}
}
+
+ private static GravitinoClient createGravitinoClient(
+ String uri, String metalake, SparkConf sparkConf) {
+ClientBuilder builder =
GravitinoClient.builder(uri).withMetalake(metalake);
+String authType =
+sparkConf.get(GravitinoSparkConfig.GRAVITINO_AUTH_TYPE,
AuthProperties.SIMPLE_AUTH_TYPE);
+if (AuthProperties.isSimple(authType)) {
+ String username =
sparkConf.get(GravitinoSparkConfig.GRAVITINO_SIMPLE_USER_NAME, null);
+ if (StringUtils.isNotBlank(username)) {
+builder.withSimpleAuth(username.trim());
+ } else {
+builder.withSimpleAuth();
+ }
+} else if (Auth
Re: [PR] [#5101] improvement(docs): Fix the document error about access control [gravitino]
jerryshao merged PR #5104: URL: https://github.com/apache/gravitino/pull/5104 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(gravitino) branch branch-0.6 updated: [#5101] improvement(docs): Fix the document error about access control (#5104)
This is an automated email from the ASF dual-hosted git repository.
jshao pushed a commit to branch branch-0.6
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/branch-0.6 by this push:
new f3dd6430a [#5101] improvement(docs): Fix the document error about
access control (#5104)
f3dd6430a is described below
commit f3dd6430a454f5afd551b614e711e9e18fb00a9f
Author: github-actions[bot]
<41898282+github-actions[bot]@users.noreply.github.com>
AuthorDate: Fri Oct 11 16:22:06 2024 +0800
[#5101] improvement(docs): Fix the document error about access control
(#5104)
### What changes were proposed in this pull request?
Fix the document error about access control
### Why are the changes needed?
Fix: #5101
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Just docs.
Co-authored-by: roryqi
---
docs/docker-image-details.md| 4 ++--
docs/security/access-control.md | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/docs/docker-image-details.md b/docs/docker-image-details.md
index 600ef9468..c0d960d03 100644
--- a/docs/docker-image-details.md
+++ b/docs/docker-image-details.md
@@ -14,7 +14,7 @@ You can deploy the service with the Gravitino Docker image.
Container startup commands
```shell
-docker run --rm -d -p 8090:8090 -p 9001:9001 apache/gravitino:0.6.0-incubating
+docker run --rm -d -p 8090:8090 -p 9001:9001 apache/gravitino:0.6.1-incubating
```
Changelog
@@ -51,7 +51,7 @@ You can deploy the standalone Gravitino Iceberg REST server
with the Docker imag
Container startup commands
```shell
-docker run --rm -d -p 9001:9001 apache/gravitino-iceberg-rest:0.6.0-incubating
+docker run --rm -d -p 9001:9001 apache/gravitino-iceberg-rest:0.6.1-incubating
```
Changelog
diff --git a/docs/security/access-control.md b/docs/security/access-control.md
index 9d98c0a37..fb8bde8cb 100644
--- a/docs/security/access-control.md
+++ b/docs/security/access-control.md
@@ -662,7 +662,7 @@ MetadataObject table = ...
Role role = client.grantPrivilegesToRole("role1", table,
Lists.newArrayList(Privileges.SelectTable.allow()));
```
-
+
### Revoke privileges from a role
Re: [PR] [#4867] feat(core): Add storage support for columns in Gravitino [gravitino]
jerryshao commented on code in PR #5078:
URL: https://github.com/apache/gravitino/pull/5078#discussion_r1796609977
##
core/src/main/java/org/apache/gravitino/storage/relational/service/MetalakeMetaService.java:
##
@@ -191,6 +192,10 @@ public boolean deleteMetalake(NameIdentifier ident,
boolean cascade) {
SessionUtils.doWithoutCommit(
TableMetaMapper.class,
mapper ->
mapper.softDeleteTableMetasByMetalakeId(metalakeId)),
+() ->
+SessionUtils.doWithoutCommit(
Review Comment:
I see, I will add them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5077] feat(core): Add the check of user before creating metadata object [gravitino]
xunliu commented on code in PR #5096:
URL: https://github.com/apache/gravitino/pull/5096#discussion_r1796661464
##
clients/client-java/src/test/java/org/apache/gravitino/client/integration/test/authorization/CheckCurrentUserIT.java:
##
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.client.integration.test.authorization;
+
+import static
org.apache.gravitino.server.GravitinoServer.WEBSERVER_CONF_PREFIX;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.Collections;
+import java.util.Map;
+import org.apache.gravitino.Catalog;
+import org.apache.gravitino.Configs;
+import org.apache.gravitino.NameIdentifier;
+import org.apache.gravitino.auth.AuthConstants;
+import org.apache.gravitino.authorization.Privileges;
+import org.apache.gravitino.authorization.SecurableObject;
+import org.apache.gravitino.authorization.SecurableObjects;
+import org.apache.gravitino.client.GravitinoAdminClient;
+import org.apache.gravitino.client.GravitinoMetalake;
+import org.apache.gravitino.exceptions.ForbiddenException;
+import org.apache.gravitino.file.Fileset;
+import org.apache.gravitino.integration.test.container.ContainerSuite;
+import org.apache.gravitino.integration.test.container.HiveContainer;
+import org.apache.gravitino.integration.test.container.KafkaContainer;
+import org.apache.gravitino.integration.test.util.AbstractIT;
+import org.apache.gravitino.rel.Column;
+import org.apache.gravitino.rel.types.Types;
+import org.apache.gravitino.server.web.JettyServerConfig;
+import org.apache.gravitino.utils.RandomNameUtils;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Tag("gravitino-docker-test")
+public class CheckCurrentUserIT extends AbstractIT {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CheckCurrentUserIT.class);
+ private static final ContainerSuite containerSuite =
ContainerSuite.getInstance();
+ private static String hmsUri;
+ private static String kafkaBootstrapServers;
+ private static GravitinoMetalake metalake;
+ private static GravitinoMetalake anotherMetalake;
+ private static String metalakeName =
RandomNameUtils.genRandomName("metalake");
+
+ @BeforeAll
+ public static void startIntegrationTest() throws Exception {
+Map configs = Maps.newHashMap();
+configs.put(Configs.ENABLE_AUTHORIZATION.getKey(), String.valueOf(true));
+configs.put(Configs.SERVICE_ADMINS.getKey(), AuthConstants.ANONYMOUS_USER);
+registerCustomConfigs(configs);
+AbstractIT.startIntegrationTest();
+
+containerSuite.startHiveContainer();
+hmsUri =
+String.format(
+"thrift://%s:%d",
+containerSuite.getHiveContainer().getContainerIpAddress(),
+HiveContainer.HIVE_METASTORE_PORT);
+
+containerSuite.startKafkaContainer();
+kafkaBootstrapServers =
+String.format(
+"%s:%d",
+containerSuite.getKafkaContainer().getContainerIpAddress(),
+KafkaContainer.DEFAULT_BROKER_PORT);
Review Comment:
Why do we need to use a Kafka container to test?
If we can use Hive container to test at the same time, we can save resources.
##
clients/client-java/src/test/java/org/apache/gravitino/client/integration/test/authorization/CheckCurrentUserIT.java:
##
Review Comment:
Currently, This UT only have throw ForbiddenException UT.
I think we need to add normal operation UT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5057] Added first part of CLI code [gravitino]
justinmclean commented on code in PR #5058:
URL: https://github.com/apache/gravitino/pull/5058#discussion_r1797060667
##
clients/cli/src/main/java/org/apache/gravitino/cli/FullName.java:
##
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.cli;
+
+import org.apache.commons.cli.CommandLine;
+
+/**
+ * Extracts different arts of a full name (dot seperated) from the command
line input. This includes
+ * metalake, catalog, schema, and table names.
+ */
+public class FullName {
+ CommandLine line;
+
+ /**
+ * Constructor for the {@code FullName} class.
+ *
+ * @param line The parsed command line arguments.
+ */
+ public FullName(CommandLine line) {
+this.line = line;
+ }
+
+ /**
+ * Retrieves the metalake name from the command line options, environment
variables, or the first
+ * part of the full name.
+ *
+ * @return The metalake name, or null if not found.
+ */
+ public String getMetalakeName() {
+String metalakeEnv = System.getenv("GRAVITINO_METALAKE");
+
+// Check if the metalake name is specified as a command line option
+if (line.hasOption(GravitinoOptions.METALAKE)) {
+ return line.getOptionValue(GravitinoOptions.METALAKE);
+ // Check if the metalake name is set as an environment variable
+} else if (metalakeEnv != null) {
+ return metalakeEnv;
+ // Extract the metalake name from the full name option
+} else if (line.hasOption(GravitinoOptions.NAME)) {
+ return line.getOptionValue(GravitinoOptions.NAME).split("\\.")[0];
+}
+
+return null;
+ }
+
+ /**
+ * Retrieves the catalog name from the command line or the second part of
the full name option.
+ *
+ * @return The catalog name, or null if not found.
+ */
+ public String getCatalogName() {
+return getNamePart(GravitinoOptions.CATALOG, 1);
+ }
+
+ /**
+ * Retrieves the schema name from the command line or the third part of the
full name option.
+ *
+ * @return The schema name, or null if not found.
+ */
+ public String getSchemaName() {
+return getNamePart(GravitinoOptions.SCHEMA, 2);
+ }
+
+ /**
+ * Retrieves the table name from the command line or the fourth part of the
full name option.
+ *
+ * @return The table name, or null if not found.
+ */
+ public String getTableName() {
+return getNamePart(GravitinoOptions.TABLE, 3);
+ }
+
+ /**
+ * Helper method to retrieve a specific part of the full name based on the
position of the part.
+ *
+ * @param entity The part of the name to obtain.
+ * @param position The position of the name part in the full name string.
+ * @return The extracted part of the name, or {@code null} if the name part
is missing or
+ * malformed.
+ */
+ public String getNamePart(String entity, int position) {
+/* Check if the name is specified as a command line option. */
+if (line.hasOption(entity)) {
+ return line.getOptionValue(entity);
+ /* Extract the name part from the full name if available. */
+} else if (line.hasOption(GravitinoOptions.NAME)) {
+ String[] names = line.getOptionValue(GravitinoOptions.NAME).split("\\.");
+
+ /* Adjust position if metalake is part of the full name. */
+ String metalakeEnv = System.getenv("GRAVITINO_METALAKE");
+ if (metalakeEnv != null) {
+position = position - 1;
+ }
+
+ if (names.length < position) {
+System.err.println(ErrorMessages.MALFORMED_NAME);
+return null;
+ }
+
+ return names[position];
+}
+
+System.err.println(ErrorMessages.MISSING_NAME);
+return null;
Review Comment:
The tests cover this scenario (see missingName test).
##
clients/cli/src/main/java/org/apache/gravitino/cli/FullName.java:
##
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the Li
Re: [PR] [#5057] Added first part of CLI code [gravitino]
justinmclean commented on code in PR #5058:
URL: https://github.com/apache/gravitino/pull/5058#discussion_r1797485740
##
clients/cli/src/main/java/org/apache/gravitino/cli/GravitinoCommandLine.java:
##
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.cli;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.HelpFormatter;
+import org.apache.commons.cli.Options;
+import org.apache.gravitino.cli.commands.AllMetalakeDetails;
+import org.apache.gravitino.cli.commands.CatalogDetails;
+import org.apache.gravitino.cli.commands.ListCatalogs;
+import org.apache.gravitino.cli.commands.ListColumns;
+import org.apache.gravitino.cli.commands.ListMetalakes;
+import org.apache.gravitino.cli.commands.ListSchema;
+import org.apache.gravitino.cli.commands.ListTables;
+import org.apache.gravitino.cli.commands.MetalakeDetails;
+import org.apache.gravitino.cli.commands.SchemaDetails;
+import org.apache.gravitino.cli.commands.TableDetails;
+import org.apache.gravitino.cli.commands.Version;
+
+/* Gravitino Command line */
+public class GravitinoCommandLine {
+
+ private CommandLine line;
+ private Options options;
+ private String entity;
+ private String command;
+
+ public static final String CMD = "gcli"; // recommended name
+ public static final String DEFAULT_URL = "http://localhost:8090";;
+
+ /**
+ * Gravitino Command line.
+ *
+ * @param line Parsed command line object.
+ * @param options Available options for the CLI.
+ * @param entity The entity to apply the command to e.g. metlake, catalog,
schema, table etc etc.
+ * @param command The type of command to run i.e. list, details, update,
delete, or create.
+ */
+ public GravitinoCommandLine(CommandLine line, Options options, String
entity, String command) {
+this.line = line;
+this.options = options;
+this.entity = entity;
+this.command = command;
+ }
+
+ /** Handles the parsed command line arguments and executes the corresponding
actions. */
+ public void handleCommandLine() {
+/* Display command usage. */
+if (line.hasOption(GravitinoOptions.HELP)) {
+ GravitinoCommandLine.displayHelp(options);
+}
+/* Display Gravitino version. */
+else if (line.hasOption(GravitinoOptions.VERSION)) {
+ new Version(getUrl()).handle();
Review Comment:
I've changed the --version so it reports the client number and added
--server to display the server version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(gravitino) branch main updated: [#5094] Fix(catalog-lakehouse-paimon): make some table properties from reserved to immutable (#5095)
This is an automated email from the ASF dual-hosted git repository.
fanng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 8de31cdb3 [#5094] Fix(catalog-lakehouse-paimon): make some table
properties from reserved to immutable (#5095)
8de31cdb3 is described below
commit 8de31cdb3cc74a82144c0899a6636a70d6588b66
Author: cai can <94670132+caica...@users.noreply.github.com>
AuthorDate: Fri Oct 11 18:52:50 2024 -0700
[#5094] Fix(catalog-lakehouse-paimon): make some table properties from
reserved to immutable (#5095)
### What changes were proposed in this pull request?
make some table properties from reserved to immutable.
```
merge-engine
sequence.field
rowkind.field
```
### Why are the changes needed?
these table properties are immutable in Paimon.
Fix: https://github.com/apache/gravitino/issues/5094
### Does this PR introduce _any_ user-facing change?
updated related doc.
### How was this patch tested?
modified UT.
Co-authored-by: caican
---
.../paimon/PaimonTablePropertiesMetadata.java | 10 ++-
.../lakehouse/paimon/TestGravitinoPaimonTable.java | 73 +-
docs/lakehouse-paimon-catalog.md | 9 ++-
3 files changed, 59 insertions(+), 33 deletions(-)
diff --git
a/catalogs/catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonTablePropertiesMetadata.java
b/catalogs/catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonTablePropertiesMetadata.java
index 1c57e5b2c..671dd9d66 100644
---
a/catalogs/catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonTablePropertiesMetadata.java
+++
b/catalogs/catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonTablePropertiesMetadata.java
@@ -18,6 +18,7 @@
*/
package org.apache.gravitino.catalog.lakehouse.paimon;
+import static
org.apache.gravitino.connector.PropertyEntry.stringImmutablePropertyEntry;
import static
org.apache.gravitino.connector.PropertyEntry.stringReservedPropertyEntry;
import com.google.common.collect.ImmutableList;
@@ -51,9 +52,12 @@ public class PaimonTablePropertiesMetadata extends
BasePropertiesMetadata {
stringReservedPropertyEntry(COMMENT, "The table comment", true),
stringReservedPropertyEntry(OWNER, "The table owner", false),
stringReservedPropertyEntry(BUCKET_KEY, "The table bucket key",
false),
-stringReservedPropertyEntry(MERGE_ENGINE, "The table merge
engine", false),
-stringReservedPropertyEntry(SEQUENCE_FIELD, "The table sequence
field", false),
-stringReservedPropertyEntry(ROWKIND_FIELD, "The table rowkind
field", false),
+stringImmutablePropertyEntry(
+MERGE_ENGINE, "The table merge engine", false, null, false,
false),
+stringImmutablePropertyEntry(
+SEQUENCE_FIELD, "The table sequence field", false, null,
false, false),
+stringImmutablePropertyEntry(
+ROWKIND_FIELD, "The table rowkind field", false, null, false,
false),
stringReservedPropertyEntry(PRIMARY_KEY, "The table primary key",
false),
stringReservedPropertyEntry(PARTITION, "The table partition",
false));
PROPERTIES_METADATA = Maps.uniqueIndex(propertyEntries,
PropertyEntry::getName);
diff --git
a/catalogs/catalog-lakehouse-paimon/src/test/java/org/apache/gravitino/catalog/lakehouse/paimon/TestGravitinoPaimonTable.java
b/catalogs/catalog-lakehouse-paimon/src/test/java/org/apache/gravitino/catalog/lakehouse/paimon/TestGravitinoPaimonTable.java
index 05e36219d..466ecb345 100644
---
a/catalogs/catalog-lakehouse-paimon/src/test/java/org/apache/gravitino/catalog/lakehouse/paimon/TestGravitinoPaimonTable.java
+++
b/catalogs/catalog-lakehouse-paimon/src/test/java/org/apache/gravitino/catalog/lakehouse/paimon/TestGravitinoPaimonTable.java
@@ -382,16 +382,41 @@ public class TestGravitinoPaimonTable {
try (PaimonCatalogOperations ops = new PaimonCatalogOperations()) {
ops.initialize(
initBackendCatalogProperties(), entity.toCatalogInfo(),
PAIMON_PROPERTIES_METADATA);
- Map map = Maps.newHashMap();
- map.put(PaimonTablePropertiesMetadata.COMMENT, "test");
- map.put(PaimonTablePropertiesMetadata.OWNER, "test");
- map.put(PaimonTablePropertiesMetadata.BUCKET_KEY, "test");
- map.put(PaimonTablePropertiesMetadata.MERGE_ENGINE, "test");
- map.put(PaimonTablePropertiesMetadata.SEQUENCE_FIELD, "test");
- map.put(PaimonTablePropertiesMetadata.ROWKIND_FIELD, "test");
- map.put(PaimonTablePropertiesMetadata.PRIMARY_KEY, "test");
- map.put(PaimonTablePropertiesMetadata.PARTITION, "test");
- for
Re: [PR] [#5094] Fix(catalog-lakehouse-paimon): make some table properties from reserved to immutable [gravitino]
FANNG1 merged PR #5095: URL: https://github.com/apache/gravitino/pull/5095 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Improvement][catalog-lakehouse-paimon] make some table properties from reserved to immutable [gravitino]
FANNG1 closed issue #5094: [Improvement][catalog-lakehouse-paimon] make some table properties from reserved to immutable URL: https://github.com/apache/gravitino/issues/5094 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Improvement][catalog-lakehouse-paimon] make some table properties from reserved to immutable [gravitino]
FANNG1 closed issue #5094: [Improvement][catalog-lakehouse-paimon] make some table properties from reserved to immutable URL: https://github.com/apache/gravitino/issues/5094 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4953] feat(hudi-catalog): add Hudi catalog IT and enable module [gravitino]
jerryshao commented on code in PR #4965:
URL: https://github.com/apache/gravitino/pull/4965#discussion_r1797529352
##
catalogs/catalog-lakehouse-hudi/build.gradle.kts:
##
@@ -115,12 +110,29 @@ dependencies {
testImplementation(libs.hadoop2.auth) {
exclude("*")
}
+ testImplementation(libs.hadoop2.hdfs)
testImplementation(libs.hadoop2.mapreduce.client.core) {
exclude("*")
}
testImplementation(libs.htrace.core4)
testImplementation(libs.junit.jupiter.api)
- testImplementation(libs.woodstox.core)
+ testImplementation(libs.mysql.driver)
+ testImplementation(libs.postgresql.driver)
+ testImplementation(libs.prometheus.dropwizard)
Review Comment:
Why do we need these jars for test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#4951] improvement(test): Reduce fields shares between different IT to make AbstractIT more independent. [gravitino]
mchades commented on code in PR #4996:
URL: https://github.com/apache/gravitino/pull/4996#discussion_r1797529498
##
integration-test-common/src/test/java/org/apache/gravitino/integration/test/util/AbstractIT.java:
##
@@ -58,61 +58,61 @@
import org.apache.gravitino.server.web.JettyServerConfig;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.TestInstance;
import org.junit.jupiter.api.extension.ExtendWith;
-import org.junit.jupiter.params.ParameterizedTest;
-import org.junit.jupiter.params.provider.CsvSource;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.shaded.org.awaitility.Awaitility;
@ExtendWith({PrintFuncNameExtension.class, CloseContainerExtension.class})
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class AbstractIT {
Review Comment:
suggest add `abstract` modifier
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#4953] feat(hudi-catalog): add Hudi catalog IT and enable module [gravitino]
jerryshao commented on code in PR #4965:
URL: https://github.com/apache/gravitino/pull/4965#discussion_r1797529512
##
catalogs/catalog-lakehouse-hudi/build.gradle.kts:
##
@@ -138,3 +150,60 @@ dependencies {
testRuntimeOnly("org.apache.hudi:hudi-spark$sparkVersion-bundle_$scalaVersion:0.15.0")
Review Comment:
Is it better not to hard code the hudi version file, can we use the version
number defined in version.toml?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#4867] feat(core): Add storage support for columns in Gravitino [gravitino]
xloya commented on code in PR #5078:
URL: https://github.com/apache/gravitino/pull/5078#discussion_r1796583051
##
core/src/main/java/org/apache/gravitino/storage/relational/service/MetalakeMetaService.java:
##
@@ -191,6 +192,10 @@ public boolean deleteMetalake(NameIdentifier ident,
boolean cascade) {
SessionUtils.doWithoutCommit(
TableMetaMapper.class,
mapper ->
mapper.softDeleteTableMetasByMetalakeId(metalakeId)),
+() ->
+SessionUtils.doWithoutCommit(
Review Comment:
I think it's also need add the recursive delete logic in CatalogMetaService,
SchemaMetaService, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#4867] feat(core): Add storage support for columns in Gravitino [gravitino]
xloya commented on code in PR #5078:
URL: https://github.com/apache/gravitino/pull/5078#discussion_r1796583051
##
core/src/main/java/org/apache/gravitino/storage/relational/service/MetalakeMetaService.java:
##
@@ -191,6 +192,10 @@ public boolean deleteMetalake(NameIdentifier ident,
boolean cascade) {
SessionUtils.doWithoutCommit(
TableMetaMapper.class,
mapper ->
mapper.softDeleteTableMetasByMetalakeId(metalakeId)),
+() ->
+SessionUtils.doWithoutCommit(
Review Comment:
I think it's also need add the recursive delete logic in
`CatalogMetaService`, `SchemaMetaService`, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#4867] feat(core): Add storage support for columns in Gravitino [gravitino]
xloya commented on code in PR #5078: URL: https://github.com/apache/gravitino/pull/5078#discussion_r1796577811 ## scripts/mysql/upgrade-0.6.0-to-0.7.0-mysql.sql: ## @@ -0,0 +1,41 @@ +-- +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file-- +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"). You may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. +-- +CREATE TABLE IF NOT EXISTS `table_column_version_info` ( +`id` BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT COMMENT 'auto increment id', Review Comment: I see. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5071] Improvement (test): Add integration tests for Trino cascading queries [gravitino]
yuqi1129 commented on code in PR #5073:
URL: https://github.com/apache/gravitino/pull/5073#discussion_r1796556952
##
.github/workflows/trino-integration-test.yml:
##
@@ -76,7 +76,7 @@ jobs:
- name: Package Gravitino
run: |
- ./gradlew compileDistribution -x test -PjdkVersion=${{
matrix.java-version }}
+ ./gradlew build compileDistribution -x test -PjdkVersion=${{
matrix.java-version }}
Review Comment:
Why do we need to add the build task? I believe the build task is included
in `compileDistribution`
##
trino-connector/integration-test/trino-test-tools/trino-cascading-env/init/trino-local/config/catalog/trino.properties:
##
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+connector.name=trino
+connection-url=jdbc:trino://trino-remote:8080/tpcds
Review Comment:
${trino-remote-address}:${trino-remote-port}
##
trino-connector/integration-test/trino-test-tools/trino-cascading-env/init/trino-local/init.sh:
##
@@ -0,0 +1,84 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#
+
+set -ex
+trino_conf_dir="$(dirname "${BASH_SOURCE-$0}")"
+trino_conf_dir="$(cd "${trino_conf_dir}">/dev/null; pwd)"
+
+cp "$trino_conf_dir/config/config.properties" /etc/trino/config.properties
+cp "$trino_conf_dir/config/jvm.config" /etc/trino/jvm.config
+cp "$trino_conf_dir/config/log4j2.properties" /etc/trino/log4j2.properties
+cp "$trino_conf_dir/config/catalog/gravitino.properties"
/etc/trino/catalog/gravitino.properties
+cp "$trino_conf_dir/config/catalog/trino.properties"
/etc/trino/catalog/trino.properties
+
+#start the gravitino server
+cp -r /opt/gravitino-server /tmp/gravitino-server
+rm -fr /tmp/gravitino-server/logs
+rm -fr /tmp/gravitino-server/data
+/tmp/gravitino-server/bin/gravitino.sh start
+
+#create test metalake
+counter=0
+while [ $counter -le 30 ]; do
+ sleep 1
Review Comment:
Better move this line to L53.
##
trino-connector/integration-test/trino-test-tools/trino-cascading-env/init/trino-local/init.sh:
##
@@ -0,0 +1,84 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#
+
+set -ex
+trino_conf_dir="$(dirname "${BASH_SOURCE-$0}")"
+trino_conf_dir="$(cd "${trino_conf_dir}">/dev/null; pwd)"
+
+cp "$trino_conf_dir/config/config.properties" /etc/trino/config.properties
+cp "$trino_conf_dir/config/jvm.config" /etc/trino/jvm.config
+cp "$trino_conf_dir/config/log4j2.properties" /etc/trino/log4j2.properties
+cp "$trino_conf_dir/config/catalog/gravitino.properties"
/etc/trino/catalog/gravitino.properties
+cp "$trino_conf_dir/config/catalog/trino.properties"
/etc/trino/catalog/trino.properties
+
+#start the gravitino server
+cp -r /opt/gravitino-ser
Re: [PR] [#5057] Added first part of CLI code [gravitino]
justinmclean commented on code in PR #5058:
URL: https://github.com/apache/gravitino/pull/5058#discussion_r1797484945
##
clients/cli/README.md:
##
@@ -0,0 +1,140 @@
+
+
+# Apache Gravitino CLI
+
+Apache Gravitino CLI is a command-line tool that interacts with the Gravitino
server to manage and query entities like metalakes, catalogs, schemas, and
tables. The tool provides options for listing information about Gravitino
entities and in future versions support creating, deleting, and updating these
entities.
+
+## Table of Contents
+
+- [Features](#features)
+- [Installation](#installation)
+- [Usage](#usage)
+- [Commands](#commands)
+- [Running Tests](#running-tests)
+- [Contributing](#contributing)
+- [License](#license)
+
+## Features
+
+- Retrieve server version
+- Provide help on usage
+- Manage Gravitino entities such as Metalakes, Catalogs, Schemas, and Tables
+- List details about Graviotino entities
+
+## Installation
+
+### Prerequisites
+
+Before you can build and run this project, it is suggested you have the
following installed:
+
+- Java 11 or higher
+
+### Build the Project
+
+1. Clone the entire Gravitino repository:
+
+```bash
+git clone https://github.com/apache/gravitino
+```
+
+2. Build the CLI sub-project using Gradle:
+
+```bash
+./gradlew :clients:cli:build
+```
+3. Create an alias:
+
+```bash
+alias gcli='java -jar
clients/cli/build/libs/gravitino-cli-0.7.0-incubating-SNAPSHOT.jar'
+```
+3. Test the command:
+```bash
+gcli --help
+```
+
+## Usage
+
+To run the Gravitino CLI, use the following command structure:
+
+```bash
+usage: gcli [options]
Review Comment:
Done
##
clients/cli/build.gradle.kts:
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+plugins {
+ `maven-publish`
+ id("java")
+ id("idea")
+}
+
+dependencies {
+ implementation(project(":clients:client-java"))
+ implementation(project(":api"))
+ implementation(project(":common"))
+ implementation("commons-cli:commons-cli:1.9.0")
+ implementation("org.slf4j:slf4j-api:2.0.16")
+ implementation("org.slf4j:slf4j-simple:2.0.16")
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5057] Added first part of CLI code [gravitino]
justinmclean commented on code in PR #5058:
URL: https://github.com/apache/gravitino/pull/5058#discussion_r1797485394
##
clients/cli/src/main/java/org/apache/gravitino/cli/FullName.java:
##
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.cli;
+
+import org.apache.commons.cli.CommandLine;
+
+/**
+ * Extracts different arts of a full name (dot seperated) from the command
line input. This includes
+ * metalake, catalog, schema, and table names.
+ */
+public class FullName {
+ CommandLine line;
+
+ /**
+ * Constructor for the {@code FullName} class.
+ *
+ * @param line The parsed command line arguments.
+ */
+ public FullName(CommandLine line) {
+this.line = line;
+ }
+
+ /**
+ * Retrieves the metalake name from the command line options, environment
variables, or the first
+ * part of the full name.
+ *
+ * @return The metalake name, or null if not found.
+ */
+ public String getMetalakeName() {
+String metalakeEnv = System.getenv("GRAVITINO_METALAKE");
+
+// Check if the metalake name is specified as a command line option
+if (line.hasOption(GravitinoOptions.METALAKE)) {
+ return line.getOptionValue(GravitinoOptions.METALAKE);
+ // Check if the metalake name is set as an environment variable
+} else if (metalakeEnv != null) {
+ return metalakeEnv;
+ // Extract the metalake name from the full name option
+} else if (line.hasOption(GravitinoOptions.NAME)) {
+ return line.getOptionValue(GravitinoOptions.NAME).split("\\.")[0];
+}
+
+return null;
Review Comment:
The code can handle the case when metalake is null so I'm not sure what you
are asking to be changed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5057] Added first part of CLI code [gravitino]
justinmclean commented on code in PR #5058:
URL: https://github.com/apache/gravitino/pull/5058#discussion_r1797493441
##
clients/cli/src/main/java/org/apache/gravitino/cli/FullName.java:
##
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.cli;
+
+import org.apache.commons.cli.CommandLine;
+
+/**
+ * Extracts different arts of a full name (dot seperated) from the command
line input. This includes
+ * metalake, catalog, schema, and table names.
+ */
+public class FullName {
+ CommandLine line;
+
+ /**
+ * Constructor for the {@code FullName} class.
+ *
+ * @param line The parsed command line arguments.
+ */
+ public FullName(CommandLine line) {
+this.line = line;
+ }
+
+ /**
+ * Retrieves the metalake name from the command line options, environment
variables, or the first
+ * part of the full name.
+ *
+ * @return The metalake name, or null if not found.
+ */
+ public String getMetalakeName() {
+String metalakeEnv = System.getenv("GRAVITINO_METALAKE");
+
+// Check if the metalake name is specified as a command line option
+if (line.hasOption(GravitinoOptions.METALAKE)) {
+ return line.getOptionValue(GravitinoOptions.METALAKE);
+ // Check if the metalake name is set as an environment variable
+} else if (metalakeEnv != null) {
+ return metalakeEnv;
+ // Extract the metalake name from the full name option
+} else if (line.hasOption(GravitinoOptions.NAME)) {
+ return line.getOptionValue(GravitinoOptions.NAME).split("\\.")[0];
+}
+
+return null;
+ }
+
+ /**
+ * Retrieves the catalog name from the command line or the second part of
the full name option.
+ *
+ * @return The catalog name, or null if not found.
+ */
+ public String getCatalogName() {
+return getNamePart(GravitinoOptions.CATALOG, 1);
+ }
+
+ /**
+ * Retrieves the schema name from the command line or the third part of the
full name option.
+ *
+ * @return The schema name, or null if not found.
+ */
+ public String getSchemaName() {
+return getNamePart(GravitinoOptions.SCHEMA, 2);
+ }
+
+ /**
+ * Retrieves the table name from the command line or the fourth part of the
full name option.
+ *
+ * @return The table name, or null if not found.
+ */
+ public String getTableName() {
+return getNamePart(GravitinoOptions.TABLE, 3);
+ }
+
+ /**
+ * Helper method to retrieve a specific part of the full name based on the
position of the part.
+ *
+ * @param entity The part of the name to obtain.
+ * @param position The position of the name part in the full name string.
+ * @return The extracted part of the name, or {@code null} if the name part
is missing or
+ * malformed.
+ */
+ public String getNamePart(String entity, int position) {
+/* Check if the name is specified as a command line option. */
+if (line.hasOption(entity)) {
+ return line.getOptionValue(entity);
+ /* Extract the name part from the full name if available. */
+} else if (line.hasOption(GravitinoOptions.NAME)) {
+ String[] names = line.getOptionValue(GravitinoOptions.NAME).split("\\.");
+
Review Comment:
BTW the UI doesn't support names with dots in them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5057] Added first part of CLI code [gravitino]
yuqi1129 commented on code in PR #5058:
URL: https://github.com/apache/gravitino/pull/5058#discussion_r1797524367
##
clients/cli/build.gradle.kts:
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+plugins {
+ `maven-publish`
+ id("java")
+ id("idea")
+}
+
+dependencies {
+ implementation(project(":clients:client-java"))
+ implementation(project(":api"))
+ implementation(project(":common"))
+ implementation("commons-cli:commons-cli:1.9.0")
+ implementation("org.slf4j:slf4j-api:2.0.16")
+ implementation("org.slf4j:slf4j-simple:2.0.16")
+
+ testImplementation(libs.junit.jupiter.api)
+ testImplementation(libs.junit.jupiter.params)
+ testImplementation(libs.mockito.core)
+
+ testRuntimeOnly(libs.junit.jupiter.engine)
+}
+
+tasks.build {
+ dependsOn("javadoc")
Review Comment:
`client-java` is for Java programmers, and `client-cli` is just a command
shell; I believe they are slightly different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5057] Added first part of CLI code [gravitino]
yuqi1129 commented on code in PR #5058:
URL: https://github.com/apache/gravitino/pull/5058#discussion_r1797524621
##
clients/cli/src/main/java/org/apache/gravitino/cli/FullName.java:
##
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.cli;
+
+import org.apache.commons.cli.CommandLine;
+
+/**
+ * Extracts different arts of a full name (dot seperated) from the command
line input. This includes
+ * metalake, catalog, schema, and table names.
+ */
+public class FullName {
+ CommandLine line;
+
+ /**
+ * Constructor for the {@code FullName} class.
+ *
+ * @param line The parsed command line arguments.
+ */
+ public FullName(CommandLine line) {
+this.line = line;
+ }
+
+ /**
+ * Retrieves the metalake name from the command line options, environment
variables, or the first
+ * part of the full name.
+ *
+ * @return The metalake name, or null if not found.
+ */
+ public String getMetalakeName() {
+String metalakeEnv = System.getenv("GRAVITINO_METALAKE");
+
+// Check if the metalake name is specified as a command line option
+if (line.hasOption(GravitinoOptions.METALAKE)) {
+ return line.getOptionValue(GravitinoOptions.METALAKE);
+ // Check if the metalake name is set as an environment variable
+} else if (metalakeEnv != null) {
+ return metalakeEnv;
+ // Extract the metalake name from the full name option
+} else if (line.hasOption(GravitinoOptions.NAME)) {
+ return line.getOptionValue(GravitinoOptions.NAME).split("\\.")[0];
+}
+
+return null;
Review Comment:
Is it necessary to check whether a null value is a valid one and terminate
the cli as soon as possible if there is an invalid input?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5019] feat: (hadoop-catalog): Add a framework to support multi-storage in a pluginized manner for fileset catalog [gravitino]
xloya commented on code in PR #5020:
URL: https://github.com/apache/gravitino/pull/5020#discussion_r1797525379
##
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/FileSystemProvider.java:
##
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.catalog.hadoop;
+
+import java.io.IOException;
+import java.util.Map;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+
+public interface FileSystemProvider {
Review Comment:
I see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#4951] improvement(test): Reduce fields shares between different IT to make AbstractIT more independent. [gravitino]
yuqi1129 commented on code in PR #4996:
URL: https://github.com/apache/gravitino/pull/4996#discussion_r1797525238
##
integration-test-common/build.gradle.kts:
##
@@ -43,6 +43,8 @@ dependencies {
testImplementation(libs.httpclient5)
testImplementation(libs.testcontainers)
testImplementation(libs.testcontainers.mysql)
+ testImplementation(libs.mysql.driver)
+ testImplementation(libs.postgresql.driver)
Review Comment:
I removed these two dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5094] Fix(catalog-lakehouse-paimon): make some table properties from reserved to immutable [gravitino]
caican00 commented on code in PR #5095: URL: https://github.com/apache/gravitino/pull/5095#discussion_r1797525769 ## docs/lakehouse-paimon-catalog.md: ## @@ -168,11 +168,16 @@ The Gravitino server doesn't allow passing the following reserved fields. | `comment` | The table comment. | | `owner`| The table owner. | | `bucket-key` | The table bucket-key. | +| `primary-key` | The table primary-key. | +| `partition`| The table partition. | Review Comment: yes, https://github.com/apache/paimon/blob/master/paimon-common/src/main/java/org/apache/paimon/CoreOptions.java#L845-L851 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5094] Fix(catalog-lakehouse-paimon): make some table properties from reserved to immutable [gravitino]
caican00 commented on PR #5095: URL: https://github.com/apache/gravitino/pull/5095#issuecomment-2408295403 > Overal LGTM @FANNG1 please review again, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4953] feat(hudi-catalog): add Hudi catalog IT and enable module [gravitino]
jerryshao commented on code in PR #4965:
URL: https://github.com/apache/gravitino/pull/4965#discussion_r1797529512
##
catalogs/catalog-lakehouse-hudi/build.gradle.kts:
##
@@ -138,3 +150,60 @@ dependencies {
testRuntimeOnly("org.apache.hudi:hudi-spark$sparkVersion-bundle_$scalaVersion:0.15.0")
Review Comment:
Is it better not to hard code the hudi version here, can we use the version
number defined in version.toml?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[I] [Improvement] We should have a full version list for documents [gravitino]
JunpingDu opened a new issue, #5108: URL: https://github.com/apache/gravitino/issues/5108 ### What would you like to be improved? Our project's documentation is version specific which is a general case for all open source projects. For users who want to quickly access the specific version of document, there should be a place. Take Apache Spark for example: https://spark.apache.org/documentation.html, you can find historic versions of documents there. In our case (gravitino), the document link (https://gravitino.apache.org/docs/) will directly go to latest version. I think we should have a full list there, given 0.6.1 is quickly out. ### How should we improve? As I mentioned above, we should have a full version list of documents that can route to specific version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(gravitino) 01/01: remove unnecessary changes
This is an automated email from the ASF dual-hosted git repository. mchades pushed a commit to branch pull/5111/head in repository https://gitbox.apache.org/repos/asf/gravitino.git commit 03bf4e60027e1493b1e0a43e67437b59f213bf55 Author: mchades AuthorDate: Fri Oct 11 19:20:45 2024 +0800 remove unnecessary changes --- docs/iceberg-rest-service.md | 15 -- docs/lakehouse-iceberg-catalog.md | 60 --- docs/lakehouse-paimon-catalog.md | 4 --- 3 files changed, 79 deletions(-) diff --git a/docs/iceberg-rest-service.md b/docs/iceberg-rest-service.md index 588113c321..e3647e48c5 100644 --- a/docs/iceberg-rest-service.md +++ b/docs/iceberg-rest-service.md @@ -83,21 +83,6 @@ You can also specify filter parameters by setting configuration entries in the s Gravitino Iceberg REST server supports OAuth2 and HTTPS, please refer to [Security](security/security.md) for more details. - Backend authentication - -For JDBC backend, you can use the `gravitino.iceberg-rest.jdbc.user` and `gravitino.iceberg-rest.jdbc.password` to authenticate the JDBC connection. For Hive backend, you can use the `gravitino.iceberg-rest.authentication.type` to specify the authentication type, and use the `gravitino.iceberg-rest.authentication.kerberos.principal` and `gravitino.iceberg-rest.authentication.kerberos.keytab-uri` to authenticate the Kerberos connection. -The detailed configuration items are as follows: - -| Configuration item| Description | Default value | Required | Since Version| -|---|---|---||--| -| `gravitino.iceberg-rest.authentication.type` | The type of authentication for Iceberg rest catalog backend. This configuration only applicable for for Hive backend, and only supports `Kerberos`, `simple` currently. As for JDBC backend, only username/password authentication was supported now. | `simple` | No | 0.6.0-incubating | -| `gravitino.iceberg-rest.authentication.impersonation-enable` | Whether to enable impersonation for the Iceberg catalog | `false` | No | 0.6.0-incubating | -| `gravitino.iceberg-rest.authentication.kerberos.principal`| The principal of the Kerberos authentication | (none)| required if the value of `gravitino.iceberg-rest.authentication.type` is Kerberos. | 0.6.0-incubating | -| `gravitino.iceberg-rest.authentication.kerberos.keytab-uri` | The URI of The keytab for the Kerberos authentication. | (none)| required if the value of `gravitino.iceberg-rest.authentication.type` is Kerberos. | 0.6.0-incubating | -| `gravitino.iceberg-rest.authentication.kerberos.check-interval-sec` | The check interval of Kerberos credential for Iceberg catalog. | 60| No | 0.6.0-incubating | -| `gravitino.iceberg-rest.authentication.kerberos.keytab-fetch-timeout-sec` | The fetch timeout of retrieving Kerberos keytab from `authentication.kerberos.keytab-uri`. | 60| No
(gravitino) branch pull/5111/head created (now 03bf4e6002)
This is an automated email from the ASF dual-hosted git repository. mchades pushed a change to branch pull/5111/head in repository https://gitbox.apache.org/repos/asf/gravitino.git at 03bf4e6002 remove unnecessary changes This branch includes the following new commits: new 03bf4e6002 remove unnecessary changes The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(gravitino) branch cherry-pick-branch-0.6-02c2ef0ec547351e4efdc8653eacf2d58d3c43cb updated (d778c0f59 -> 03bf4e600)
This is an automated email from the ASF dual-hosted git repository. mchades pushed a change to branch cherry-pick-branch-0.6-02c2ef0ec547351e4efdc8653eacf2d58d3c43cb in repository https://gitbox.apache.org/repos/asf/gravitino.git from d778c0f59 [MINOR] fix(docs): change version info from 0.6.0 to 0.6.0-incubating (#5107) add 03bf4e600 remove unnecessary changes No new revisions were added by this update. Summary of changes: docs/iceberg-rest-service.md | 15 -- docs/lakehouse-iceberg-catalog.md | 60 --- docs/lakehouse-paimon-catalog.md | 4 --- 3 files changed, 79 deletions(-)
Re: [PR] [#5019] feat: (hadoop-catalog): Add a framework to support multi-storage in a pluginized manner for fileset catalog [gravitino]
yuqi1129 commented on code in PR #5020:
URL: https://github.com/apache/gravitino/pull/5020#discussion_r1796902289
##
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/HadoopCatalogOperations.java:
##
@@ -742,4 +785,44 @@ private boolean checkSingleFile(Fileset fileset) {
fileset.name());
}
}
+
+ static FileSystem getFileSystem(Path path, Map config)
throws IOException {
+Map newConfig = Maps.newHashMap(config);
+String scheme;
+Path fsPath;
+if (path != null) {
+ scheme = path.toUri().getScheme();
+ if (scheme == null) {
+scheme = LOCAL_FILE_SCHEMA;
Review Comment:
> I think if the sheme is null, then it should be the one got from
DEFAULT_FS, not the local file system.
Yeah. this is my fault.
Since the interface is `FileSystem getFileSystem(Map
config)`, it requires that configuration `fs.defaultFS` is explicitly set if
we do not provide the path when getting a filesystem instance.
> Besides, your assumption is that user have to set fs.defaultFS is config,
right?
There are the following two situations, It depends on which kind of `user`
you refer
- For ordinary users, they do not need to specify the `fs.defaultFS` when
creating a Hadoop catalog, the value of `fs.defaultFS` will be extracted from
the storage path they set.
- For developers, due to the interface limitation, we must set
`fs.defaultFS` explicitly, or we can't get the proper file system instance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5057] Added first part of CLI code [gravitino]
diqiu50 commented on code in PR #5058:
URL: https://github.com/apache/gravitino/pull/5058#discussion_r1796841490
##
clients/cli/src/main/java/org/apache/gravitino/cli/FullName.java:
##
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.cli;
+
+import org.apache.commons.cli.CommandLine;
+
+/**
+ * Extracts different arts of a full name (dot seperated) from the command
line input. This includes
+ * metalake, catalog, schema, and table names.
+ */
+public class FullName {
+ CommandLine line;
+
+ /**
+ * Constructor for the {@code FullName} class.
+ *
+ * @param line The parsed command line arguments.
+ */
+ public FullName(CommandLine line) {
+this.line = line;
+ }
+
+ /**
+ * Retrieves the metalake name from the command line options, environment
variables, or the first
+ * part of the full name.
+ *
+ * @return The metalake name, or null if not found.
+ */
+ public String getMetalakeName() {
+String metalakeEnv = System.getenv("GRAVITINO_METALAKE");
+
+// Check if the metalake name is specified as a command line option
+if (line.hasOption(GravitinoOptions.METALAKE)) {
+ return line.getOptionValue(GravitinoOptions.METALAKE);
+ // Check if the metalake name is set as an environment variable
+} else if (metalakeEnv != null) {
+ return metalakeEnv;
+ // Extract the metalake name from the full name option
+} else if (line.hasOption(GravitinoOptions.NAME)) {
+ return line.getOptionValue(GravitinoOptions.NAME).split("\\.")[0];
+}
+
+return null;
+ }
+
+ /**
+ * Retrieves the catalog name from the command line or the second part of
the full name option.
+ *
+ * @return The catalog name, or null if not found.
+ */
+ public String getCatalogName() {
+return getNamePart(GravitinoOptions.CATALOG, 1);
+ }
+
+ /**
+ * Retrieves the schema name from the command line or the third part of the
full name option.
+ *
+ * @return The schema name, or null if not found.
+ */
+ public String getSchemaName() {
+return getNamePart(GravitinoOptions.SCHEMA, 2);
+ }
+
+ /**
+ * Retrieves the table name from the command line or the fourth part of the
full name option.
+ *
+ * @return The table name, or null if not found.
+ */
+ public String getTableName() {
+return getNamePart(GravitinoOptions.TABLE, 3);
+ }
+
+ /**
+ * Helper method to retrieve a specific part of the full name based on the
position of the part.
+ *
+ * @param entity The part of the name to obtain.
+ * @param position The position of the name part in the full name string.
+ * @return The extracted part of the name, or {@code null} if the name part
is missing or
+ * malformed.
+ */
+ public String getNamePart(String entity, int position) {
+/* Check if the name is specified as a command line option. */
+if (line.hasOption(entity)) {
+ return line.getOptionValue(entity);
+ /* Extract the name part from the full name if available. */
+} else if (line.hasOption(GravitinoOptions.NAME)) {
+ String[] names = line.getOptionValue(GravitinoOptions.NAME).split("\\.");
+
+ /* Adjust position if metalake is part of the full name. */
+ String metalakeEnv = System.getenv("GRAVITINO_METALAKE");
+ if (metalakeEnv != null) {
+position = position - 1;
+ }
+
+ if (names.length < position) {
Review Comment:
Is it `<=`?
##
clients/cli/src/main/java/org/apache/gravitino/cli/FullName.java:
##
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS,
Re: [PR] [#4940] feat(lakehouse-paimon): Support OSS filesystem for Paimon catalog. [gravitino]
FANNG1 merged PR #4941: URL: https://github.com/apache/gravitino/pull/4941 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [FEATURE] Support OSS storage for paimon catalog. [gravitino]
FANNG1 closed issue #4940: [FEATURE] Support OSS storage for paimon catalog. URL: https://github.com/apache/gravitino/issues/4940 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [FEATURE] Support OSS storage for paimon catalog. [gravitino]
FANNG1 closed issue #4940: [FEATURE] Support OSS storage for paimon catalog. URL: https://github.com/apache/gravitino/issues/4940 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(gravitino) branch main updated: [#4940] feat(lakehouse-paimon): Support OSS filesystem for Paimon catalog. (#4941)
This is an automated email from the ASF dual-hosted git repository.
fanng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 0d2435e25 [#4940] feat(lakehouse-paimon): Support OSS filesystem for
Paimon catalog. (#4941)
0d2435e25 is described below
commit 0d2435e2594384b0ba658548adff46af5026e762
Author: Qi Yu
AuthorDate: Fri Oct 11 20:46:22 2024 +0800
[#4940] feat(lakehouse-paimon): Support OSS filesystem for Paimon catalog.
(#4941)
### What changes were proposed in this pull request?
Support OSS filesystem for Paimon catalog.
### Why are the changes needed?
For better user experience.
Fix: #4940
### Does this PR introduce _any_ user-facing change?
N/A.
### How was this patch tested?
Only test local as there is no simulator for OSS.
---
catalogs/catalog-lakehouse-paimon/build.gradle.kts | 1 +
.../paimon/PaimonCatalogPropertiesMetadata.java| 36 +++
.../paimon/storage/PaimonOSSFileSystemConfig.java | 104 +
.../paimon/storage/PaimonS3FileSystemConfig.java | 104 +
.../lakehouse/paimon/utils/CatalogUtils.java | 27 --
.../integration/test/CatalogPaimonBaseIT.java | 2 +-
...alogPaimonS3IT.java => CatalogPaimonOSSIT.java} | 75 +--
.../paimon/integration/test/CatalogPaimonS3IT.java | 3 +-
docs/iceberg-rest-service.md | 12 +--
docs/lakehouse-paimon-catalog.md | 33 ---
gradle/libs.versions.toml | 1 +
11 files changed, 298 insertions(+), 100 deletions(-)
diff --git a/catalogs/catalog-lakehouse-paimon/build.gradle.kts
b/catalogs/catalog-lakehouse-paimon/build.gradle.kts
index c81959ee7..8fee91745 100644
--- a/catalogs/catalog-lakehouse-paimon/build.gradle.kts
+++ b/catalogs/catalog-lakehouse-paimon/build.gradle.kts
@@ -124,6 +124,7 @@ dependencies {
testImplementation(libs.postgresql.driver)
testImplementation(libs.bundles.log4j)
testImplementation(libs.junit.jupiter.params)
+ testImplementation(libs.paimon.oss)
testImplementation(libs.paimon.s3)
testImplementation(libs.paimon.spark)
testImplementation(libs.testcontainers)
diff --git
a/catalogs/catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonCatalogPropertiesMetadata.java
b/catalogs/catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonCatalogPropertiesMetadata.java
index 589d79672..0c6a87c02 100644
---
a/catalogs/catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonCatalogPropertiesMetadata.java
+++
b/catalogs/catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonCatalogPropertiesMetadata.java
@@ -31,10 +31,13 @@ import java.util.List;
import java.util.Map;
import
org.apache.gravitino.catalog.lakehouse.paimon.authentication.AuthenticationConfig;
import
org.apache.gravitino.catalog.lakehouse.paimon.authentication.kerberos.KerberosConfig;
-import org.apache.gravitino.catalog.lakehouse.paimon.storage.S3StorageConfig;
+import
org.apache.gravitino.catalog.lakehouse.paimon.storage.PaimonOSSFileSystemConfig;
+import
org.apache.gravitino.catalog.lakehouse.paimon.storage.PaimonS3FileSystemConfig;
+import org.apache.gravitino.catalog.lakehouse.paimon.utils.CatalogUtils;
import org.apache.gravitino.connector.BaseCatalogPropertiesMetadata;
import org.apache.gravitino.connector.PropertiesMetadata;
import org.apache.gravitino.connector.PropertyEntry;
+import org.apache.gravitino.storage.OSSProperties;
import org.apache.gravitino.storage.S3Properties;
/**
@@ -55,7 +58,7 @@ public class PaimonCatalogPropertiesMetadata extends
BaseCatalogPropertiesMetada
public static final Map GRAVITINO_CONFIG_TO_PAIMON =
ImmutableMap.of(GRAVITINO_CATALOG_BACKEND, PAIMON_METASTORE, WAREHOUSE,
WAREHOUSE, URI, URI);
private static final Map> PROPERTIES_METADATA;
- private static final Map KERBEROS_CONFIGURATION =
+ public static final Map KERBEROS_CONFIGURATION =
ImmutableMap.of(
KerberosConfig.PRINCIPAL_KEY,
KerberosConfig.PRINCIPAL_KEY,
@@ -74,6 +77,12 @@ public class PaimonCatalogPropertiesMetadata extends
BaseCatalogPropertiesMetada
S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY, S3_SECRET_KEY,
S3Properties.GRAVITINO_S3_ENDPOINT, S3_ENDPOINT);
+ public static final Map OSS_CONFIGURATION =
+ ImmutableMap.of(
+ OSSProperties.GRAVITINO_OSS_ACCESS_KEY_ID,
PaimonOSSFileSystemConfig.OSS_ACCESS_KEY,
+ OSSProperties.GRAVITINO_OSS_ACCESS_KEY_SECRET,
PaimonOSSFileSystemConfig.OSS_SECRET_KEY,
+ OSSProperties.GRAVITINO_OSS_ENDPOINT,
PaimonOSSFileSystemConfig.OSS_ENDPOINT);
+
static {
List> propertyEntries =
ImmutableList.of
Re: [PR] [#5094] Fix(catalog-lakehouse-paimon): make some table properties from reserved to immutable [gravitino]
FANNG1 commented on PR #5095: URL: https://github.com/apache/gravitino/pull/5095#issuecomment-2407348939 Overal LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5094] Fix(catalog-lakehouse-paimon): make some table properties from reserved to immutable [gravitino]
FANNG1 commented on code in PR #5095: URL: https://github.com/apache/gravitino/pull/5095#discussion_r1796903247 ## docs/lakehouse-paimon-catalog.md: ## @@ -168,11 +168,16 @@ The Gravitino server doesn't allow passing the following reserved fields. | `comment` | The table comment. | | `owner`| The table owner. | | `bucket-key` | The table bucket-key. | +| `primary-key` | The table primary-key. | +| `partition`| The table partition. | Review Comment: table partition key? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5057] Added first part of CLI code [gravitino]
justinmclean commented on code in PR #5058: URL: https://github.com/apache/gravitino/pull/5058#discussion_r1797047448 ## clients/cli/examples.sh: ## @@ -0,0 +1,111 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +#!/bin/bash + +# These examples assume you have the Apache Gravitino playground running. + +unset GRAVITINO_METALAKE +alias gcli='java -jar clients/cli/build/libs/gravitino-cli-0.7.0-incubating-SNAPSHOT.jar' + +# display help +gcli --help + +# display version +gcli --version + +# metalake details +gcli details + +# metalake list +gcli list + +# metalake details (all these command are equivalent) +gcli metalake details --name metalake_demo +gcli metalake details --metalake metalake_demo +gcli --command details --metalake metalake_demo +gcli metalake --name metalake_demo details +gcli details --name metalake_demo +gcli details --metalake metalake_demo + +# list all catalogs in a metalake +gcli metalake list --name metalake_demo + +# list catalog schema +gcli catalog list --name metalake_demo.catalog_iceberg +gcli catalog list --name metalake_demo.catalog_mysql +gcli catalog list --name metalake_demo.catalog_postgres +gcli catalog list --name metalake_demo.catalog_hive + +# list catalog details +gcli catalog details --name metalake_demo.catalog_iceberg +gcli catalog details --name metalake_demo.catalog_mysql +gcli catalog details --name metalake_demo.catalog_postgres +gcli catalog details --name metalake_demo.catalog_hive + +# list schema tables +gcli schema list --name metalake_demo.catalog_postgres.hr +gcli schema list --name metalake_demo.catalog_mysql.db +gcli schema list --name metalake_demo.catalog_hive.sales + +# list schema details +gcli schema details --name metalake_demo.catalog_postgres.hr +gcli schema details --name metalake_demo.catalog_mysql.db +gcli schema details --name metalake_demo.catalog_hive.sales + +# list table details +gcli table list --name metalake_demo.catalog_postgres.hr.departments +gcli table list --name metalake_demo.catalog_mysql.db.iceberg_tables +gcli table list --name metalake_demo.catalog_hive.sales.products + +# Exmaples where metalake is set in an evironment variable +export GRAVITINO_METALAKE=metalake_demo + +# metalake details +gcli metalake details + +# list all catalogs in a metalake +gcli metalake list Review Comment: All commmands, as was suggested in the review of the design, are in the form entity first then the action to perform. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5057] Added first part of CLI code [gravitino]
justinmclean commented on code in PR #5058:
URL: https://github.com/apache/gravitino/pull/5058#discussion_r1797060667
##
clients/cli/src/main/java/org/apache/gravitino/cli/FullName.java:
##
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.cli;
+
+import org.apache.commons.cli.CommandLine;
+
+/**
+ * Extracts different arts of a full name (dot seperated) from the command
line input. This includes
+ * metalake, catalog, schema, and table names.
+ */
+public class FullName {
+ CommandLine line;
+
+ /**
+ * Constructor for the {@code FullName} class.
+ *
+ * @param line The parsed command line arguments.
+ */
+ public FullName(CommandLine line) {
+this.line = line;
+ }
+
+ /**
+ * Retrieves the metalake name from the command line options, environment
variables, or the first
+ * part of the full name.
+ *
+ * @return The metalake name, or null if not found.
+ */
+ public String getMetalakeName() {
+String metalakeEnv = System.getenv("GRAVITINO_METALAKE");
+
+// Check if the metalake name is specified as a command line option
+if (line.hasOption(GravitinoOptions.METALAKE)) {
+ return line.getOptionValue(GravitinoOptions.METALAKE);
+ // Check if the metalake name is set as an environment variable
+} else if (metalakeEnv != null) {
+ return metalakeEnv;
+ // Extract the metalake name from the full name option
+} else if (line.hasOption(GravitinoOptions.NAME)) {
+ return line.getOptionValue(GravitinoOptions.NAME).split("\\.")[0];
+}
+
+return null;
+ }
+
+ /**
+ * Retrieves the catalog name from the command line or the second part of
the full name option.
+ *
+ * @return The catalog name, or null if not found.
+ */
+ public String getCatalogName() {
+return getNamePart(GravitinoOptions.CATALOG, 1);
+ }
+
+ /**
+ * Retrieves the schema name from the command line or the third part of the
full name option.
+ *
+ * @return The schema name, or null if not found.
+ */
+ public String getSchemaName() {
+return getNamePart(GravitinoOptions.SCHEMA, 2);
+ }
+
+ /**
+ * Retrieves the table name from the command line or the fourth part of the
full name option.
+ *
+ * @return The table name, or null if not found.
+ */
+ public String getTableName() {
+return getNamePart(GravitinoOptions.TABLE, 3);
+ }
+
+ /**
+ * Helper method to retrieve a specific part of the full name based on the
position of the part.
+ *
+ * @param entity The part of the name to obtain.
+ * @param position The position of the name part in the full name string.
+ * @return The extracted part of the name, or {@code null} if the name part
is missing or
+ * malformed.
+ */
+ public String getNamePart(String entity, int position) {
+/* Check if the name is specified as a command line option. */
+if (line.hasOption(entity)) {
+ return line.getOptionValue(entity);
+ /* Extract the name part from the full name if available. */
+} else if (line.hasOption(GravitinoOptions.NAME)) {
+ String[] names = line.getOptionValue(GravitinoOptions.NAME).split("\\.");
+
+ /* Adjust position if metalake is part of the full name. */
+ String metalakeEnv = System.getenv("GRAVITINO_METALAKE");
+ if (metalakeEnv != null) {
+position = position - 1;
+ }
+
+ if (names.length < position) {
+System.err.println(ErrorMessages.MALFORMED_NAME);
+return null;
+ }
+
+ return names[position];
+}
+
+System.err.println(ErrorMessages.MISSING_NAME);
+return null;
Review Comment:
The tests cover this scenario (see misisngName).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5057] Added first part of CLI code [gravitino]
justinmclean commented on code in PR #5058:
URL: https://github.com/apache/gravitino/pull/5058#discussion_r1797049049
##
clients/cli/build.gradle.kts:
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+plugins {
+ `maven-publish`
+ id("java")
+ id("idea")
+}
+
+dependencies {
+ implementation(project(":clients:client-java"))
+ implementation(project(":api"))
+ implementation(project(":common"))
+ implementation("commons-cli:commons-cli:1.9.0")
+ implementation("org.slf4j:slf4j-api:2.0.16")
+ implementation("org.slf4j:slf4j-simple:2.0.16")
+
+ testImplementation(libs.junit.jupiter.api)
+ testImplementation(libs.junit.jupiter.params)
+ testImplementation(libs.mockito.core)
+
+ testRuntimeOnly(libs.junit.jupiter.engine)
+}
+
+tasks.build {
+ dependsOn("javadoc")
Review Comment:
I was going on what client-java did. Is it not correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] fix(docs): change version info from 0.6.0 to 0.6.0-incubating [gravitino]
FANNG1 merged PR #5111: URL: https://github.com/apache/gravitino/pull/5111 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(gravitino) branch branch-0.6 updated: [MINOR] fix(docs): change version info from 0.6.0 to 0.6.0-incubating (#5111)
This is an automated email from the ASF dual-hosted git repository. fanng pushed a commit to branch branch-0.6 in repository https://gitbox.apache.org/repos/asf/gravitino.git The following commit(s) were added to refs/heads/branch-0.6 by this push: new 9298b8f8f [MINOR] fix(docs): change version info from 0.6.0 to 0.6.0-incubating (#5111) 9298b8f8f is described below commit 9298b8f8f6c9d8195c7b428391fdddb855ed0df1 Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> AuthorDate: Fri Oct 11 19:45:59 2024 +0800 [MINOR] fix(docs): change version info from 0.6.0 to 0.6.0-incubating (#5111) ### What changes were proposed in this pull request? change version info from 0.6.0 to 0.6.0-incubating ### Why are the changes needed? Version 0.6.0 is inaccurate. ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? no need - Co-authored-by: mchades --- MAINTAINERS.md | 4 +- docs/apache-hive-catalog.md| 2 +- docs/gravitino-server-config.md| 67 +++--- docs/hadoop-catalog.md | 14 ++--- docs/how-to-use-relational-backend-storage.md | 6 +- docs/iceberg-rest-service.md | 2 +- docs/jdbc-doris-catalog.md | 20 +++ docs/jdbc-mysql-catalog.md | 42 +++--- docs/jdbc-postgresql-catalog.md| 2 +- docs/lakehouse-iceberg-catalog.md | 18 +++--- docs/lakehouse-paimon-catalog.md | 25 docs/manage-relational-metadata-using-gravitino.md | 24 12 files changed, 112 insertions(+), 114 deletions(-) diff --git a/MAINTAINERS.md b/MAINTAINERS.md index f9b693345..5c8a466ce 100644 --- a/MAINTAINERS.md +++ b/MAINTAINERS.md @@ -61,8 +61,8 @@ something like that, all PRs should have related issues. 6. After PR is merged, please check the related issue: - If the issue is not closed, please close it as fixed manually. - Assign the issue "Assignees" to the PR author. - - Starting from 0.6.0, we will use the "labels" to manage the release versions, so please add - the corresponding labels to the issue. For example, if the issue is fixed in 0.6.0, please + - Starting from 0.6.0-incubating, we will use the "labels" to manage the release versions, so please add + the corresponding labels to the issue. For example, if the issue is fixed in 0.6.0-incubating, please add the label "0.6.0". If the issue is fixed both in 0.6.0 and 0.5.1, please add both labels. ## Policy on backporting bug fixes diff --git a/docs/apache-hive-catalog.md b/docs/apache-hive-catalog.md index d43f22f2f..8c494d0d5 100644 --- a/docs/apache-hive-catalog.md +++ b/docs/apache-hive-catalog.md @@ -125,7 +125,7 @@ The following table lists the data types mapped from the Hive catalog to Graviti | `uniontype` | `union` | 0.2.0 | :::info -Since 0.6.0, the data types other than listed above are mapped to Gravitino **[External Type](./manage-relational-metadata-using-gravitino.md#external-type)** that represents an unresolvable data type from the Hive catalog. +Since 0.6.0-incubating, the data types other than listed above are mapped to Gravitino **[External Type](./manage-relational-metadata-using-gravitino.md#external-type)** that represents an unresolvable data type from the Hive catalog. ::: ### Table properties diff --git a/docs/gravitino-server-config.md b/docs/gravitino-server-config.md index a09dcd5ed..e9bda9e05 100644 --- a/docs/gravitino-server-config.md +++ b/docs/gravitino-server-config.md @@ -23,20 +23,20 @@ The `gravitino.conf` file lists the configuration items in the following table. ### Apache Gravitino HTTP Server configuration -| Configuration item | Description | Default value | Required | Since version | -|--|---|--|--|---| -| `gravitino.server.webserver.host`| The host of the Gravitino server. | `0.0.0.0`
Re: [PR] [#5057] Added first part of CLI code [gravitino]
yuqi1129 commented on code in PR #5058:
URL: https://github.com/apache/gravitino/pull/5058#discussion_r1796715573
##
clients/cli/build.gradle.kts:
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+plugins {
+ `maven-publish`
+ id("java")
+ id("idea")
+}
+
+dependencies {
+ implementation(project(":clients:client-java"))
+ implementation(project(":api"))
+ implementation(project(":common"))
+ implementation("commons-cli:commons-cli:1.9.0")
+ implementation("org.slf4j:slf4j-api:2.0.16")
+ implementation("org.slf4j:slf4j-simple:2.0.16")
+
+ testImplementation(libs.junit.jupiter.api)
+ testImplementation(libs.junit.jupiter.params)
+ testImplementation(libs.mockito.core)
+
+ testRuntimeOnly(libs.junit.jupiter.engine)
+}
+
+tasks.build {
+ dependsOn("javadoc")
Review Comment:
Does the client cli module require the Java document?
##
clients/cli/src/main/java/org/apache/gravitino/cli/FullName.java:
##
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.cli;
+
+import org.apache.commons.cli.CommandLine;
+
+/**
+ * Extracts different arts of a full name (dot seperated) from the command
line input. This includes
+ * metalake, catalog, schema, and table names.
+ */
+public class FullName {
+ CommandLine line;
+
+ /**
+ * Constructor for the {@code FullName} class.
+ *
+ * @param line The parsed command line arguments.
+ */
+ public FullName(CommandLine line) {
+this.line = line;
+ }
+
+ /**
+ * Retrieves the metalake name from the command line options, environment
variables, or the first
+ * part of the full name.
+ *
+ * @return The metalake name, or null if not found.
+ */
+ public String getMetalakeName() {
+String metalakeEnv = System.getenv("GRAVITINO_METALAKE");
+
+// Check if the metalake name is specified as a command line option
+if (line.hasOption(GravitinoOptions.METALAKE)) {
+ return line.getOptionValue(GravitinoOptions.METALAKE);
+ // Check if the metalake name is set as an environment variable
+} else if (metalakeEnv != null) {
+ return metalakeEnv;
+ // Extract the metalake name from the full name option
+} else if (line.hasOption(GravitinoOptions.NAME)) {
+ return line.getOptionValue(GravitinoOptions.NAME).split("\\.")[0];
+}
+
+return null;
Review Comment:
I believe we need to handle the scenario when metalake is null.
##
clients/cli/src/main/java/org/apache/gravitino/cli/FullName.java:
##
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.cli;
+
Re: [PR] [#5019] feat: (hadoop-catalog): Add a framework to support multi-storage in a pluginized manner for fileset catalog [gravitino]
yuqi1129 commented on code in PR #5020:
URL: https://github.com/apache/gravitino/pull/5020#discussion_r1796854218
##
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/fs/HDFSFileSystemProvider.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog.hadoop.fs;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Map;
+import org.apache.gravitino.catalog.hadoop.FileSystemProvider;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+public class HDFSFileSystemProvider implements FileSystemProvider {
+
+ @Override
+ public FileSystem getFileSystem(Map config) throws
IOException {
+Configuration configuration = new Configuration();
+config.forEach(configuration::set);
+
+String pathString = configuration.get("fs.defaultFS");
+if (pathString == null) {
+ throw new IllegalArgumentException("The path should be specified.");
+}
+
+URI uri = new Path(pathString).toUri();
+if (uri.getScheme() != null && !uri.getScheme().equals("hdfs")) {
+ throw new IllegalArgumentException("The path should be a HDFS path.");
+}
+
+// Should we call DistributedFileSystem to create file system instance
explicitly? If we
+// explicitly create a HDFS file system here, we can't reuse the file
system cache in the
+// FileSystem class.
+String impl = configuration.get("fs.hdfs.impl");
+if (impl == null) {
+ configuration.set("fs.hdfs.impl",
"org.apache.hadoop.hdfs.DistributedFileSystem");
+} else {
+ if (!impl.equals("org.apache.hadoop.hdfs.DistributedFileSystem")) {
+throw new IllegalArgumentException(
+"The HDFS file system implementation class should be
'org.apache.hadoop.hdfs.DistributedFileSystem'.");
+ }
+}
+
+try {
+ if
(HDFSFileSystemProvider.class.getClassLoader().loadClass(configuration.get("fs.hdfs.impl"))
+ == null) {
+throw new IllegalArgumentException(
+"The HDFS file system implementation class is not found.");
+ }
+} catch (ClassNotFoundException e) {
+ throw new IllegalArgumentException("The HDFS file system implementation
class is not found.");
+}
+
+return FileSystem.newInstance(uri, configuration);
Review Comment:
Yeah, I have noticed this scenario, Since GVFS uses `newInstance`, the
Providers also need to use the same method that always creates a file system
instance, or some test will fail.
https://github.com/apache/gravitino/blob/02c2ef0ec547351e4efdc8653eacf2d58d3c43cb/clients/filesystem-hadoop3/src/test/java/org/apache/gravitino/filesystem/hadoop/integration/test/GravitinoVirtualFileSystemIT.java#L227-L243
L242 will fail as the `fs` has been closed by code in L232-L238.
So I have no choice but to use `newInstance` to replace `get` in the
providers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5019] feat: (hadoop-catalog): Add a framework to support multi-storage in a pluginized manner for fileset catalog [gravitino]
yuqi1129 commented on code in PR #5020:
URL: https://github.com/apache/gravitino/pull/5020#discussion_r1796859126
##
clients/filesystem-hadoop3/src/main/java/org/apache/gravitino/filesystem/hadoop/GravitinoVirtualFileSystem.java:
##
@@ -385,6 +423,23 @@ private FilesetContextPair getFilesetContext(Path
virtualPath, FilesetDataOperat
return new FilesetContextPair(new Path(actualFileLocation), fs);
}
+ private Map getConfigMap(Configuration configuration, URI
uri) {
+Map maps = Maps.newHashMap();
+configuration.forEach(
+entry -> {
+ String key = entry.getKey();
+ if (key.startsWith(GRAVITINO_BYPASS_PREFIX)) {
+maps.put(key.substring(GRAVITINO_BYPASS_PREFIX.length()),
entry.getValue());
+ } else if (!key.startsWith("fs.gvfs.")) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
svn commit: r72262 - /dev/incubator/gravitino/v0.6.1-incubating-rc4/
Author: mchades Date: Fri Oct 11 12:03:51 2024 New Revision: 72262 Log: Apache Gravitino v0.6.1-incubating-rc4 Added: dev/incubator/gravitino/v0.6.1-incubating-rc4/ dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-bin.tar.gz (with props) dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-bin.tar.gz.asc dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-bin.tar.gz.sha512 dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-src.tar.gz (with props) dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-src.tar.gz.asc dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-src.tar.gz.sha512 dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-iceberg-rest-server-0.6.1-incubating-bin.tar.gz (with props) dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-iceberg-rest-server-0.6.1-incubating-bin.tar.gz.asc dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-iceberg-rest-server-0.6.1-incubating-bin.tar.gz.sha512 dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-trino-connector-0.6.1-incubating.tar.gz (with props) dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-trino-connector-0.6.1-incubating.tar.gz.asc dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-trino-connector-0.6.1-incubating.tar.gz.sha512 Added: dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-bin.tar.gz == Binary file - no diff available. Propchange: dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-bin.tar.gz -- svn:mime-type = application/octet-stream Added: dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-bin.tar.gz.asc == --- dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-bin.tar.gz.asc (added) +++ dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-bin.tar.gz.asc Fri Oct 11 12:03:51 2024 @@ -0,0 +1,7 @@ +-BEGIN PGP SIGNATURE- + +iHUEABYIAB0WIQSZ8SFHEYOr0eN/HPOcnHS9+/12NgUCZwkTqgAKCRCcnHS9+/12 +Nqi1AP9r8toJ436sLwP1ETfUGnc1wFg/JEVzGNqUrqGIqugQ/gD/ViHYgeB4Ycf0 +sVXT9Mhns0raMAaQzZyxy4ZkG3mfMAs= +=K6iN +-END PGP SIGNATURE- Added: dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-bin.tar.gz.sha512 == --- dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-bin.tar.gz.sha512 (added) +++ dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-bin.tar.gz.sha512 Fri Oct 11 12:03:51 2024 @@ -0,0 +1 @@ +887b745428ebfd781ab0be051f2459101e450bc3f94a10d4cba0b621dbd54a5acb8cf9a14b108e56856a45f486f6313c984682bb1f296915aebefbfd4c8f8909 gravitino-0.6.1-incubating-bin.tar.gz Added: dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-src.tar.gz == Binary file - no diff available. Propchange: dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-src.tar.gz -- svn:mime-type = application/octet-stream Added: dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-src.tar.gz.asc == --- dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-src.tar.gz.asc (added) +++ dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-src.tar.gz.asc Fri Oct 11 12:03:51 2024 @@ -0,0 +1,7 @@ +-BEGIN PGP SIGNATURE- + +iHUEABYIAB0WIQSZ8SFHEYOr0eN/HPOcnHS9+/12NgUCZwkSmwAKCRCcnHS9+/12 +NtWUAP9iCvrPVCVy9/R0n5UhjgDfEtr/tn2pKfkp48Ay6lhXOgEAmTNMCaxu6DzA +mtCpyJTQarvfintXAJNc0zW7U0R0XQg= +=6x0h +-END PGP SIGNATURE- Added: dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-src.tar.gz.sha512 == --- dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-src.tar.gz.sha512 (added) +++ dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-0.6.1-incubating-src.tar.gz.sha512 Fri Oct 11 12:03:51 2024 @@ -0,0 +1 @@ +b999448025167b2fbb5a05fa471b5df8dc551a47d8dfb1ca2f9771a5bbb081b0764942f9d563daf05cfc32a76810c53377f7bfa9db14be058ccf1925177eb669 gravitino-0.6.1-incubating-src.tar.gz Added: dev/incubator/gravitino/v0.6.1-incubating-rc4/gravitino-iceberg-rest-server-0.6.1-incubating-bin.tar.gz == Binary file - no diff available.
Re: [PR] [#5019] feat: (hadoop-catalog): Add a framework to support multi-storage in a pluginized manner for fileset catalog [gravitino]
yuqi1129 commented on code in PR #5020:
URL: https://github.com/apache/gravitino/pull/5020#discussion_r1796864197
##
clients/filesystem-hadoop3/src/main/java/org/apache/gravitino/filesystem/hadoop/GravitinoVirtualFileSystem.java:
##
@@ -125,13 +136,33 @@ public void initialize(URI name, Configuration
configuration) throws IOException
initializeClient(configuration);
+initializePluginFileSystem(configuration);
+
this.workingDirectory = new Path(name);
this.uri = URI.create(name.getScheme() + "://" + name.getAuthority());
setConf(configuration);
super.initialize(uri, getConf());
}
+ private void initializePluginFileSystem(Configuration configuration) {
+String fileSystemProviders =
configuration.get("fs.gvfs.filesystem.providers");
Review Comment:
> I think you need define this config in the
GravitinoVirtualFileSystemConfiguration
OK.
> and how about use fs.gravitino.filesystem.providers
I will use another PR to add some documents about it after S3 and GCS are
finished, as GVFS doesn't depend on this configuration in the current code base
to support HDFS and localfile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [#5106] improve(auth-ranger): Filter Catalog securiable object in authorization Ranger [gravitino]
xunliu opened a new pull request, #5113: URL: https://github.com/apache/gravitino/pull/5113 ### What changes were proposed in this pull request? Filter Catalog securiable object in the authorization Ranger. ### Why are the changes needed? Fix: #5106 ### Does this PR introduce _any_ user-facing change? N/A ### How was this patch tested? Add ITs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5055] support spark authentication with Gravitino [gravitino]
yuqi1129 commented on code in PR #5040: URL: https://github.com/apache/gravitino/pull/5040#discussion_r1797001132 ## docs/spark-connector/spark-authentication-with-gravitino.md: ## @@ -0,0 +1,47 @@ +--- +title: "Spark authentication with Gravitino server" +slug: /spark-connector/spark-authentication +keyword: spark connector authentication oauth2 kerberos +license: "This software is licensed under the Apache License version 2." +--- + +## Overview + +Spark connector supports `simple` `oauth2` and `kerberos` authentication when accessing Gravitino server. + +| Property | Type | Default Value | Description | Required | Since Version | +|--||---|-|--|---| +| spark.sql.gravitino.authType | string | `simple` | The authentication mechanisms when communicating with Gravitino server, supports `simple`, `oauth2` and `kerberos`. | No | 0.7.0 | + +## Simple mode + +In the simple mode, you could specify the username by configuration. + +| Property| Type | Default Value | Description | Required | Since Version | +|-||---|---|--|---| +| spark.sql.gravitino.simple.userName | string | None | The user name to access the Gravitino | No | 0.7.0 | Review Comment: @jerqi Do you have any opinions on this matter? I think `simpleAuth` is okay but `oauth2` and `kerberos` also need to be changed accordingly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Improvement] We should have a full version list for documents [gravitino]
mchades commented on issue #5108: URL: https://github.com/apache/gravitino/issues/5108#issuecomment-2407484657 we already have the version list in the top left corner of the page, do we still need a separate page to list the historical versions? https://github.com/user-attachments/assets/e678431b-2eff-4c29-8d92-a5b9535b4579";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4867] feat(core): Add storage support for columns in Gravitino [gravitino]
jerryshao commented on code in PR #5078:
URL: https://github.com/apache/gravitino/pull/5078#discussion_r1797009827
##
core/src/main/java/org/apache/gravitino/storage/relational/service/CatalogMetaService.java:
##
@@ -215,6 +215,10 @@ public boolean deleteCatalog(NameIdentifier identifier,
boolean cascade) {
SessionUtils.doWithoutCommit(
FilesetMetaMapper.class,
mapper ->
mapper.softDeleteFilesetMetasByCatalogId(catalogId)),
+ () ->
+ SessionUtils.doWithoutCommit(
Review Comment:
Sorry my fault, I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5057] Added first part of CLI code [gravitino]
justinmclean commented on code in PR #5058:
URL: https://github.com/apache/gravitino/pull/5058#discussion_r1797021945
##
clients/cli/src/main/java/org/apache/gravitino/cli/FullName.java:
##
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.cli;
+
+import org.apache.commons.cli.CommandLine;
+
+/**
+ * Extracts different arts of a full name (dot seperated) from the command
line input. This includes
+ * metalake, catalog, schema, and table names.
+ */
+public class FullName {
+ CommandLine line;
+
+ /**
+ * Constructor for the {@code FullName} class.
+ *
+ * @param line The parsed command line arguments.
+ */
+ public FullName(CommandLine line) {
+this.line = line;
+ }
+
+ /**
+ * Retrieves the metalake name from the command line options, environment
variables, or the first
+ * part of the full name.
+ *
+ * @return The metalake name, or null if not found.
+ */
+ public String getMetalakeName() {
+String metalakeEnv = System.getenv("GRAVITINO_METALAKE");
+
+// Check if the metalake name is specified as a command line option
+if (line.hasOption(GravitinoOptions.METALAKE)) {
+ return line.getOptionValue(GravitinoOptions.METALAKE);
+ // Check if the metalake name is set as an environment variable
+} else if (metalakeEnv != null) {
+ return metalakeEnv;
+ // Extract the metalake name from the full name option
+} else if (line.hasOption(GravitinoOptions.NAME)) {
+ return line.getOptionValue(GravitinoOptions.NAME).split("\\.")[0];
+}
+
+return null;
+ }
+
+ /**
+ * Retrieves the catalog name from the command line or the second part of
the full name option.
+ *
+ * @return The catalog name, or null if not found.
+ */
+ public String getCatalogName() {
+return getNamePart(GravitinoOptions.CATALOG, 1);
+ }
+
+ /**
+ * Retrieves the schema name from the command line or the third part of the
full name option.
+ *
+ * @return The schema name, or null if not found.
+ */
+ public String getSchemaName() {
+return getNamePart(GravitinoOptions.SCHEMA, 2);
+ }
+
+ /**
+ * Retrieves the table name from the command line or the fourth part of the
full name option.
+ *
+ * @return The table name, or null if not found.
+ */
+ public String getTableName() {
+return getNamePart(GravitinoOptions.TABLE, 3);
+ }
+
+ /**
+ * Helper method to retrieve a specific part of the full name based on the
position of the part.
+ *
+ * @param entity The part of the name to obtain.
+ * @param position The position of the name part in the full name string.
+ * @return The extracted part of the name, or {@code null} if the name part
is missing or
+ * malformed.
+ */
+ public String getNamePart(String entity, int position) {
+/* Check if the name is specified as a command line option. */
+if (line.hasOption(entity)) {
+ return line.getOptionValue(entity);
+ /* Extract the name part from the full name if available. */
+} else if (line.hasOption(GravitinoOptions.NAME)) {
+ String[] names = line.getOptionValue(GravitinoOptions.NAME).split("\\.");
+
+ /* Adjust position if metalake is part of the full name. */
+ String metalakeEnv = System.getenv("GRAVITINO_METALAKE");
+ if (metalakeEnv != null) {
+position = position - 1;
+ }
+
+ if (names.length < position) {
Review Comment:
As the index is zero based this is correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [SIP] support pre event [gravitino]
FANNG1 opened a new pull request, #5110: URL: https://github.com/apache/gravitino/pull/5110 ### What changes were proposed in this pull request? (Please outline the changes and how this PR fixes the issue.) ### Why are the changes needed? (Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug.) Fix: # (issue) ### Does this PR introduce _any_ user-facing change? (Please list the user-facing changes introduced by your change, including 1. Change in user-facing APIs. 2. Addition or removal of property keys.) ### How was this patch tested? (Please test your changes, and provide instructions on how to test it: 1. If you add a feature or fix a bug, add a test to cover your changes. 2. If you fix a flaky test, repeat it for many times to prove it works.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(gravitino) tag v0.6.1-incubating-rc4 created (now d2b42e0f1)
This is an automated email from the ASF dual-hosted git repository. mchades pushed a change to tag v0.6.1-incubating-rc4 in repository https://gitbox.apache.org/repos/asf/gravitino.git at d2b42e0f1 (commit) This tag includes the following new commits: new d2b42e0f1 Preparing Gravitino release v0.6.1-incubating-rc4 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(gravitino) branch branch-0.6 updated (9298b8f8f -> 72bb01fa4)
This is an automated email from the ASF dual-hosted git repository. mchades pushed a change to branch branch-0.6 in repository https://gitbox.apache.org/repos/asf/gravitino.git from 9298b8f8f [MINOR] fix(docs): change version info from 0.6.0 to 0.6.0-incubating (#5111) add d2b42e0f1 Preparing Gravitino release v0.6.1-incubating-rc4 new 72bb01fa4 Preparing development version 0.6.2-incubating-SNAPSHOT The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes:
Re: [I] [Improvement] make some table properties from reserved to immutable [gravitino]
caican00 commented on issue #5094: URL: https://github.com/apache/gravitino/issues/5094#issuecomment-2407246905 > IMO, we could make `merge-engine` , `sequence.field`, `rowkind.field` to immutable, @caican00 WDYT? yes, it is resonable. i will update the related pr. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5019] feat: (hadoop-catalog): Add a framework to support multi-storage in a pluginized manner for fileset catalog [gravitino]
yuqi1129 commented on code in PR #5020:
URL: https://github.com/apache/gravitino/pull/5020#discussion_r1796848483
##
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/FileSystemProvider.java:
##
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.catalog.hadoop;
+
+import java.io.IOException;
+import java.util.Map;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+
+public interface FileSystemProvider {
Review Comment:
Currently, there seems to be no resources managed by providers. Providers
only provide file system instances that have already implemented the close
interface.
Furthermore, I don't think `FileSystemProvider` could manage the life cycle
of a file system. Please let me know if you have any further thoughts on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(gravitino) 01/01: Preparing Gravitino release v0.6.1-incubating-rc4
This is an automated email from the ASF dual-hosted git repository. mchades pushed a commit to tag v0.6.1-incubating-rc4 in repository https://gitbox.apache.org/repos/asf/gravitino.git commit d2b42e0f17c86dde2092aef372775ad115e5fafa Author: Minghuang Li AuthorDate: Fri Oct 11 11:51:13 2024 + Preparing Gravitino release v0.6.1-incubating-rc4 --- clients/client-python/setup.py | 2 +- gradle.properties | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/clients/client-python/setup.py b/clients/client-python/setup.py index 5b6d6a9b5..5b36e26f8 100644 --- a/clients/client-python/setup.py +++ b/clients/client-python/setup.py @@ -29,7 +29,7 @@ except FileNotFoundError: setup( name="apache-gravitino", description="Python lib/client for Apache Gravitino", -version="0.6.2.dev0", +version="0.6.1", long_description=long_description, long_description_content_type="text/markdown", author="Apache Software Foundation", diff --git a/gradle.properties b/gradle.properties index da3630411..32f955f9e 100644 --- a/gradle.properties +++ b/gradle.properties @@ -23,7 +23,7 @@ org.gradle.caching=true org.gradle.jvmargs=-Xmx4g # version that is going to be updated automatically by releases -version = 0.6.2-incubating-SNAPSHOT +version = 0.6.1-incubating # sonatype credentials SONATYPE_USER = admin
(gravitino) 01/01: Preparing development version 0.6.2-incubating-SNAPSHOT
This is an automated email from the ASF dual-hosted git repository. mchades pushed a commit to branch branch-0.6 in repository https://gitbox.apache.org/repos/asf/gravitino.git commit 72bb01fa4f24c8509e82880d3991835fa5d1e4a8 Author: Minghuang Li AuthorDate: Fri Oct 11 11:51:13 2024 + Preparing development version 0.6.2-incubating-SNAPSHOT --- clients/client-python/setup.py | 2 +- gradle.properties | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/clients/client-python/setup.py b/clients/client-python/setup.py index 5b36e26f8..5b6d6a9b5 100644 --- a/clients/client-python/setup.py +++ b/clients/client-python/setup.py @@ -29,7 +29,7 @@ except FileNotFoundError: setup( name="apache-gravitino", description="Python lib/client for Apache Gravitino", -version="0.6.1", +version="0.6.2.dev0", long_description=long_description, long_description_content_type="text/markdown", author="Apache Software Foundation", diff --git a/gradle.properties b/gradle.properties index 32f955f9e..da3630411 100644 --- a/gradle.properties +++ b/gradle.properties @@ -23,7 +23,7 @@ org.gradle.caching=true org.gradle.jvmargs=-Xmx4g # version that is going to be updated automatically by releases -version = 0.6.1-incubating +version = 0.6.2-incubating-SNAPSHOT # sonatype credentials SONATYPE_USER = admin
Re: [PR] [#5019] feat: (hadoop-catalog): Add a framework to support multi-storage in a pluginized manner for fileset catalog [gravitino]
yuqi1129 commented on code in PR #5020:
URL: https://github.com/apache/gravitino/pull/5020#discussion_r1796854218
##
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/fs/HDFSFileSystemProvider.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog.hadoop.fs;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Map;
+import org.apache.gravitino.catalog.hadoop.FileSystemProvider;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+public class HDFSFileSystemProvider implements FileSystemProvider {
+
+ @Override
+ public FileSystem getFileSystem(Map config) throws
IOException {
+Configuration configuration = new Configuration();
+config.forEach(configuration::set);
+
+String pathString = configuration.get("fs.defaultFS");
+if (pathString == null) {
+ throw new IllegalArgumentException("The path should be specified.");
+}
+
+URI uri = new Path(pathString).toUri();
+if (uri.getScheme() != null && !uri.getScheme().equals("hdfs")) {
+ throw new IllegalArgumentException("The path should be a HDFS path.");
+}
+
+// Should we call DistributedFileSystem to create file system instance
explicitly? If we
+// explicitly create a HDFS file system here, we can't reuse the file
system cache in the
+// FileSystem class.
+String impl = configuration.get("fs.hdfs.impl");
+if (impl == null) {
+ configuration.set("fs.hdfs.impl",
"org.apache.hadoop.hdfs.DistributedFileSystem");
+} else {
+ if (!impl.equals("org.apache.hadoop.hdfs.DistributedFileSystem")) {
+throw new IllegalArgumentException(
+"The HDFS file system implementation class should be
'org.apache.hadoop.hdfs.DistributedFileSystem'.");
+ }
+}
+
+try {
+ if
(HDFSFileSystemProvider.class.getClassLoader().loadClass(configuration.get("fs.hdfs.impl"))
+ == null) {
+throw new IllegalArgumentException(
+"The HDFS file system implementation class is not found.");
+ }
+} catch (ClassNotFoundException e) {
+ throw new IllegalArgumentException("The HDFS file system implementation
class is not found.");
+}
+
+return FileSystem.newInstance(uri, configuration);
Review Comment:
Yeah, I have noticed this scenario, since GVFS uses `newInstance`, the
Providers also need to use the same method that always creates a file system
instance, or some test will fail.
https://github.com/apache/gravitino/blob/02c2ef0ec547351e4efdc8653eacf2d58d3c43cb/clients/filesystem-hadoop3/src/test/java/org/apache/gravitino/filesystem/hadoop/integration/test/GravitinoVirtualFileSystemIT.java#L227-L243
L242 will fail as the `fs` has been closed by code in L232-L238.
So I have no choice but to use `newInstance` to replace `get` in the
providers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5057] Added first part of CLI code [gravitino]
justinmclean commented on code in PR #5058:
URL: https://github.com/apache/gravitino/pull/5058#discussion_r1797025808
##
clients/cli/src/main/java/org/apache/gravitino/cli/FullName.java:
##
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.cli;
+
+import org.apache.commons.cli.CommandLine;
+
+/**
+ * Extracts different arts of a full name (dot seperated) from the command
line input. This includes
+ * metalake, catalog, schema, and table names.
+ */
+public class FullName {
+ CommandLine line;
+
+ /**
+ * Constructor for the {@code FullName} class.
+ *
+ * @param line The parsed command line arguments.
+ */
+ public FullName(CommandLine line) {
+this.line = line;
+ }
+
+ /**
+ * Retrieves the metalake name from the command line options, environment
variables, or the first
+ * part of the full name.
+ *
+ * @return The metalake name, or null if not found.
+ */
+ public String getMetalakeName() {
+String metalakeEnv = System.getenv("GRAVITINO_METALAKE");
+
+// Check if the metalake name is specified as a command line option
+if (line.hasOption(GravitinoOptions.METALAKE)) {
+ return line.getOptionValue(GravitinoOptions.METALAKE);
+ // Check if the metalake name is set as an environment variable
+} else if (metalakeEnv != null) {
+ return metalakeEnv;
+ // Extract the metalake name from the full name option
+} else if (line.hasOption(GravitinoOptions.NAME)) {
+ return line.getOptionValue(GravitinoOptions.NAME).split("\\.")[0];
+}
+
+return null;
+ }
+
+ /**
+ * Retrieves the catalog name from the command line or the second part of
the full name option.
+ *
+ * @return The catalog name, or null if not found.
+ */
+ public String getCatalogName() {
+return getNamePart(GravitinoOptions.CATALOG, 1);
+ }
+
+ /**
+ * Retrieves the schema name from the command line or the third part of the
full name option.
+ *
+ * @return The schema name, or null if not found.
+ */
+ public String getSchemaName() {
+return getNamePart(GravitinoOptions.SCHEMA, 2);
+ }
+
+ /**
+ * Retrieves the table name from the command line or the fourth part of the
full name option.
+ *
+ * @return The table name, or null if not found.
+ */
+ public String getTableName() {
+return getNamePart(GravitinoOptions.TABLE, 3);
+ }
+
+ /**
+ * Helper method to retrieve a specific part of the full name based on the
position of the part.
+ *
+ * @param entity The part of the name to obtain.
+ * @param position The position of the name part in the full name string.
+ * @return The extracted part of the name, or {@code null} if the name part
is missing or
+ * malformed.
+ */
+ public String getNamePart(String entity, int position) {
+/* Check if the name is specified as a command line option. */
+if (line.hasOption(entity)) {
+ return line.getOptionValue(entity);
+ /* Extract the name part from the full name if available. */
+} else if (line.hasOption(GravitinoOptions.NAME)) {
+ String[] names = line.getOptionValue(GravitinoOptions.NAME).split("\\.");
+
Review Comment:
If the metalake is specified in one of the default ways, a catalog would
have no dot. Names themselves will not have a dot in them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[I] [Improvement] use incubating version in docs [gravitino]
mchades opened a new issue, #5114: URL: https://github.com/apache/gravitino/issues/5114 ### What would you like to be improved? Gravitino has become an Apache Incubator project since version 0.6.0, so it is necessary to follow the Apache versioning convention like `0.6.0-incubating` instead of `0.6.0`. ### How should we improve? visit all docs and update to incubating version after 0.6.0. Such as: https://github.com/apache/gravitino/blob/344e98df063548232f08a45a22fa06ebc26f6583/docs/how-to-use-gvfs.md?plain=1#L338-L348 should change the `Since version` column value to: ```md | Configuration item | Description | Default value | Required | Since version | |--|--|---|---|---| | `server_uri` | The Gravitino server uri, e.g. `http://localhost:8090`. | (none)| Yes | 0.6.0-incubating |. | `metalake_name` | The metalake name which the fileset belongs to. | (none)| Yes | 0.6.0-incubating |. | `cache_size` | The cache capacity of the Gravitino Virtual File System. | `20` | No| 0.6.0-incubating |. | `cache_expired_time` | The value of time that the cache expires after accessing in the Gravitino Virtual File System. The value is in `seconds`. | `3600`| No| 0.6.0-incubating |. | `auth_type` | The auth type to initialize the Gravitino client to use with the Gravitino Virtual File System. Currently supports `simple` and `oauth2` auth types. | `simple` | No| 0.6.0-incubating |. | `oauth2_server_uri` | The auth server URI for the Gravitino client when using `oauth2` auth type. | (none)| Yes if you use `oauth2` auth type | 0.7.0-incubating |. | `oauth2_credential` | The auth credential for the Gravitino client when using `oauth2` auth type. | (none)| Yes if you use `oauth2` auth type | 0.7.0-incubating |. | `oauth2_path`| The auth server path for the Gravitino client when using `oauth2` auth type. Please remove the first slash `/` from the path, for example `oauth/token`. | (none)| Yes if you use `oauth2` auth type | 0.7.0-incubating |. | `oauth2_scope` | The auth scope for the Gravitino client when using `oauth2` auth type with the Gravitino Virtual File System. | (none)| Yes if you use `oauth2` auth type | 0.7.0-incubating |. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
