Re: [PR] [#4508]feat(iceberg) support load custom catalog as backend [gravitino]
FANNG1 commented on code in PR #4563:
URL: https://github.com/apache/gravitino/pull/4563#discussion_r1721300638
##
iceberg/iceberg-common/src/test/java/org/apache/gravitino/iceberg/common/utils/TestIcebergCatalogUtil.java:
##
@@ -78,4 +78,52 @@ void testLoadCatalog() {
IcebergCatalogUtil.loadCatalogBackend("other");
});
}
+
+ @Test
+ void testValidLoadCustomCatalog() {
+Catalog catalog;
+Map config = new HashMap<>();
+
+config.put("catalog-backend-impl",
"org.apache.iceberg.inmemory.InMemoryCatalog");
+catalog =
+IcebergCatalogUtil.loadCatalogBackend(
+IcebergCatalogBackend.valueOf("CUSTOM"), new
IcebergConfig(config));
+Assertions.assertTrue(catalog instanceof InMemoryCatalog);
+
+config.clear();
+config.put("catalog-backend-impl", "org.apache.iceberg.hive.HiveCatalog");
+catalog =
+IcebergCatalogUtil.loadCatalogBackend(
+IcebergCatalogBackend.valueOf("CUSTOM"), new
IcebergConfig(config));
+Assertions.assertTrue(catalog instanceof HiveCatalog);
+
+config.clear();
+config.put(
+"catalog-backend-impl",
"org.apache.gravitino.iceberg.common.utils.CustomCatalogForTest");
Review Comment:
seems use `org.apache.gravitino.iceberg.common.utils.CustomCatalogForTest`
is enough, no need to test `memory` or `hive`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#4508]feat(iceberg) support load custom catalog as backend [gravitino]
FANNG1 commented on PR #4563: URL: https://github.com/apache/gravitino/pull/4563#issuecomment-2295814583 Do you think is it necessary to support a custom catalog for the Gravitino Iceberg catalog not only for the Iceberg REST service? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [#4506] fix(UI): Hide jdbc-password value in details page [gravitino]
featherchen opened a new pull request, #4577: URL: https://github.com/apache/gravitino/pull/4577 ### What changes were proposed in this pull request? Hide jdbc-password value in details page. ### Why are the changes needed? Fix: #4506 ### Does this PR introduce _any_ user-facing change? Jdbc-password value is not palintext now. ### How was this patch tested?   -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4506] fix(UI): Hide jdbc-password value in details page [gravitino]
featherchen commented on PR #4561: URL: https://github.com/apache/gravitino/pull/4561#issuecomment-2295824010 > @featherchen the auto cherry-pick action is failed for this PR (https://github.com/apache/gravitino/actions/runs/10446609068). Would you please cherry-pick the commit to "branch-0.6" and create a new PR against branch-0.6, thanks a lot. I have created a PR(#4577) to branch-0.6, please help me review whether I made it in a right way, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4508]feat(iceberg) support load custom catalog as backend [gravitino]
theoryxu commented on code in PR #4563:
URL: https://github.com/apache/gravitino/pull/4563#discussion_r1721309322
##
iceberg/iceberg-common/src/test/java/org/apache/gravitino/iceberg/common/utils/TestIcebergCatalogUtil.java:
##
@@ -78,4 +78,52 @@ void testLoadCatalog() {
IcebergCatalogUtil.loadCatalogBackend("other");
});
}
+
+ @Test
+ void testValidLoadCustomCatalog() {
+Catalog catalog;
+Map config = new HashMap<>();
+
+config.put("catalog-backend-impl",
"org.apache.iceberg.inmemory.InMemoryCatalog");
+catalog =
+IcebergCatalogUtil.loadCatalogBackend(
+IcebergCatalogBackend.valueOf("CUSTOM"), new
IcebergConfig(config));
+Assertions.assertTrue(catalog instanceof InMemoryCatalog);
+
+config.clear();
+config.put("catalog-backend-impl", "org.apache.iceberg.hive.HiveCatalog");
+catalog =
+IcebergCatalogUtil.loadCatalogBackend(
+IcebergCatalogBackend.valueOf("CUSTOM"), new
IcebergConfig(config));
+Assertions.assertTrue(catalog instanceof HiveCatalog);
+
+config.clear();
+config.put(
+"catalog-backend-impl",
"org.apache.gravitino.iceberg.common.utils.CustomCatalogForTest");
Review Comment:
ok, I'll remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
yuqi1129 commented on code in PR #4515:
URL: https://github.com/apache/gravitino/pull/4515#discussion_r1721304932
##
authorizations/authorization-ranger/src/main/java/org/apache/gravitino/authorization/ranger/RangerAuthorizationPlugin.java:
##
@@ -0,0 +1,1023 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.authorization.ranger;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.errorprone.annotations.FormatMethod;
+import com.google.errorprone.annotations.FormatString;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.stream.Collectors;
+import org.apache.gravitino.MetadataObject;
+import org.apache.gravitino.authorization.Group;
+import org.apache.gravitino.authorization.Owner;
+import org.apache.gravitino.authorization.Privilege;
+import org.apache.gravitino.authorization.Role;
+import org.apache.gravitino.authorization.RoleChange;
+import org.apache.gravitino.authorization.SecurableObject;
+import org.apache.gravitino.authorization.SecurableObjects;
+import org.apache.gravitino.authorization.User;
+import org.apache.gravitino.authorization.ranger.defines.VXGroup;
+import org.apache.gravitino.authorization.ranger.defines.VXGroupList;
+import org.apache.gravitino.authorization.ranger.defines.VXUser;
+import org.apache.gravitino.authorization.ranger.defines.VXUserList;
+import org.apache.gravitino.connector.AuthorizationPropertiesMeta;
+import org.apache.gravitino.connector.authorization.AuthorizationPlugin;
+import org.apache.gravitino.exceptions.AuthorizationPluginException;
+import org.apache.ranger.RangerServiceException;
+import org.apache.ranger.plugin.model.RangerPolicy;
+import org.apache.ranger.plugin.model.RangerRole;
+import org.apache.ranger.plugin.util.GrantRevokeRoleRequest;
+import org.apache.ranger.plugin.util.SearchFilter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Ranger authorization operations plugin abstract class. */
+public abstract class RangerAuthorizationPlugin implements AuthorizationPlugin
{
+ private static final Logger LOG =
LoggerFactory.getLogger(RangerAuthorizationPlugin.class);
+
+ protected String catalogProvider;
+ protected RangerClientExt rangerClient;
+ protected String rangerServiceName;
+ /** Mapping Gravitino privilege name to the underlying authorization system
privileges. */
+ protected Map> mapPrivileges = null;
+ // The owner privileges, the owner can do anything on the metadata object
+ protected Set ownerPrivileges = null;
+
+ /**
+ * Because Ranger doesn't support the precise filter, Ranger will return the
policy meets the
+ * wildcard(*,?) conditions, just like `*.*.*` policy will match
`db1.table1.column1` So we need
+ * to manual precise filter the policies.
+ */
+ // Search Ranger policy filter keys
+ protected List policyFilterKeys = null;
+ // Search Ranger policy precise filter keys
+ protected List policyPreciseFilterKeys = null;
+
+ public static final String MANAGED_BY_GRAVITINO = "MANAGED_BY_GRAVITINO";
+
+ // TODO: Maybe need to move to the configuration in the future
+ public static final String RANGER_ADMIN_NAME = "admin";
+
+ public RangerAuthorizationPlugin(String catalogProvider, Map
config) {
+super();
+this.catalogProvider = catalogProvider;
+String rangerUrl =
config.get(AuthorizationPropertiesMeta.RANGER_ADMIN_URL);
+String authType = config.get(AuthorizationPropertiesMeta.RANGER_AUTH_TYPE);
+String username = config.get(AuthorizationPropertiesMeta.RANGER_USERNAME);
+// Apache Ranger Password should be minimum 8 characters with min one
alphabet and one numeric.
+String password = config.get(AuthorizationPropertiesMeta.RANGER_PASSWORD);
+rangerServiceName =
config.get(AuthorizationPropertiesMeta.RANGER_SERVICE_NAME);
+check(rangerUrl != null, "Ranger admin URL is req
[PR] [#4573] docs(docker): Add the document for 0.6.0 [gravitino]
jerqi opened a new pull request, #4578: URL: https://github.com/apache/gravitino/pull/4578 ### What changes were proposed in this pull request? Add the document for 0.6.0 ### Why are the changes needed? (Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug.) Fix: #4573 ### Does this PR introduce _any_ user-facing change? Just document. ### How was this patch tested? No need. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Subtask] Support PaimonCatalog implementation to manage Paimon table operations [gravitino]
justinmclean commented on issue #2738: URL: https://github.com/apache/gravitino/issues/2738#issuecomment-2295835915 0.6.0 should be released in the next couple of weeks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4573] docs(docker): Add the document for 0.6.0 [gravitino]
yuqi1129 commented on PR #4578: URL: https://github.com/apache/gravitino/pull/4578#issuecomment-2295836284 This PR seems to be duplicated with #4523. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [MINOR] fix(doc): Add missing tag api in openAPI [gravitino]
jerryshao opened a new pull request, #4579: URL: https://github.com/apache/gravitino/pull/4579 ### What changes were proposed in this pull request? This PR proposes to add a missing tag openapi to list all the metadata objects for a tag. ### Why are the changes needed? This is missing to add. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual verification -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4508]feat(iceberg) support load custom catalog as backend [gravitino]
theoryxu commented on PR #4563: URL: https://github.com/apache/gravitino/pull/4563#issuecomment-2295839733 > Do you think is it necessary to support a custom catalog for the Gravitino Iceberg catalog not only for the Iceberg REST service? I want to keep a PR as simple as possible. Maybe that part putting in another PR is better？ of course, if it's confusing, I will put them together. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] fix(doc): Add missing tag api in openAPI [gravitino]
jerqi commented on code in PR #4579: URL: https://github.com/apache/gravitino/pull/4579#discussion_r1721318596 ## docs/open-api/catalogs.yaml: ## @@ -256,8 +256,10 @@ components: enum: - hive - lakehouse-iceberg +- lakehouse-paimon - jdbc-mysql - jdbc-postgresql +- jdbc-doris - hadoop Review Comment: We should add kafka catalog here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4573] docs(docker): Add the document for 0.6.0 [gravitino]
jerqi commented on PR #4578: URL: https://github.com/apache/gravitino/pull/4578#issuecomment-2295846733 > This PR seems to be duplicated with #4523. If you add the document about 0.6.0, I can close this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4573] docs(docker): Add the document for 0.6.0 [gravitino]
jerqi closed pull request #4578: [#4573] docs(docker): Add the document for 0.6.0 URL: https://github.com/apache/gravitino/pull/4578 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4072] improvement(docker-image): Transfer docker hub from datastrato to apache [gravitino]
jerqi commented on code in PR #4523: URL: https://github.com/apache/gravitino/pull/4523#discussion_r1721337994 ## docs/docker-image-details.md: ## @@ -82,23 +83,22 @@ Changelog Changelog -- trino:435-gravitino-0.5.1 - - Based on Gravitino 0.5.1, you can know more information from 0.5.1 release notes. +- apache/gravitino-playground:trino-435-gravitino-0.6.0-incubating (Switch to Apache official DockerHub repository) + - Use `datastrato/trino:435-gravitino-0.5.1` Dockerfile to rebuild the image. Review Comment: Is this comment right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4072] improvement(docker-image): Transfer docker hub from datastrato to apache [gravitino]
jerqi commented on code in PR #4523: URL: https://github.com/apache/gravitino/pull/4523#discussion_r1721342532 ## docs/docker-image-details.md: ## @@ -112,28 +112,32 @@ You can use these kinds of Docker images to facilitate integration testing of al You can use this kind of image to test the catalog of Apache Hive with kerberos enable Changelog -- gravitino-ci-kerberos-hive:0.1.5 + +- apache/gravitino-ci:kerberos-hive-0.1.5 (Switch to Apache official DockerHub repository) Review Comment: Maybe we would better use 0.1.6 here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4072] improvement(docker-image): Transfer docker hub from datastrato to apache [gravitino]
jerqi commented on PR #4523: URL: https://github.com/apache/gravitino/pull/4523#issuecomment-2295885090 After you release the images, you should update the playground, too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] fix(doc): Add missing tag api in openAPI [gravitino]
mchades commented on code in PR #4579:
URL: https://github.com/apache/gravitino/pull/4579#discussion_r1721349531
##
docs/open-api/tags.yaml:
##
@@ -284,6 +284,41 @@ paths:
"5xx":
$ref: "./openapi.yaml#/components/responses/ServerErrorResponse"
+
+ /metalakes/{metalake}/tags/{tag}/objects:
+parameters:
+ - $ref: "./openapi.yaml#/components/parameters/metalake"
+ - $ref: "./openapi.yaml#/components/parameters/tag"
+get:
+ tags:
+- tag
+ summary: list metadata objects associated with tag
Review Comment:
Is this too long since it will be treated as the title in the directory?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [Enhancement] Change exposed ports to avoid possible conflicts [gravitino-playground]
FANNG1 commented on code in PR #65: URL: https://github.com/apache/gravitino-playground/pull/65#discussion_r1721351128 ## docker-compose.yaml: ## @@ -22,9 +22,9 @@ services: image: datastrato/hive:2.7.3-no-yarn ports: - "3307:3306" - - "9000:9000" - - "9083:9083" - - "50070:50070" + - "9003:9000" Review Comment: use `19000` not `9003` for 9000 port to keep consistent with other port changes. ## docker-compose.yaml: ## @@ -22,9 +22,9 @@ services: image: datastrato/hive:2.7.3-no-yarn ports: - "3307:3306" - - "9000:9000" - - "9083:9083" - - "50070:50070" + - "9003:9000" Review Comment: use `19000` not `9003` for 9000 port to keep consistent with other port changes? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Enhancement] Change exposed ports to avoid possible conflicts [gravitino-playground]
jerryshao merged PR #65: URL: https://github.com/apache/gravitino-playground/pull/65 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(gravitino-playground) branch main updated: [Enhancement] Change exposed ports to avoid possible conflicts (#65)
This is an automated email from the ASF dual-hosted git repository. jshao pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/gravitino-playground.git The following commit(s) were added to refs/heads/main by this push: new fd68dd2 [Enhancement] Change exposed ports to avoid possible conflicts (#65) fd68dd2 is described below commit fd68dd2c59617451cf065f8e2e1d5397928464b3 Author: Shaofeng Shi AuthorDate: Mon Aug 19 15:51:53 2024 +0800 [Enhancement] Change exposed ports to avoid possible conflicts (#65) See issue description: [ #60](https://github.com/apache/gravitino-playground/issues/60) In this PR, I changed the port to services' non-default port (exclude Gravitino), this will avoid conflict if user already runs a service in local, such as mysql, pg, spark, trino, jupyter etc. I manually tested it, works well. --- README.md | 14 +++--- docker-compose.yaml | 16 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 9ae2272..f7abd71 100644 --- a/README.md +++ b/README.md @@ -34,11 +34,11 @@ The playground runs a number of services. The TCP ports used may clash with exis | Docker container | Ports used | |---|--| | playground-gravitino | 8090 9001| -| playground-hive | 3307 9000 9083 50070 | -| playground-mysql | 3306 | -| playground-postgresql | 5342 | -| playground-trino | 8080 | -| playground-jupyter| | +| playground-hive | 3307 9003 9084 50071 | +| playground-mysql | 13306| +| playground-postgresql | 15342| +| playground-trino | 18080| +| playground-jupyter| 1| ## Start playground @@ -77,7 +77,7 @@ trino@container_id:/$ trino ### Using Jupiter Notebook -1. Open the Jupyter Notebook in the browser at [http://localhost:](http://localhost:). +1. Open the Jupyter Notebook in the browser at [http://localhost:1](http://localhost:1). 2. Open the `gravitino-trino-example.ipynb` notebook. @@ -217,7 +217,7 @@ LlamaIndex as a unified data source, then use LlamaIndex and LLM to query both t non-tabular data with one natural language query. The demo is located in the `jupyter` folder, you can open the `gravitino_llama_index_demo.ipynb` -demo via Jupyter Notebook by [http://localhost:](http://localhost:). +demo via Jupyter Notebook by [http://localhost:1](http://localhost:1). The scenario of this demo is that basic structured city statistics data is stored in MySQL, and detailed city introductions are stored in PDF files. The user wants to know the answers to the diff --git a/docker-compose.yaml b/docker-compose.yaml index 658b821..90fa009 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -22,9 +22,9 @@ services: image: datastrato/hive:2.7.3-no-yarn ports: - "3307:3306" - - "9000:9000" - - "9083:9083" - - "50070:50070" + - "9003:9000" + - "9084:9083" + - "50071:50070" container_name: playground-hive environment: - HADOOP_USER_NAME=root @@ -71,7 +71,7 @@ services: trino: image: datastrato/trino:435-gravitino-0.5.1 ports: - - "8080:8080" + - "18080:8080" container_name: playground-trino environment: - HADOOP_USER_NAME=root @@ -102,7 +102,7 @@ services: POSTGRES_PASSWORD: postgres ALLOW_IP_RANGE: 0.0.0.0/0 ports: - - "5432:5432" + - "15432:5432" volumes: - ./init/postgres:/docker-entrypoint-initdb.d/ deploy: @@ -115,7 +115,7 @@ services: image: mysql:8.0 container_name: playground-mysql ports: - - "3306:3306" + - "13306:3306" volumes: - ./init/mysql:/docker-entrypoint-initdb.d/ environment: @@ -147,7 +147,7 @@ services: environment: - HADOOP_USER_NAME=root ports: - - "4040:4040" + - "14040:4040" volumes: - ./init/spark:/tmp/spark deploy: @@ -160,7 +160,7 @@ services: image: jupyter/minimal-notebook container_name: playground-jupyter ports: - - : + - 1: volumes: - ./init/jupyter:/tmp/gravitino entrypoint: /bin/bash /tmp/gravitino/init.sh
Re: [PR] [Enhancement] Change exposed ports to avoid possible conflicts [gravitino-playground]
jerryshao commented on code in PR #65: URL: https://github.com/apache/gravitino-playground/pull/65#discussion_r1721354332 ## docker-compose.yaml: ## @@ -22,9 +22,9 @@ services: image: datastrato/hive:2.7.3-no-yarn ports: - "3307:3306" - - "9000:9000" - - "9083:9083" - - "50070:50070" + - "9003:9000" Review Comment: Sorry I just merged this, @shaofengshi can you please submit another PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(gravitino-playground) branch branch-0.5 created (now fd68dd2)
This is an automated email from the ASF dual-hosted git repository. yuqi4733 pushed a change to branch branch-0.5 in repository https://gitbox.apache.org/repos/asf/gravitino-playground.git at fd68dd2 [Enhancement] Change exposed ports to avoid possible conflicts (#65) No new revisions were added by this update.
Re: [PR] [#4072] improvement(docker-image): Transfer docker hub from datastrato to apache [gravitino]
yuqi1129 commented on code in PR #4523: URL: https://github.com/apache/gravitino/pull/4523#discussion_r1721359537 ## docs/docker-image-details.md: ## @@ -112,28 +112,32 @@ You can use these kinds of Docker images to facilitate integration testing of al You can use this kind of image to test the catalog of Apache Hive with kerberos enable Changelog -- gravitino-ci-kerberos-hive:0.1.5 + +- apache/gravitino-ci:kerberos-hive-0.1.5 (Switch to Apache official DockerHub repository) Review Comment: We can keep the current version as it is because they are now different Docker repos. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4072] improvement(docker-image): Transfer docker hub from datastrato to apache [gravitino]
yuqi1129 commented on code in PR #4523: URL: https://github.com/apache/gravitino/pull/4523#discussion_r1721360446 ## docs/docker-image-details.md: ## @@ -82,23 +83,22 @@ Changelog Changelog -- trino:435-gravitino-0.5.1 - - Based on Gravitino 0.5.1, you can know more information from 0.5.1 release notes. +- apache/gravitino-playground:trino-435-gravitino-0.6.0-incubating (Switch to Apache official DockerHub repository) + - Use `datastrato/trino:435-gravitino-0.5.1` Dockerfile to rebuild the image. Review Comment: Yes, I will modify it as all the docker images are based on the release-0.6 NOT 0.5.1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Enhancement] Change exposed ports to avoid possible conflicts [gravitino-playground]
shaofengshi commented on code in PR #65: URL: https://github.com/apache/gravitino-playground/pull/65#discussion_r1721362662 ## docker-compose.yaml: ## @@ -22,9 +22,9 @@ services: image: datastrato/hive:2.7.3-no-yarn ports: - "3307:3306" - - "9000:9000" - - "9083:9083" - - "50070:50070" + - "9003:9000" Review Comment: here in the "hive:2.7.3-no-yarn", as the first port was forwarded from 3307 to 3306, so I choose the "+1" for all ports in this image; as 9001 port is already used in Mac (Port 9001 is used by the NMC server to make RPC calls between the NMC client (Java instance) and the NMC server. ), so changed to 9003, which is not used by any known services. So I think don't need to change to 19000, as it will be inconsitent with others in this docker container. @FANNG1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4460] feat(core): Add the method call of the authorizationPlugin [gravitino]
xunliu closed pull request #4461: [#4460] feat(core): Add the method call of the authorizationPlugin URL: https://github.com/apache/gravitino/pull/4461 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [FEATURE] Support fuse for Fileset [gravitino]
coolderli commented on issue #4558: URL: https://github.com/apache/gravitino/issues/4558#issuecomment-2296027918 @jerryshao @shaofengshi @xunliu Can you share your thoughts？Thanks. cc @xloya @YxAc @zhoukangcn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4508]feat(iceberg) support load custom catalog as backend [gravitino]
theoryxu commented on PR #4563: URL: https://github.com/apache/gravitino/pull/4563#issuecomment-2296045350 > Do you think is it necessary to support a custom catalog for the Gravitino Iceberg catalog not only for the Iceberg REST service? I planned to do it in a separate PR to keep this one simple, but to avoid confusion, I'll combine them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Improvement] support Gravitino-based multi catalog support for Gravitino Iceberg REST server [gravitino]
theoryxu commented on issue #4581: URL: https://github.com/apache/gravitino/issues/4581#issuecomment-2296070846 I can take it; please assign it to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(gravitino) branch branch-0.6 updated: [#4506] fix(UI): Hide jdbc-password value in details page (#4577)
This is an automated email from the ASF dual-hosted git repository.
jshao pushed a commit to branch branch-0.6
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/branch-0.6 by this push:
new 7875ba629 [#4506] fix(UI): Hide jdbc-password value in details page
(#4577)
7875ba629 is described below
commit 7875ba629157e60735d7d95554ff9ff9374d6c55
Author: Liang Chun
AuthorDate: Mon Aug 19 18:13:31 2024 +0900
[#4506] fix(UI): Hide jdbc-password value in details page (#4577)
### What changes were proposed in this pull request?
Hide jdbc-password value in details page.
### Why are the changes needed?
Fix: #4506
### Does this PR introduce _any_ user-facing change?
Jdbc-password value is not palintext now.
### How was this patch tested?
---
.../metalake/rightContent/tabsContent/detailsView/DetailsView.js| 2 +-
web/src/components/DetailsDrawer.js | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git
a/web/src/app/metalakes/metalake/rightContent/tabsContent/detailsView/DetailsView.js
b/web/src/app/metalakes/metalake/rightContent/tabsContent/detailsView/DetailsView.js
index 41313c121..6e21eabdb 100644
---
a/web/src/app/metalakes/metalake/rightContent/tabsContent/detailsView/DetailsView.js
+++
b/web/src/app/metalakes/metalake/rightContent/tabsContent/detailsView/DetailsView.js
@@ -196,7 +196,7 @@ const DetailsView = () => {
: `props-value-${item.key}`
}
>
-{item.value}
+{item.key === 'jdbc-password' ? '[HIDDEN]' :
item.value}
diff --git a/web/src/components/DetailsDrawer.js
b/web/src/components/DetailsDrawer.js
index 9479e3067..553f0e6fd 100644
--- a/web/src/components/DetailsDrawer.js
+++ b/web/src/components/DetailsDrawer.js
@@ -206,7 +206,7 @@ const DetailsDrawer = props => {
data-refer={`details-props-value-${item.value}`}
data-prev-refer={`details-props-key-${item.key}`}
>
-{item.value}
+{item.key === 'jdbc-password' ? '[HIDDEN]' :
item.value}
)
Re: [PR] [MINOR] fix(doc): Add missing tag api in openAPI [gravitino]
jerryshao merged PR #4579: URL: https://github.com/apache/gravitino/pull/4579 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(gravitino) branch main updated: [MINOR] fix(doc): Add missing tag api in openAPI (#4579)
This is an automated email from the ASF dual-hosted git repository.
jshao pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new ab20382ff [MINOR] fix(doc): Add missing tag api in openAPI (#4579)
ab20382ff is described below
commit ab20382ff3db761bd1c2217fbacd0de89ff9e29c
Author: Jerry Shao
AuthorDate: Mon Aug 19 17:14:09 2024 +0800
[MINOR] fix(doc): Add missing tag api in openAPI (#4579)
### What changes were proposed in this pull request?
This PR proposes to add a missing tag openapi to list all the metadata
objects for a tag.
### Why are the changes needed?
This is missing to add.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual verification
---
docs/open-api/catalogs.yaml | 3 ++
docs/open-api/openapi.yaml | 17 +
docs/open-api/tags.yaml | 88 +
3 files changed, 101 insertions(+), 7 deletions(-)
diff --git a/docs/open-api/catalogs.yaml b/docs/open-api/catalogs.yaml
index ab68c06ed..8ec5d0799 100644
--- a/docs/open-api/catalogs.yaml
+++ b/docs/open-api/catalogs.yaml
@@ -256,9 +256,12 @@ components:
enum:
- hive
- lakehouse-iceberg
+- lakehouse-paimon
- jdbc-mysql
- jdbc-postgresql
+- jdbc-doris
- hadoop
+- kafka
comment:
type: string
description: A comment about the catalog
diff --git a/docs/open-api/openapi.yaml b/docs/open-api/openapi.yaml
index edac71ec6..a0381bb6e 100644
--- a/docs/open-api/openapi.yaml
+++ b/docs/open-api/openapi.yaml
@@ -70,6 +70,9 @@ paths:
/metalakes/{metalake}/tags/{metadataObjectType}/{metadataObjectFullName}/{tag}:
$ref:
"./tags.yaml#/paths/~1metalakes~1%7Bmetalake%7D~1tags~1%7BmetadataObjectType%7D~1%7BmetadataObjectFullName%7D~1%7Btag%7D"
+ /metalakes/{metalake}/tags/{tag}/objects:
+$ref:
"./tags.yaml#/paths/~1metalakes~1%7Bmetalake%7D~1tags~1%7Btag%7D~1objects"
+
/metalakes/{metalake}/catalogs:
$ref: "./catalogs.yaml#/paths/~1metalakes~1%7Bmetalake%7D~1catalogs"
@@ -427,13 +430,13 @@ components:
schema:
type: string
enum:
- - "catalog"
- - "schema"
- - "table"
- - "fileset"
- - "topic"
- - "role"
- - "metalake"
+ - "CATALOG"
+ - "SCHEMA"
+ - "TABLE"
+ - "FILESET"
+ - "TOPIC"
+ - "ROLE"
+ - "METALAKE"
metadataObjectFullName:
name: metadataObjectFullName
diff --git a/docs/open-api/tags.yaml b/docs/open-api/tags.yaml
index 61d9ae1fc..54db400b4 100644
--- a/docs/open-api/tags.yaml
+++ b/docs/open-api/tags.yaml
@@ -284,6 +284,41 @@ paths:
"5xx":
$ref: "./openapi.yaml#/components/responses/ServerErrorResponse"
+
+ /metalakes/{metalake}/tags/{tag}/objects:
+parameters:
+ - $ref: "./openapi.yaml#/components/parameters/metalake"
+ - $ref: "./openapi.yaml#/components/parameters/tag"
+get:
+ tags:
+- tag
+ summary: list metadata objects for tag
+ operationId: listTagObjects
+ responses:
+"200":
+ description: Returns the list of metadata objects associated with
specified tag
+ content:
+application/vnd.gravitino.v1+json:
+ schema:
+$ref: "#/components/responses/MetadataObjectListResponse"
+ examples:
+TagResponse:
+ $ref: "#/components/examples/MetadataObjectListResponse"
+"400":
+ $ref: "./openapi.yaml#/components/responses/BadRequestErrorResponse"
+"404":
+ description: Not Found - The specified tag does not exist
+ content:
+application/vnd.gravitino.v1+json:
+ schema:
+$ref: "./openapi.yaml#/components/schemas/ErrorModel"
+ examples:
+NoSuchTagException:
+ $ref: "#/components/examples/NoSuchTagException"
+"5xx":
+ $ref: "./openapi.yaml#/components/responses/ServerErrorResponse"
+
+
components:
parameters:
details:
@@ -324,6 +359,25 @@ components:
description: Whether the tag is inherited from the parent metadata
object
nullable: true
+MetadataObject:
+ type: object
+ description: A metadata object
+ properties:
+fullName:
+ type: string
+ description: The name of the metadata object, separated by dots
+type:
+ type: string
+ description: The type of the metadata object
+ enum:
+- "CATALOG"
+- "SCHEMA"
+- "TABLE"
+- "FILESET"
+- "TOPIC"
+- "R
Re: [PR] [#4506] fix(UI): Hide jdbc-password value in details page [gravitino]
jerryshao merged PR #4577: URL: https://github.com/apache/gravitino/pull/4577 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Improvement] Avoid using known ports for playgournd docker containers [gravitino-playground]
jerqi commented on issue #60: URL: https://github.com/apache/gravitino-playground/issues/60#issuecomment-2296074137 You should close this issue because the pull request is merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug report] Create fileset schema with a strange error [gravitino]
shaofengshi opened a new issue, #4582:
URL: https://github.com/apache/gravitino/issues/4582
### Version
main branch
### Describe what's wrong
Firstly, create a hadoop catalog, for example:
`curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
"name": "schema2",
"comment": "comment",
"properties": {
"location": "hdfs://hive:9000"
}
}' http://localhost:8090/api/metalakes/metalake_demo/catalogs/hadoop2/schemas
`
Then create a schema with location:
`curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
"name": "schema2",
"comment": "comment",
"properties": {
"location": "hdfs://hive:9000/user2"
}
}' http://localhost:8090/api/metalakes/metalake_demo/catalogs/hadoop2/schemas
`
It reports an error:
`{"code":1001,"type":"IllegalArgumentException","message":"Failed to operate
schema(s) [schema2] operation [CREATE] under catalog [hadoop2], reason
[Relative path in absolute URI:
hdfs://hive:9000schema2]","stack":["java.lang.IllegalArgumentException:
java.net.URISyntaxException: Relative path in absolute URI:
hdfs://hive:9000schema2","\tat
org.apache.hadoop.fs.Path.initialize(Path.java:259)","\tat
org.apache.hadoop.fs.Path.(Path.java:157)","\tat
org.apache.hadoop.fs.Path.(Path.java:125)","\tat
com.datastrato.gravitino.catalog.hadoop.HadoopCatalogOperations.lambda$getSchemaPath$8(HadoopCatalogOperations.java:619)","\tat
java.util.Optional.map(Optional.java:215)","\tat
com.datastrato.gravitino.catalog.hadoop.HadoopCatalogOperations.getSchemaPath(HadoopCatalogOperations.java:619)","\tat
com.datastrato.gravitino.catalog.hadoop.HadoopCatalogOperations.createSchema(HadoopCatalogOperations.java:384)","\tat
com.datastrato.gravitino.catalog.SchemaOperationDispatcher.lambda$cre
ateSchema$4(SchemaOperationDispatcher.java:100)","\tat
com.datastrato.gravitino.catalog.CatalogManager$CatalogWrapper.lambda$doWithSchemaOps$0(CatalogManager.java:103)","\tat
com.datastrato.gravitino.utils.IsolatedClassLoader.withClassLoader(IsolatedClassLoader.java:72)","\tat
com.datastrato.gravitino.catalog.CatalogManager$CatalogWrapper.doWithSchemaOps(CatalogManager.java:98)","\tat
com.datastrato.gravitino.catalog.SchemaOperationDispatcher.lambda$createSchema$5(SchemaOperationDispatcher.java:100)","\tat
com.datastrato.gravitino.catalog.OperationDispatcher.doWithCatalog(OperationDispatcher.java:107)","\tat
com.datastrato.gravitino.catalog.SchemaOperationDispatcher.createSc`
Please note here: " [Relative path in absolute URI:
hdfs://hive:9000schema2]", which misses a "/" separator.
Here has two issues:
1) The system should automatically add the separator if missing;
2) As the location was specified the schema creation request, seems it
doesn't use this property, but use its catalog's storage location, and then
contact the schema name as the location.
### Error message and/or stacktrace
{"code":1001,"type":"IllegalArgumentException","message":"Failed to operate
schema(s) [schema2] operation [CREATE] under catalog [hadoop2], reason
[Relative path in absolute URI:
hdfs://hive:9000schema2]","stack":["java.lang.IllegalArgumentException:
java.net.URISyntaxException: Relative path in absolute URI:
hdfs://hive:9000schema2","\tat
org.apache.hadoop.fs.Path.initialize(Path.java:259)","\tat
org.apache.hadoop.fs.Path.(Path.java:157)","\tat
org.apache.hadoop.fs.Path.(Path.java:125)","\tat
com.datastrato.gravitino.catalog.hadoop.HadoopCatalogOperations.lambda$getSchemaPath$8(HadoopCatalogOperations.java:619)","\tat
java.util.Optional.map(Optional.java:215)","\tat
com.datastrato.gravitino.catalog.hadoop.HadoopCatalogOperations.getSchemaPath(HadoopCatalogOperations.java:619)","\tat
com.datastrato.gravitino.catalog.hadoop.HadoopCatalogOperations.createSchema(HadoopCatalogOperations.java:384)","\tat
com.datastrato.gravitino.catalog.SchemaOperationDispatcher.lambda$crea
teSchema$4(SchemaOperationDispatcher.java:100)","\tat
com.datastrato.gravitino.catalog.CatalogManager$CatalogWrapper.lambda$doWithSchemaOps$0(CatalogManager.java:103)","\tat
com.datastrato.gravitino.utils.IsolatedClassLoader.withClassLoader(IsolatedClassLoader.java:72)","\tat
com.datastrato.gravitino.catalog.CatalogManager$CatalogWrapper.doWithSchemaOps(CatalogManager.java:98)","\tat
com.datastrato.gravitino.catalog.SchemaOperationDispatcher.lambda$createSchema$5(SchemaOperationDispatcher.java:100)","\tat
com.datastrato.gravitino.catalog.OperationDispatcher.doWithCatalog(OperationDispatcher.java:107)","\tat
com.datastrato.gravitino.catalog.SchemaOperationDispatcher.createSchema(SchemaOperationDispatcher.java:98)","\tat
com.datastrato.gravitino.catalog.SchemaNormalizeDispatcher.createSchema(SchemaNormalizeDispatcher.java:47)","\tat
com.datastrato.gravitino.listener.SchemaEventDispatcher
Re: [I] [Improvement] support Gravitino-based multi catalog support for Gravitino Iceberg REST server [gravitino]
FANNG1 commented on issue #4581: URL: https://github.com/apache/gravitino/issues/4581#issuecomment-2296087436 > I can take it; please assign it to me. Thanks @theoryxu for taking this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4508]feat(iceberg) support load custom catalog as backend [gravitino]
FANNG1 commented on PR #4563: URL: https://github.com/apache/gravitino/pull/4563#issuecomment-2296091808 > > Do you think is it necessary to support a custom catalog for the Gravitino Iceberg catalog not only for the Iceberg REST service? > > I planned to do it in a separate PR to keep this one simple, but to avoid confusion, I'll combine them. I prefer to split the PR to make it more clear, could you create corresponding issues to add custom catalog backend for Graviitno server? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4508]feat(iceberg) support load custom catalog as backend [gravitino]
FANNG1 commented on code in PR #4563: URL: https://github.com/apache/gravitino/pull/4563#discussion_r1721479495 ## docs/iceberg-rest-service.md: ## @@ -162,6 +162,14 @@ If you have a JDBC Iceberg catalog prior, you must set `catalog-backend-name` to You must download the corresponding JDBC driver to the `iceberg-rest-server/libs` directory. ::: + Custom backend configuration +| Configuration item | Description | Default value| Required | Since Version | +||-|--|--|---| +| `gravitino.iceberg-rest.catalog-backend` | The Catalog backend of the Gravitino Iceberg REST catalog service. Use the value **`custom`** for a Custom catalog. | `memory` | Yes | 0.7.0 | Review Comment: should be `0.2.0` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4508]feat(iceberg) support load custom catalog as backend [gravitino]
FANNG1 commented on PR #4563: URL: https://github.com/apache/gravitino/pull/4563#issuecomment-2296098394 LGTM, except one comment -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#34] add Gravitino Spark connector to playground [gravitino-playground]
jerqi commented on PR #45: URL: https://github.com/apache/gravitino-playground/pull/45#issuecomment-2296099930 You should submit the document to the main repo, too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#51] update gravitino-fileset-example.ipynb with latest python api [gravitino-playground]
jerqi commented on PR #52: URL: https://github.com/apache/gravitino-playground/pull/52#issuecomment-2296103111 Do you need to update the version of python client? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4508]feat(iceberg) support load custom catalog as backend [gravitino]
theoryxu commented on PR #4563: URL: https://github.com/apache/gravitino/pull/4563#issuecomment-2296111724 > > > Do you think is it necessary to support a custom catalog for the Gravitino Iceberg catalog not only for the Iceberg REST service? > > > > > > I planned to do it in a separate PR to keep this one simple, but to avoid confusion, I'll combine them. > > I prefer to split the PR to make it more clear, could you create corresponding issues to add custom catalog backend for Graviitno server? 👌，the corresponding issue is here: #4584 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Subtask] Too many jobs for Backend Integration Test CI [gravitino]
mchades commented on issue #4583: URL: https://github.com/apache/gravitino/issues/4583#issuecomment-2296115518 @yuqi1129 @xunliu @jerryshao @FANNG1 WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4508]feat(iceberg) support load custom catalog as backend [gravitino]
theoryxu commented on code in PR #4563: URL: https://github.com/apache/gravitino/pull/4563#discussion_r1721494030 ## docs/iceberg-rest-service.md: ## @@ -162,6 +162,14 @@ If you have a JDBC Iceberg catalog prior, you must set `catalog-backend-name` to You must download the corresponding JDBC driver to the `iceberg-rest-server/libs` directory. ::: + Custom backend configuration +| Configuration item | Description | Default value| Required | Since Version | +||-|--|--|---| +| `gravitino.iceberg-rest.catalog-backend` | The Catalog backend of the Gravitino Iceberg REST catalog service. Use the value **`custom`** for a Custom catalog. | `memory` | Yes | 0.7.0 | Review Comment: fixed it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [MINOR] fix(doc): Add missing tag api in openAPI (#4579) [gravitino]
jerryshao opened a new pull request, #4585: URL: https://github.com/apache/gravitino/pull/4585 ### What changes were proposed in this pull request? This PR proposes to add a missing tag openapi to list all the metadata objects for a tag. ### Why are the changes needed? This is missing to add. ### Does this PR introduce _any_ user-facing change? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug report] bug about hdfs root catalog [gravitino]
heziyi399 opened a new issue, #4586: URL: https://github.com/apache/gravitino/issues/4586 ### Version main branch ### Describe what's wrong now I want to use hadoop catalog,i hava create metalake,catalog,schema,fileset,the localtion is: https://github.com/user-attachments/assets/7d8e0c56-677b-4762-aef4-8b4df02e430b";> https://github.com/user-attachments/assets/3ad09329-b99f-4c53-beee-e3541188ffd4";> https://github.com/user-attachments/assets/8e6824f0-39d8-487c-9384-db5fcf4ceb26";> You can see that this location is the root directory。 I want to get file by using gravitino catalog，so I obtain files through the command line: https://github.com/user-attachments/assets/319e21b6-7063-4ebe-8918-4ae94a9d2fe0";> https://github.com/user-attachments/assets/5bb1cfb8-bd39-43a5-a5d7-a36b8c71ce21";> You can see that this result comes with a prefix and an error message“does not exist.”.But if I don't use the location of the root directory ,the result is normal： https://github.com/user-attachments/assets/b5610884-326a-432e-b02a-850bbfd40b1e";> https://github.com/user-attachments/assets/9535f229-5c60-4301-bf78-25cccf6bf502";> ### Error message and/or stacktrace https://github.com/user-attachments/assets/bc6672f5-c4d3-4716-a2da-4da36442b157";> ### How to reproduce 0.5.1 ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Subtask] Too many jobs for Backend Integration Test CI [gravitino]
jerryshao commented on issue #4583: URL: https://github.com/apache/gravitino/issues/4583#issuecomment-2296165997 Yeah, we can take a try to reduce the task number, there're too many tasks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Enhancement] Change exposed ports to avoid possible conflicts [gravitino-playground]
FANNG1 commented on code in PR #65: URL: https://github.com/apache/gravitino-playground/pull/65#discussion_r1721540676 ## docker-compose.yaml: ## @@ -22,9 +22,9 @@ services: image: datastrato/hive:2.7.3-no-yarn ports: - "3307:3306" - - "9000:9000" - - "9083:9083" - - "50070:50070" + - "9003:9000" Review Comment: It's a little complex to know the rules, I prefer to keep a simple +1 rule -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#34] add Gravitino Spark connector to playground [gravitino-playground]
FANNG1 commented on PR #45: URL: https://github.com/apache/gravitino-playground/pull/45#issuecomment-2296214111 > Will it influence the demo of Iceberg REST catalog? Is it clear for user to use Spark connector and REST service catalog. It is a confusing for user. The original document makes Spark and Trino use same catalog name `iceberg_catalog`, but they are different, maybe we could add some comment to tell user `iceberg_catalog` and `rest_catalog` refer to the same underlying catalog, WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug report] bug about hdfs root catalog [gravitino]
jerryshao commented on issue #4586: URL: https://github.com/apache/gravitino/issues/4586#issuecomment-2296218052 @xloya would you please take a look at this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [FEATURE] Add the method call of authorization plugin [gravitino]
xunliu closed issue #4460: [FEATURE] Add the method call of authorization plugin URL: https://github.com/apache/gravitino/issues/4460 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4460] feat(core): Add the method call of the authorizationPlugin [gravitino]
xunliu merged PR #4461: URL: https://github.com/apache/gravitino/pull/4461 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(gravitino) branch main updated: [#4460] feat(core): Add the method call of the authorizationPlugin (#4461)
This is an automated email from the ASF dual-hosted git repository.
liuxun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 79fda5fe3 [#4460] feat(core): Add the method call of the
authorizationPlugin (#4461)
79fda5fe3 is described below
commit 79fda5fe3be322b68ae782fc67107425891d1c9c
Author: roryqi
AuthorDate: Mon Aug 19 18:24:10 2024 +0800
[#4460] feat(core): Add the method call of the authorizationPlugin (#4461)
### What changes were proposed in this pull request?
If we want to push the privileges down the underlying system, we need to
call methods of the underlying system authorization plugin.
### Why are the changes needed?
Fix: #4460
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Add some test cases.
---
.../java/org/apache/gravitino/GravitinoEnv.java| 18 +
.../gravitino/SupportsRelationOperations.java | 8 +-
.../authorization/AuthorizationUtils.java | 77
.../authorization/FutureGrantManager.java | 133 +++
.../gravitino/authorization/PermissionManager.java | 392 -
.../gravitino/authorization/RoleManager.java | 19 +
.../gravitino/hook/CatalogHookDispatcher.java | 10 +
.../gravitino/storage/relational/JDBCBackend.java | 18 +
.../relational/mapper/MetalakeMetaMapper.java | 2 +-
.../storage/relational/mapper/RoleMetaMapper.java | 17 +
.../relational/service/GroupMetaService.java | 18 +
.../relational/service/OwnerMetaService.java | 19 +-
.../relational/service/RoleMetaService.java| 67 +++-
.../relational/service/UserMetaService.java| 18 +
.../apache/gravitino/utils/NameIdentifierUtil.java | 14 +
.../authorization/TestAccessControlManager.java| 20 ++
.../TestAccessControlManagerForPermissions.java| 34 ++
.../authorization/TestFutureGrantManager.java | 170 +
.../relational/service/TestRoleMetaService.java| 100 ++
.../gravitino/server/web/rest/RoleOperations.java | 99 +++---
20 files changed, 1001 insertions(+), 252 deletions(-)
diff --git a/core/src/main/java/org/apache/gravitino/GravitinoEnv.java
b/core/src/main/java/org/apache/gravitino/GravitinoEnv.java
index 6e97ed1eb..04de93186 100644
--- a/core/src/main/java/org/apache/gravitino/GravitinoEnv.java
+++ b/core/src/main/java/org/apache/gravitino/GravitinoEnv.java
@@ -21,6 +21,7 @@ package org.apache.gravitino;
import com.google.common.base.Preconditions;
import org.apache.gravitino.authorization.AccessControlDispatcher;
import org.apache.gravitino.authorization.AccessControlManager;
+import org.apache.gravitino.authorization.FutureGrantManager;
import org.apache.gravitino.authorization.OwnerManager;
import org.apache.gravitino.auxiliary.AuxiliaryServiceManager;
import org.apache.gravitino.catalog.CatalogDispatcher;
@@ -111,6 +112,7 @@ public class GravitinoEnv {
private TagManager tagManager;
private EventBus eventBus;
private OwnerManager ownerManager;
+ private FutureGrantManager futureGrantManager;
protected GravitinoEnv() {}
@@ -287,10 +289,24 @@ public class GravitinoEnv {
return tagManager;
}
+ /**
+ * Get the OwnerManager associated with the Gravitino environment.
+ *
+ * @return The OwnerManager instance.
+ */
public OwnerManager ownerManager() {
return ownerManager;
}
+ /**
+ * Get the FutureGrantManager associated with the Gravitino environment.
+ *
+ * @return The FutureGrantManager instance.
+ */
+ public FutureGrantManager futureGrantManager() {
+return futureGrantManager;
+ }
+
public void start() {
auxServiceManager.serviceStart();
metricsSystem.start();
@@ -410,9 +426,11 @@ public class GravitinoEnv {
this.accessControlDispatcher = accessControlHookDispatcher;
this.ownerManager = new OwnerManager(entityStore);
+ this.futureGrantManager = new FutureGrantManager(entityStore);
} else {
this.accessControlDispatcher = null;
this.ownerManager = null;
+ this.futureGrantManager = null;
}
this.auxServiceManager = new AuxiliaryServiceManager();
diff --git
a/core/src/main/java/org/apache/gravitino/SupportsRelationOperations.java
b/core/src/main/java/org/apache/gravitino/SupportsRelationOperations.java
index 5a63ceee0..617f72ab9 100644
--- a/core/src/main/java/org/apache/gravitino/SupportsRelationOperations.java
+++ b/core/src/main/java/org/apache/gravitino/SupportsRelationOperations.java
@@ -30,7 +30,13 @@ public interface SupportsRelationOperations {
/** Relation is an abstraction which connects two entities. */
enum Type {
/** The owner relationship */
-OWNER_REL
+OWNER_REL,
+/** Metadata objet and role relationship */
+METADATA_OBJECT_ROLE_REL,
+/** Role and user relationship
Re: [I] [FEATURE] Add the method call of authorization plugin [gravitino]
xunliu closed issue #4460: [FEATURE] Add the method call of authorization plugin URL: https://github.com/apache/gravitino/issues/4460 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
xunliu commented on PR #4515: URL: https://github.com/apache/gravitino/pull/4515#issuecomment-2296236058 @yuqi1129 I fixed all problems based on your comments, Please help me review again. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [FEATURE] Support associating tags when creating a metadata object [gravitino]
LauraXia123 opened a new issue, #4588: URL: https://github.com/apache/gravitino/issues/4588 ### Describe the feature We can associate tags with a existing metadata object right now. Need to support associating tags when creating a metadata object ### Motivation _No response_ ### Describe the solution _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [FEATURE] Support fuse for Fileset [gravitino]
jerryshao commented on issue #4558: URL: https://github.com/apache/gravitino/issues/4558#issuecomment-2296282629 Thanks @coolderli to bring this out. I think HCFS API is not Posix compliant API, so using HCFS API with fuse has many limitations. I don't know how well fsspec support Posix, we need to investigate. The concern from my side is the performance of using python to achieve fuse, using fuse requires lots of context switch (between user space and kernel space) and it will affect the performance a lot. If we use a dynamic language, then the performance will be worse. Currently, we don't have a better solution, maybe we should investigate more to have a better solution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
yuqi1129 commented on code in PR #4515:
URL: https://github.com/apache/gravitino/pull/4515#discussion_r1721492852
##
authorizations/authorization-ranger/src/main/java/org/apache/gravitino/authorization/ranger/RangerClientExt.java:
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.authorization.ranger;
+
+import com.google.common.collect.ImmutableMap;
+import com.sun.jersey.api.client.GenericType;
+import com.sun.jersey.api.client.UniformInterfaceException;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.Map;
+import javax.ws.rs.HttpMethod;
+import javax.ws.rs.core.Response;
+import org.apache.gravitino.authorization.ranger.defines.VXGroup;
+import org.apache.gravitino.authorization.ranger.defines.VXGroupList;
+import org.apache.gravitino.authorization.ranger.defines.VXUser;
+import org.apache.gravitino.authorization.ranger.defines.VXUserList;
+import org.apache.ranger.RangerClient;
+import org.apache.ranger.RangerServiceException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Apache Ranger client extension
+ * The class extends the RangerClient class and provides additional methods to
create, search and
+ * delete users and groups
+ */
+public class RangerClientExt extends RangerClient {
+ private static final Logger LOG =
LoggerFactory.getLogger(RangerClientExt.class);
+ private static final String URI_USER_BASE = "/service/xusers/users";
+ private static final String URI_USER_BY_ID = URI_USER_BASE + "/%d";
+ private static final String URI_GROUP_BASE = "/service/xusers/groups";
+ private static final String URI_GROUP_BY_ID = URI_GROUP_BASE + "/%d";
+ private static final String URI_CREATE_EXTERNAL_USER = URI_USER_BASE +
"/external";
+
+ // Ranger user APIs
+ private static final API SEARCH_USER = new API(URI_USER_BASE,
HttpMethod.GET, Response.Status.OK);
+ private static final API CREATE_EXTERNAL_USER =
+ new API(URI_CREATE_EXTERNAL_USER, HttpMethod.POST, Response.Status.OK);
+ private static final API DELETE_USER =
+ new API(URI_USER_BY_ID, HttpMethod.DELETE, Response.Status.NO_CONTENT);
+
+ // Ranger group APIs
+ private static final API CREATE_GROUP =
+ new API(URI_GROUP_BASE, HttpMethod.POST, Response.Status.OK);
+ private static final API SEARCH_GROUP =
+ new API(URI_GROUP_BASE, HttpMethod.GET, Response.Status.OK);
+ // private static final API GET_GROUP = new API(URI_GROUP_BY_ID,
HttpMethod.GET,
+ // Response.Status.OK);
+ private static final API DELETE_GROUP =
+ new API(URI_GROUP_BY_ID, HttpMethod.DELETE, Response.Status.NO_CONTENT);
+
+ // apache/ranger/intg/src/main/java/org/apache/ranger/RangerClient.java
+ // The private method callAPI of Ranger is called by reflection
+ // private T callAPI(API api, Map params, Object
request, GenericType
+ // responseType) throws RangerServiceException
+ private Method callAPIMethodGenericResponseType;
+
+ // private T callAPI(API api, Map params, Object
request, Class
+ // responseType) throws RangerServiceException
+ private Method callAPIMethodClassResponseType;
+ // private void callAPI(API api, Map params) throws
RangerServiceException
+ private Method callAPIMethodNonResponse;
+
+ public RangerClientExt(String hostName, String authType, String username,
String password) {
+super(hostName, authType, username, password, null);
+
+// initialize callAPI method
+try {
+ callAPIMethodGenericResponseType =
+ RangerClient.class.getDeclaredMethod(
+ "callAPI", API.class, Map.class, Object.class,
GenericType.class);
+ callAPIMethodGenericResponseType.setAccessible(true);
+
+ callAPIMethodNonResponse =
+ RangerClient.class.getDeclaredMethod("callAPI", API.class,
Map.class);
+ callAPIMethodNonResponse.setAccessible(true);
+
+ callAPIMethodClassResponseType =
+ RangerClient.class.getDeclaredMethod(
+ "callAPI", API.class, Map.class, Object.class, Class.class);
+ callAPIMethodClassResponseType.setAccessible(true);
+} catch (NoSuchMethodException e) {
+ throw new RuntimeException(e);
+
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
jerqi commented on code in PR #4515:
URL: https://github.com/apache/gravitino/pull/4515#discussion_r1721648253
##
api/src/main/java/org/apache/gravitino/authorization/RoleChange.java:
##
@@ -149,7 +189,101 @@ public int hashCode() {
*/
@Override
public String toString() {
- return "REMOVESECURABLEOBJECT " + securableObject;
+ return "REMOVESECURABLEOBJECT " + roleName + " " + securableObject;
+}
+ }
+
+ /**
+ * A UpdateSecurableObject is to update securable object's privilege from
role.
+ * The securable object's metadata entity must be the same as new securable
object's metadata
+ * entity.
+ * The securable object's privilege must be different as new securable
object's privilege.
+ */
+ final class UpdateSecurableObject implements RoleChange {
+private final String roleName;
+private final SecurableObject securableObject;
+private final SecurableObject newSecurableObject;
+
+private UpdateSecurableObject(
+String roleName, SecurableObject securableObject, SecurableObject
newSecurableObject) {
+ if (!securableObject.fullName().equals(newSecurableObject.fullName())) {
+throw new IllegalArgumentException(
+"The securable object's metadata entity must be same as new
securable object's metadata entity.");
+ }
+ if
(securableObject.privileges().containsAll(newSecurableObject.privileges())) {
Review Comment:
Why do we need this limit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
jerqi commented on PR #4515: URL: https://github.com/apache/gravitino/pull/4515#issuecomment-2296375571 If we set a schema the privilege `read table`, will it present schema.* in the Ranger? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4089] fix(hive catalog): the problem of slow acquisition of hive table list [gravitino]
yuqi1129 closed pull request #4469: [#4089] fix(hive catalog): the problem of slow acquisition of hive table list URL: https://github.com/apache/gravitino/pull/4469 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
jerqi commented on code in PR #4515:
URL: https://github.com/apache/gravitino/pull/4515#discussion_r1721662285
##
core/src/main/java/org/apache/gravitino/connector/authorization/UserGroupAuthorizationPlugin.java:
##
@@ -98,46 +98,15 @@ interface UserGroupAuthorizationPlugin {
Boolean onGroupAcquired(Group group) throws RuntimeException;
/**
- * After granting roles to a user from Gravitino, this method is called to
grant roles to the user
- * in the underlying system.
- *
- * @param user The entity of the User.
- * @param roles The entities of the Roles.
- * @return True if the Grant was successful, false if the Grant was failed.
- * @throws RuntimeException If granting roles to a user encounters storage
issues.
- */
- Boolean onGrantedRolesToUser(List roles, User user) throws
RuntimeException;
-
- /**
- * After revoking roles from a user from Gravitino, this method is called to
revoke roles from the
- * user in the underlying system.
- *
- * @param user The entity of the User.
- * @param roles The entities of the Roles.
- * @return True if the revoke was successfully removed, false if the revoke
failed.
- * @throws RuntimeException If revoking roles from a user encounters storage
issues.
- */
- Boolean onRevokedRolesFromUser(List roles, User user) throws
RuntimeException;
-
- /**
- * After granting roles to a group from Gravitino, this method is called to
grant roles to the
- * group in the underlying system.
- *
- * @param group The entity of the Group.
- * @param roles The entities of the Roles.
- * @return True if the revoke was successfully removed, False if the revoke
failed.
- * @throws RuntimeException If granting roles to a group encounters storage
issues.
- */
- Boolean onGrantedRolesToGroup(List roles, Group group) throws
RuntimeException;
-
- /**
- * After revoking roles from a group from Gravitino, this method is called
to revoke roles from
- * the group in the underlying system.
+ * After set a Owner to Gravitino, this method is called to set the Owner to
the underlying
+ * system.
*
- * @param group The entity of the Group.
- * @param roles The entities of the Roles.
- * @return True if the revoke was successfully removed, False if the revoke
failed.
- * @throws RuntimeException If revoking roles from a group encounters
storage issues.
+ * @param metadataObject The metadata entity.
+ * @param preOwner The previous owner.
+ * @param newOwner The new owner.
+ * @return True if the set Owner was successfully set, false if the set
Owner failed.
+ * @throws RuntimeException If adding the Group encounters storage issues.
*/
- Boolean onRevokedRolesFromGroup(List roles, Group group) throws
RuntimeException;
+ Boolean onOwnerSet(MetadataObject metadataObject, Owner preOwner, Owner
newOwner)
Review Comment:
We don't call this method yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
jerqi commented on code in PR #4515:
URL: https://github.com/apache/gravitino/pull/4515#discussion_r1721675565
##
authorizations/authorization-ranger/src/main/java/org/apache/gravitino/authorization/ranger/RangerHiveAuthorizationPlugin.java:
##
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.authorization.ranger;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Set;
+import org.apache.gravitino.authorization.Privilege;
+
+/**
+ * RangerHiveAuthorizationPlugin is a plugin for Apache Ranger to manage the
Hive authorization of
+ * the Apache Gravitino.
+ */
+public class RangerHiveAuthorizationPlugin extends RangerAuthorizationPlugin {
+ public RangerHiveAuthorizationPlugin(String catalogProvider, Map config) {
+super(catalogProvider, config);
+ }
+
+ /**
+ * Ranger hive's privilege have `select`, `update`, `create`, `drop`,
`alter`, `index`, `lock`,
+ * `read`, `write`, `repladmin`, `serviceadmin`, `refresh` and `all`.
Reference:
+ *
ranger/agents-common/src/main/resources/service-defs/ranger-servicedef-hive.json
+ */
+ @Override
+ protected void initMapPrivileges() {
+mapPrivileges =
+ImmutableMap.>builder()
Review Comment:
Before we read or write a table, we should have `use catalog` and `use
schema` privilege first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#50] Add support for helm chart [gravitino-playground]
danhuawang commented on code in PR #56: URL: https://github.com/apache/gravitino-playground/pull/56#discussion_r1721676175 ## helm-chart/values.yaml: ## @@ -0,0 +1,150 @@ +# Global settings +global: + storageClass: "" + namespace: gravitino-playground + +# Hive settings +hive: + image: +repository: datastrato/hive +tag: 2.7.3-no-yarn +pullPolicy: IfNotPresent + resources: +limits: + cpu: 3 + memory: 3Gi +requests: + cpu: 1 + memory: 1Gi + +# Gravitino settings +gravitino: + image: +repository: datastrato/gravitino +tag: 0.5.1 +pullPolicy: IfNotPresent + resources: +limits: + cpu: 500m + memory: 500Mi +requests: + cpu: 200m + memory: 200Mi + +# Trino settings +trino: + image: +repository: datastrato/trino +tag: 435-gravitino-0.5.1 +pullPolicy: IfNotPresent + resources: +limits: + cpu: 1 + memory: 2Gi +requests: + cpu: 500m + memory: 1Gi + +# PostgreSQL settings +postgresql: + image: +repository: postgres +tag: "13" +pullPolicy: IfNotPresent + resources: +limits: + cpu: 1 + memory: 500Mi +requests: + cpu: 200m + memory: 200Mi + persistence: +enabled: true +size: 1Gi + +# MySQL settings +mysql: + image: +repository: mysql +tag: "8.0" +pullPolicy: IfNotPresent + resources: +limits: + cpu: 1 + memory: 500Mi +requests: + cpu: 200m + memory: 200Mi + persistence: +enabled: true +size: 1Gi + +# Spark settings +spark: + image: +repository: spark +tag: 3.4.1-scala2.12-java11-ubuntu +pullPolicy: IfNotPresent + resources: +limits: + cpu: 1 + memory: 1Gi +requests: + cpu: 500m + memory: 500Mi + +# Jupyter settings +jupyter: + image: +repository: jupyter/minimal-notebook +tag: latest +pullPolicy: IfNotPresent + resources: +limits: + cpu: 500m + memory: 500Mi +requests: + cpu: 200m + memory: 200Mi + persistence: +enabled: true +size: 1Gi + +# Service specific settings +service: + type: ClusterIP Review Comment: Yes, I agreed to use ClusterIP in demo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#50] Add support for helm chart [gravitino-playground]
danhuawang commented on code in PR #56: URL: https://github.com/apache/gravitino-playground/pull/56#discussion_r1721680582 ## helm-chart/values.yaml: ## @@ -0,0 +1,150 @@ +# Global settings +global: + storageClass: "" + namespace: gravitino-playground + +# Hive settings +hive: + image: +repository: datastrato/hive +tag: 2.7.3-no-yarn +pullPolicy: IfNotPresent + resources: +limits: + cpu: 3 + memory: 3Gi +requests: + cpu: 1 + memory: 1Gi + +# Gravitino settings +gravitino: + image: +repository: datastrato/gravitino +tag: 0.5.1 +pullPolicy: IfNotPresent + resources: +limits: + cpu: 500m + memory: 500Mi +requests: + cpu: 200m + memory: 200Mi + +# Trino settings +trino: + image: +repository: datastrato/trino +tag: 435-gravitino-0.5.1 +pullPolicy: IfNotPresent + resources: +limits: + cpu: 1 + memory: 2Gi +requests: + cpu: 500m + memory: 1Gi + +# PostgreSQL settings +postgresql: + image: +repository: postgres +tag: "13" +pullPolicy: IfNotPresent + resources: +limits: + cpu: 1 + memory: 500Mi +requests: + cpu: 200m + memory: 200Mi + persistence: +enabled: true +size: 1Gi + +# MySQL settings +mysql: + image: +repository: mysql +tag: "8.0" +pullPolicy: IfNotPresent + resources: +limits: + cpu: 1 + memory: 500Mi +requests: + cpu: 200m + memory: 200Mi + persistence: +enabled: true +size: 1Gi + +# Spark settings +spark: + image: +repository: spark +tag: 3.4.1-scala2.12-java11-ubuntu +pullPolicy: IfNotPresent + resources: +limits: + cpu: 1 + memory: 1Gi +requests: + cpu: 500m + memory: 500Mi + +# Jupyter settings +jupyter: + image: +repository: jupyter/minimal-notebook +tag: latest +pullPolicy: IfNotPresent + resources: +limits: + cpu: 500m + memory: 500Mi +requests: + cpu: 200m + memory: 200Mi + persistence: +enabled: true +size: 1Gi + +# Service specific settings +service: + type: ClusterIP Review Comment: Ingress is not necessary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#50] Add support for helm chart [gravitino-playground]
danhuawang commented on PR #56: URL: https://github.com/apache/gravitino-playground/pull/56#issuecomment-2296416879 I didn't see the dependency logic in docker-compose.yaml that is implemented in helm chart yaml. I think these dependencies are important for pod launch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#50] Add support for helm chart [gravitino-playground]
danhuawang commented on PR #56: URL: https://github.com/apache/gravitino-playground/pull/56#issuecomment-2296442320 After I try to access gravitino ui , gravitino pod is restarting. I think the resources for the pod of gravitino is not enough . After I changed to the following configuration, it don't restart again. ``` gravitino: serviceName: &gravitino_host_ip gravitino image: repository: datastrato/gravitino tag: 0.5.1 pullPolicy: IfNotPresent resources: limits: cpu: 1000m memory: 2Gi requests: cpu: 500m memory: 1000Mi ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
xunliu commented on PR #4515: URL: https://github.com/apache/gravitino/pull/4515#issuecomment-2296473673 > If we set a schema the privilege read table, will it present schema.* in the Ranger? Yes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
lw-yang commented on code in PR #4515:
URL: https://github.com/apache/gravitino/pull/4515#discussion_r1721691630
##
authorizations/authorization-ranger/src/main/java/org/apache/gravitino/authorization/ranger/RangerAuthorizationPlugin.java:
##
@@ -0,0 +1,1015 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.authorization.ranger;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.errorprone.annotations.FormatMethod;
+import com.google.errorprone.annotations.FormatString;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.stream.Collectors;
+import org.apache.gravitino.MetadataObject;
+import org.apache.gravitino.authorization.Group;
+import org.apache.gravitino.authorization.Owner;
+import org.apache.gravitino.authorization.Privilege;
+import org.apache.gravitino.authorization.Role;
+import org.apache.gravitino.authorization.RoleChange;
+import org.apache.gravitino.authorization.SecurableObject;
+import org.apache.gravitino.authorization.SecurableObjects;
+import org.apache.gravitino.authorization.User;
+import org.apache.gravitino.authorization.ranger.defines.VXGroup;
+import org.apache.gravitino.authorization.ranger.defines.VXGroupList;
+import org.apache.gravitino.authorization.ranger.defines.VXUser;
+import org.apache.gravitino.authorization.ranger.defines.VXUserList;
+import org.apache.gravitino.connector.AuthorizationPropertiesMeta;
+import org.apache.gravitino.connector.authorization.AuthorizationPlugin;
+import org.apache.gravitino.exceptions.AuthorizationPluginException;
+import org.apache.ranger.RangerServiceException;
+import org.apache.ranger.plugin.model.RangerPolicy;
+import org.apache.ranger.plugin.model.RangerRole;
+import org.apache.ranger.plugin.util.GrantRevokeRoleRequest;
+import org.apache.ranger.plugin.util.SearchFilter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Ranger authorization operations plugin abstract class. */
+public abstract class RangerAuthorizationPlugin implements AuthorizationPlugin
{
+ private static final Logger LOG =
LoggerFactory.getLogger(RangerAuthorizationPlugin.class);
+
+ protected String catalogProvider;
+ protected RangerClientExt rangerClient;
+ protected String rangerServiceName;
+ /** Mapping Gravitino privilege name to the underlying authorization system
privileges. */
+ protected Map> mapPrivileges = null;
+ // The owner privileges, the owner can do anything on the metadata object
+ protected Set ownerPrivileges = null;
+
+ /**
+ * Because Ranger doesn't support the precise filter, Ranger will return the
policy meets the
+ * wildcard(*,?) conditions, just like `*.*.*` policy will match
`db1.table1.column1` So we need
+ * to manual precise filter the policies.
+ */
+ // Search Ranger policy filter keys
+ protected List policyFilterKeys = null;
+ // Search Ranger policy precise filter keys
+ protected List policyPreciseFilterKeys = null;
+
+ public static final String MANAGED_BY_GRAVITINO = "MANAGED_BY_GRAVITINO";
+
+ // TODO: Maybe need to move to the configuration in the future
+ public static final String RANGER_ADMIN_NAME = "admin";
+
+ public RangerAuthorizationPlugin(String catalogProvider, Map
config) {
+super();
+this.catalogProvider = catalogProvider;
+String rangerUrl =
config.get(AuthorizationPropertiesMeta.RANGER_ADMIN_URL);
+String authType = config.get(AuthorizationPropertiesMeta.RANGER_AUTH_TYPE);
+String username = config.get(AuthorizationPropertiesMeta.RANGER_USERNAME);
+// Apache Ranger Password should be minimum 8 characters with min one
alphabet and one numeric.
+String password = config.get(AuthorizationPropertiesMeta.RANGER_PASSWORD);
+rangerServiceName =
config.get(AuthorizationPropertiesMeta.RANGER_SERVICE_NAME);
+check(rangerUrl != null, "Ranger admin URL is requ
Re: [PR] [#4508]feat(iceberg) support load custom catalog as backend in the Iceberg REST service [gravitino]
FANNG1 merged PR #4563: URL: https://github.com/apache/gravitino/pull/4563 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(gravitino) branch main updated: [#4508]feat(iceberg-rest-server): support load custom catalog as backend in the Iceberg REST service (#4563)
This is an automated email from the ASF dual-hosted git repository.
fanng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 837b53b1c [#4508]feat(iceberg-rest-server): support load custom
catalog as backend in the Iceberg REST service (#4563)
837b53b1c is described below
commit 837b53b1cf4d273756ae98e1ca4e8a9a2e1124d3
Author: theoryxu
AuthorDate: Mon Aug 19 21:03:40 2024 +0800
[#4508]feat(iceberg-rest-server): support load custom catalog as backend in
the Iceberg REST service (#4563)
### What changes were proposed in this pull request?
support load custom catalog as backend
### Why are the changes needed?
improvement: #4508
### Does this PR introduce _any_ user-facing change?
1. add a property key
### How was this patch tested?
1. add UT
2. manual test
-
Co-authored-by: theoryxu
---
.../lakehouse/iceberg/IcebergConstants.java| 1 +
docs/iceberg-rest-service.md | 8 +
.../iceberg/common/IcebergCatalogBackend.java | 3 +-
.../gravitino/iceberg/common/IcebergConfig.java| 8 +
.../iceberg/common/utils/IcebergCatalogUtil.java | 13 +
.../common/utils/CustomCatalogForTest.java}| 32
.../common/utils/TestIcebergCatalogUtil.java | 34 ++
7 files changed, 92 insertions(+), 7 deletions(-)
diff --git
a/catalogs/catalog-common/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergConstants.java
b/catalogs/catalog-common/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergConstants.java
index 6e4aae37a..d07a6c487 100644
---
a/catalogs/catalog-common/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergConstants.java
+++
b/catalogs/catalog-common/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergConstants.java
@@ -21,6 +21,7 @@ package org.apache.gravitino.catalog.lakehouse.iceberg;
public class IcebergConstants {
// Iceberg catalog properties constants
public static final String CATALOG_BACKEND = "catalog-backend";
+ public static final String CATALOG_BACKEND_IMPL = "catalog-backend-impl";
public static final String GRAVITINO_JDBC_USER = "jdbc-user";
public static final String ICEBERG_JDBC_USER = "jdbc.user";
diff --git a/docs/iceberg-rest-service.md b/docs/iceberg-rest-service.md
index c1a4a8111..1753cc949 100644
--- a/docs/iceberg-rest-service.md
+++ b/docs/iceberg-rest-service.md
@@ -162,6 +162,14 @@ If you have a JDBC Iceberg catalog prior, you must set
`catalog-backend-name` to
You must download the corresponding JDBC driver to the
`iceberg-rest-server/libs` directory.
:::
+ Custom backend configuration
+| Configuration item | Description
| Default value| Required | Since Version |
+||-|--|--|---|
+| `gravitino.iceberg-rest.catalog-backend` | The Catalog backend of the
Gravitino Iceberg REST catalog service. Use the value **`custom`** for a Custom
catalog. | `memory` | Yes | 0.2.0 |
+| `gravitino.iceberg-rest.catalog-backend-impl` | The fully-qualified class
name of a custom catalog implementation, only worked if `catalog-backend` is
`custom`.| (none) | No | 0.7.0 |
+
+If you want to use a custom Iceberg Catalog as `catalog-backend`, you can add
a corresponding jar file to the classpath and load a custom Iceberg Catalog
implementation by specifying the `catalog-backend-impl` property.
+
Multi catalog support
The Gravitino Iceberg REST server supports multiple catalogs and offers a
configuration-based catalog management system.
diff --git
a/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/IcebergCatalogBackend.java
b/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/IcebergCatalogBackend.java
index 63fb07605..4cdedc826 100644
---
a/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/IcebergCatalogBackend.java
+++
b/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/IcebergCatalogBackend.java
@@ -22,5 +22,6 @@ public enum IcebergCatalogBackend {
HIVE,
JDBC,
MEMORY,
- REST
+ REST,
+ CUSTOM
}
diff --git
a/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/IcebergConfig.java
b/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/IcebergConfig.java
index b75fc88d6..bc1e47a06 100644
---
a/iceberg/iceberg-common/src/main/java/or
Re: [PR] [#4508]feat(iceberg) support load custom catalog as backend in the Iceberg REST service [gravitino]
FANNG1 commented on PR #4563: URL: https://github.com/apache/gravitino/pull/4563#issuecomment-2296531124 @theoryxu , merged to main, thanks for your contribution -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug report] bug about hdfs root catalog [gravitino]
xloya commented on issue #4586: URL: https://github.com/apache/gravitino/issues/4586#issuecomment-2296532624 Have reproduced the issue, will fix this tomorrow. @heziyi399 Thanks for reporting this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Improvement] Support loading custom catalog as backend In Iceberg rest catalog server [gravitino]
FANNG1 closed issue #4508: [Improvement] Support loading custom catalog as backend In Iceberg rest catalog server URL: https://github.com/apache/gravitino/issues/4508 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] fix(doc): Add missing tag api in openAPI (#4579) [gravitino]
jerryshao merged PR #4585: URL: https://github.com/apache/gravitino/pull/4585 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(gravitino) branch branch-0.6 updated: [MINOR] fix(doc): Add missing tag api in openAPI (#4579) (#4585)
This is an automated email from the ASF dual-hosted git repository.
jshao pushed a commit to branch branch-0.6
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/branch-0.6 by this push:
new d440eed33 [MINOR] fix(doc): Add missing tag api in openAPI (#4579)
(#4585)
d440eed33 is described below
commit d440eed33937527b8804aaf7a9b61afca3d19157
Author: Jerry Shao
AuthorDate: Mon Aug 19 22:24:25 2024 +0800
[MINOR] fix(doc): Add missing tag api in openAPI (#4579) (#4585)
### What changes were proposed in this pull request?
This PR proposes to add a missing tag openapi to list all the metadata
objects for a tag.
### Why are the changes needed?
This is missing to add.
### Does this PR introduce _any_ user-facing change?
No.
---
docs/open-api/catalogs.yaml | 3 ++
docs/open-api/openapi.yaml | 17 +
docs/open-api/tags.yaml | 88 +
3 files changed, 101 insertions(+), 7 deletions(-)
diff --git a/docs/open-api/catalogs.yaml b/docs/open-api/catalogs.yaml
index ab68c06ed..8ec5d0799 100644
--- a/docs/open-api/catalogs.yaml
+++ b/docs/open-api/catalogs.yaml
@@ -256,9 +256,12 @@ components:
enum:
- hive
- lakehouse-iceberg
+- lakehouse-paimon
- jdbc-mysql
- jdbc-postgresql
+- jdbc-doris
- hadoop
+- kafka
comment:
type: string
description: A comment about the catalog
diff --git a/docs/open-api/openapi.yaml b/docs/open-api/openapi.yaml
index edac71ec6..a0381bb6e 100644
--- a/docs/open-api/openapi.yaml
+++ b/docs/open-api/openapi.yaml
@@ -70,6 +70,9 @@ paths:
/metalakes/{metalake}/tags/{metadataObjectType}/{metadataObjectFullName}/{tag}:
$ref:
"./tags.yaml#/paths/~1metalakes~1%7Bmetalake%7D~1tags~1%7BmetadataObjectType%7D~1%7BmetadataObjectFullName%7D~1%7Btag%7D"
+ /metalakes/{metalake}/tags/{tag}/objects:
+$ref:
"./tags.yaml#/paths/~1metalakes~1%7Bmetalake%7D~1tags~1%7Btag%7D~1objects"
+
/metalakes/{metalake}/catalogs:
$ref: "./catalogs.yaml#/paths/~1metalakes~1%7Bmetalake%7D~1catalogs"
@@ -427,13 +430,13 @@ components:
schema:
type: string
enum:
- - "catalog"
- - "schema"
- - "table"
- - "fileset"
- - "topic"
- - "role"
- - "metalake"
+ - "CATALOG"
+ - "SCHEMA"
+ - "TABLE"
+ - "FILESET"
+ - "TOPIC"
+ - "ROLE"
+ - "METALAKE"
metadataObjectFullName:
name: metadataObjectFullName
diff --git a/docs/open-api/tags.yaml b/docs/open-api/tags.yaml
index 61d9ae1fc..54db400b4 100644
--- a/docs/open-api/tags.yaml
+++ b/docs/open-api/tags.yaml
@@ -284,6 +284,41 @@ paths:
"5xx":
$ref: "./openapi.yaml#/components/responses/ServerErrorResponse"
+
+ /metalakes/{metalake}/tags/{tag}/objects:
+parameters:
+ - $ref: "./openapi.yaml#/components/parameters/metalake"
+ - $ref: "./openapi.yaml#/components/parameters/tag"
+get:
+ tags:
+- tag
+ summary: list metadata objects for tag
+ operationId: listTagObjects
+ responses:
+"200":
+ description: Returns the list of metadata objects associated with
specified tag
+ content:
+application/vnd.gravitino.v1+json:
+ schema:
+$ref: "#/components/responses/MetadataObjectListResponse"
+ examples:
+TagResponse:
+ $ref: "#/components/examples/MetadataObjectListResponse"
+"400":
+ $ref: "./openapi.yaml#/components/responses/BadRequestErrorResponse"
+"404":
+ description: Not Found - The specified tag does not exist
+ content:
+application/vnd.gravitino.v1+json:
+ schema:
+$ref: "./openapi.yaml#/components/schemas/ErrorModel"
+ examples:
+NoSuchTagException:
+ $ref: "#/components/examples/NoSuchTagException"
+"5xx":
+ $ref: "./openapi.yaml#/components/responses/ServerErrorResponse"
+
+
components:
parameters:
details:
@@ -324,6 +359,25 @@ components:
description: Whether the tag is inherited from the parent metadata
object
nullable: true
+MetadataObject:
+ type: object
+ description: A metadata object
+ properties:
+fullName:
+ type: string
+ description: The name of the metadata object, separated by dots
+type:
+ type: string
+ description: The type of the metadata object
+ enum:
+- "CATALOG"
+- "SCHEMA"
+- "TABLE"
+- "FILESET"
+- "TOPIC"
+- "ROLE"
+- "METALAKE"
+
re
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
xunliu commented on code in PR #4515:
URL: https://github.com/apache/gravitino/pull/4515#discussion_r1721863651
##
authorizations/authorization-ranger/src/main/java/org/apache/gravitino/authorization/ranger/RangerAuthorizationPlugin.java:
##
@@ -0,0 +1,1015 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.authorization.ranger;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.errorprone.annotations.FormatMethod;
+import com.google.errorprone.annotations.FormatString;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.stream.Collectors;
+import org.apache.gravitino.MetadataObject;
+import org.apache.gravitino.authorization.Group;
+import org.apache.gravitino.authorization.Owner;
+import org.apache.gravitino.authorization.Privilege;
+import org.apache.gravitino.authorization.Role;
+import org.apache.gravitino.authorization.RoleChange;
+import org.apache.gravitino.authorization.SecurableObject;
+import org.apache.gravitino.authorization.SecurableObjects;
+import org.apache.gravitino.authorization.User;
+import org.apache.gravitino.authorization.ranger.defines.VXGroup;
+import org.apache.gravitino.authorization.ranger.defines.VXGroupList;
+import org.apache.gravitino.authorization.ranger.defines.VXUser;
+import org.apache.gravitino.authorization.ranger.defines.VXUserList;
+import org.apache.gravitino.connector.AuthorizationPropertiesMeta;
+import org.apache.gravitino.connector.authorization.AuthorizationPlugin;
+import org.apache.gravitino.exceptions.AuthorizationPluginException;
+import org.apache.ranger.RangerServiceException;
+import org.apache.ranger.plugin.model.RangerPolicy;
+import org.apache.ranger.plugin.model.RangerRole;
+import org.apache.ranger.plugin.util.GrantRevokeRoleRequest;
+import org.apache.ranger.plugin.util.SearchFilter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Ranger authorization operations plugin abstract class. */
+public abstract class RangerAuthorizationPlugin implements AuthorizationPlugin
{
+ private static final Logger LOG =
LoggerFactory.getLogger(RangerAuthorizationPlugin.class);
+
+ protected String catalogProvider;
+ protected RangerClientExt rangerClient;
+ protected String rangerServiceName;
+ /** Mapping Gravitino privilege name to the underlying authorization system
privileges. */
+ protected Map> mapPrivileges = null;
+ // The owner privileges, the owner can do anything on the metadata object
+ protected Set ownerPrivileges = null;
+
+ /**
+ * Because Ranger doesn't support the precise filter, Ranger will return the
policy meets the
+ * wildcard(*,?) conditions, just like `*.*.*` policy will match
`db1.table1.column1` So we need
+ * to manual precise filter the policies.
+ */
+ // Search Ranger policy filter keys
+ protected List policyFilterKeys = null;
+ // Search Ranger policy precise filter keys
+ protected List policyPreciseFilterKeys = null;
+
+ public static final String MANAGED_BY_GRAVITINO = "MANAGED_BY_GRAVITINO";
+
+ // TODO: Maybe need to move to the configuration in the future
+ public static final String RANGER_ADMIN_NAME = "admin";
+
+ public RangerAuthorizationPlugin(String catalogProvider, Map
config) {
+super();
+this.catalogProvider = catalogProvider;
+String rangerUrl =
config.get(AuthorizationPropertiesMeta.RANGER_ADMIN_URL);
+String authType = config.get(AuthorizationPropertiesMeta.RANGER_AUTH_TYPE);
+String username = config.get(AuthorizationPropertiesMeta.RANGER_USERNAME);
+// Apache Ranger Password should be minimum 8 characters with min one
alphabet and one numeric.
+String password = config.get(AuthorizationPropertiesMeta.RANGER_PASSWORD);
+rangerServiceName =
config.get(AuthorizationPropertiesMeta.RANGER_SERVICE_NAME);
+check(rangerUrl != null, "Ranger admin URL is requi
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
xunliu commented on code in PR #4515:
URL: https://github.com/apache/gravitino/pull/4515#discussion_r1721883659
##
authorizations/authorization-ranger/src/main/java/org/apache/gravitino/authorization/ranger/RangerHiveAuthorizationPlugin.java:
##
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.authorization.ranger;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Set;
+import org.apache.gravitino.authorization.Privilege;
+
+/**
+ * RangerHiveAuthorizationPlugin is a plugin for Apache Ranger to manage the
Hive authorization of
+ * the Apache Gravitino.
+ */
+public class RangerHiveAuthorizationPlugin extends RangerAuthorizationPlugin {
+ public RangerHiveAuthorizationPlugin(String catalogProvider, Map config) {
+super(catalogProvider, config);
+ }
+
+ /**
+ * Ranger hive's privilege have `select`, `update`, `create`, `drop`,
`alter`, `index`, `lock`,
+ * `read`, `write`, `repladmin`, `serviceadmin`, `refresh` and `all`.
Reference:
+ *
ranger/agents-common/src/main/resources/service-defs/ranger-servicedef-hive.json
+ */
+ @Override
+ protected void initMapPrivileges() {
+mapPrivileges =
+ImmutableMap.>builder()
+.put(
+Privilege.Name.CREATE_SCHEMA,
+ImmutableSet.of(RangerDefines.ACCESS_TYPE_HIVE_SELECT))
+.put(
+Privilege.Name.CREATE_TABLE,
ImmutableSet.of(RangerDefines.ACCESS_TYPE_HIVE_CREATE))
Review Comment:
These are two scenarios here:
1. If you have SecurableObjects('catalog.schema.*', CREATE_TABLE), this
permission mean is you can create any table under the `catalog.schema`.
2. If you have SecurableObjects('catalog.schema.tab1', CREATE_TABLE), this
permission mean is you can create table `tab1` on the `catalog.schema`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
xunliu commented on code in PR #4515:
URL: https://github.com/apache/gravitino/pull/4515#discussion_r1721883659
##
authorizations/authorization-ranger/src/main/java/org/apache/gravitino/authorization/ranger/RangerHiveAuthorizationPlugin.java:
##
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.authorization.ranger;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Set;
+import org.apache.gravitino.authorization.Privilege;
+
+/**
+ * RangerHiveAuthorizationPlugin is a plugin for Apache Ranger to manage the
Hive authorization of
+ * the Apache Gravitino.
+ */
+public class RangerHiveAuthorizationPlugin extends RangerAuthorizationPlugin {
+ public RangerHiveAuthorizationPlugin(String catalogProvider, Map config) {
+super(catalogProvider, config);
+ }
+
+ /**
+ * Ranger hive's privilege have `select`, `update`, `create`, `drop`,
`alter`, `index`, `lock`,
+ * `read`, `write`, `repladmin`, `serviceadmin`, `refresh` and `all`.
Reference:
+ *
ranger/agents-common/src/main/resources/service-defs/ranger-servicedef-hive.json
+ */
+ @Override
+ protected void initMapPrivileges() {
+mapPrivileges =
+ImmutableMap.>builder()
+.put(
+Privilege.Name.CREATE_SCHEMA,
+ImmutableSet.of(RangerDefines.ACCESS_TYPE_HIVE_SELECT))
+.put(
+Privilege.Name.CREATE_TABLE,
ImmutableSet.of(RangerDefines.ACCESS_TYPE_HIVE_CREATE))
Review Comment:
These are two scenarios here:
1. If you have SecurableObjects('catalog.schema.*', CREATE_TABLE), this
permission mean is you can create any table under the `catalog.schema`.
2. If you have SecurableObjects('catalog.schema.tab1', CREATE_TABLE), this
permission mean is you can create table `tab1` under the `catalog.schema`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#4417] feat(build): Add release script to automate the release process [gravitino]
jerryshao commented on PR #4518: URL: https://github.com/apache/gravitino/pull/4518#issuecomment-2296768095 @yuqi1129 @jerqi please help to review, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4583] improvement(CI): reduce Backend Integration Test jobs [gravitino]
jerryshao commented on PR #4589: URL: https://github.com/apache/gravitino/pull/4589#issuecomment-2296776509 I think we can also add some component level fine-grained control. For example, if the code change only happens in one independent component, then we can only run the related components test, what do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4277] feat(client/server): Add the `getFileLocation` interface in the Java Client/Server [gravitino]
jerryshao commented on PR #4281: URL: https://github.com/apache/gravitino/pull/4281#issuecomment-2296783231 @xloya is it ready for review? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
xunliu commented on code in PR #4515:
URL: https://github.com/apache/gravitino/pull/4515#discussion_r1721845748
##
authorizations/authorization-ranger/src/main/java/org/apache/gravitino/authorization/ranger/RangerAuthorizationPlugin.java:
##
@@ -0,0 +1,1015 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.authorization.ranger;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.errorprone.annotations.FormatMethod;
+import com.google.errorprone.annotations.FormatString;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.stream.Collectors;
+import org.apache.gravitino.MetadataObject;
+import org.apache.gravitino.authorization.Group;
+import org.apache.gravitino.authorization.Owner;
+import org.apache.gravitino.authorization.Privilege;
+import org.apache.gravitino.authorization.Role;
+import org.apache.gravitino.authorization.RoleChange;
+import org.apache.gravitino.authorization.SecurableObject;
+import org.apache.gravitino.authorization.SecurableObjects;
+import org.apache.gravitino.authorization.User;
+import org.apache.gravitino.authorization.ranger.defines.VXGroup;
+import org.apache.gravitino.authorization.ranger.defines.VXGroupList;
+import org.apache.gravitino.authorization.ranger.defines.VXUser;
+import org.apache.gravitino.authorization.ranger.defines.VXUserList;
+import org.apache.gravitino.connector.AuthorizationPropertiesMeta;
+import org.apache.gravitino.connector.authorization.AuthorizationPlugin;
+import org.apache.gravitino.exceptions.AuthorizationPluginException;
+import org.apache.ranger.RangerServiceException;
+import org.apache.ranger.plugin.model.RangerPolicy;
+import org.apache.ranger.plugin.model.RangerRole;
+import org.apache.ranger.plugin.util.GrantRevokeRoleRequest;
+import org.apache.ranger.plugin.util.SearchFilter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Ranger authorization operations plugin abstract class. */
+public abstract class RangerAuthorizationPlugin implements AuthorizationPlugin
{
+ private static final Logger LOG =
LoggerFactory.getLogger(RangerAuthorizationPlugin.class);
+
+ protected String catalogProvider;
+ protected RangerClientExt rangerClient;
+ protected String rangerServiceName;
+ /** Mapping Gravitino privilege name to the underlying authorization system
privileges. */
+ protected Map> mapPrivileges = null;
+ // The owner privileges, the owner can do anything on the metadata object
+ protected Set ownerPrivileges = null;
+
+ /**
+ * Because Ranger doesn't support the precise filter, Ranger will return the
policy meets the
+ * wildcard(*,?) conditions, just like `*.*.*` policy will match
`db1.table1.column1` So we need
+ * to manual precise filter the policies.
+ */
+ // Search Ranger policy filter keys
+ protected List policyFilterKeys = null;
+ // Search Ranger policy precise filter keys
+ protected List policyPreciseFilterKeys = null;
+
+ public static final String MANAGED_BY_GRAVITINO = "MANAGED_BY_GRAVITINO";
+
+ // TODO: Maybe need to move to the configuration in the future
+ public static final String RANGER_ADMIN_NAME = "admin";
+
+ public RangerAuthorizationPlugin(String catalogProvider, Map
config) {
+super();
+this.catalogProvider = catalogProvider;
+String rangerUrl =
config.get(AuthorizationPropertiesMeta.RANGER_ADMIN_URL);
+String authType = config.get(AuthorizationPropertiesMeta.RANGER_AUTH_TYPE);
+String username = config.get(AuthorizationPropertiesMeta.RANGER_USERNAME);
+// Apache Ranger Password should be minimum 8 characters with min one
alphabet and one numeric.
+String password = config.get(AuthorizationPropertiesMeta.RANGER_PASSWORD);
+rangerServiceName =
config.get(AuthorizationPropertiesMeta.RANGER_SERVICE_NAME);
+check(rangerUrl != null, "Ranger admin URL is requi
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
xunliu commented on code in PR #4515:
URL: https://github.com/apache/gravitino/pull/4515#discussion_r1721943510
##
authorizations/authorization-ranger/src/main/java/org/apache/gravitino/authorization/ranger/RangerAuthorizationPlugin.java:
##
@@ -0,0 +1,1015 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.authorization.ranger;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.errorprone.annotations.FormatMethod;
+import com.google.errorprone.annotations.FormatString;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.stream.Collectors;
+import org.apache.gravitino.MetadataObject;
+import org.apache.gravitino.authorization.Group;
+import org.apache.gravitino.authorization.Owner;
+import org.apache.gravitino.authorization.Privilege;
+import org.apache.gravitino.authorization.Role;
+import org.apache.gravitino.authorization.RoleChange;
+import org.apache.gravitino.authorization.SecurableObject;
+import org.apache.gravitino.authorization.SecurableObjects;
+import org.apache.gravitino.authorization.User;
+import org.apache.gravitino.authorization.ranger.defines.VXGroup;
+import org.apache.gravitino.authorization.ranger.defines.VXGroupList;
+import org.apache.gravitino.authorization.ranger.defines.VXUser;
+import org.apache.gravitino.authorization.ranger.defines.VXUserList;
+import org.apache.gravitino.connector.AuthorizationPropertiesMeta;
+import org.apache.gravitino.connector.authorization.AuthorizationPlugin;
+import org.apache.gravitino.exceptions.AuthorizationPluginException;
+import org.apache.ranger.RangerServiceException;
+import org.apache.ranger.plugin.model.RangerPolicy;
+import org.apache.ranger.plugin.model.RangerRole;
+import org.apache.ranger.plugin.util.GrantRevokeRoleRequest;
+import org.apache.ranger.plugin.util.SearchFilter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Ranger authorization operations plugin abstract class. */
+public abstract class RangerAuthorizationPlugin implements AuthorizationPlugin
{
+ private static final Logger LOG =
LoggerFactory.getLogger(RangerAuthorizationPlugin.class);
+
+ protected String catalogProvider;
+ protected RangerClientExt rangerClient;
+ protected String rangerServiceName;
+ /** Mapping Gravitino privilege name to the underlying authorization system
privileges. */
+ protected Map> mapPrivileges = null;
+ // The owner privileges, the owner can do anything on the metadata object
+ protected Set ownerPrivileges = null;
+
+ /**
+ * Because Ranger doesn't support the precise filter, Ranger will return the
policy meets the
+ * wildcard(*,?) conditions, just like `*.*.*` policy will match
`db1.table1.column1` So we need
+ * to manual precise filter the policies.
+ */
+ // Search Ranger policy filter keys
+ protected List policyFilterKeys = null;
+ // Search Ranger policy precise filter keys
+ protected List policyPreciseFilterKeys = null;
+
+ public static final String MANAGED_BY_GRAVITINO = "MANAGED_BY_GRAVITINO";
+
+ // TODO: Maybe need to move to the configuration in the future
+ public static final String RANGER_ADMIN_NAME = "admin";
+
+ public RangerAuthorizationPlugin(String catalogProvider, Map
config) {
+super();
+this.catalogProvider = catalogProvider;
+String rangerUrl =
config.get(AuthorizationPropertiesMeta.RANGER_ADMIN_URL);
+String authType = config.get(AuthorizationPropertiesMeta.RANGER_AUTH_TYPE);
+String username = config.get(AuthorizationPropertiesMeta.RANGER_USERNAME);
+// Apache Ranger Password should be minimum 8 characters with min one
alphabet and one numeric.
+String password = config.get(AuthorizationPropertiesMeta.RANGER_PASSWORD);
+rangerServiceName =
config.get(AuthorizationPropertiesMeta.RANGER_SERVICE_NAME);
+check(rangerUrl != null, "Ranger admin URL is requi
[PR] [Improvement] create and alter failed since the operations in catalog are async [gravitino]
noidname01 opened a new pull request, #4590:
URL: https://github.com/apache/gravitino/pull/4590
### What changes were proposed in this pull request?
NOTE: This is draft PR
Remove reload action and fix the following logic of ITs
### Why are the changes needed?
Fix: #3729
### Does this PR introduce _any_ user-facing change?
* create topic
* alter topic
* create schema
* alter schema
* create table
* alter table
### How was this patch tested?
Modified ITs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] [Improvement] create and alter failed since the operations in catalog are async [gravitino]
noidname01 commented on issue #3729: URL: https://github.com/apache/gravitino/issues/3729#issuecomment-2297415717 @mchades Sorry for the inactiveness in this PR. I've created a draft PR, I have done the main logic modification, the remaining to-do works is modifying the logic of ITs, which still use reload logic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug report] "jdbc-url" checkValue only isNotBlank [gravitino]
LiuQhahah opened a new issue, #4591: URL: https://github.com/apache/gravitino/issues/4591 ### Version main branch ### Describe what's wrong When checking the input value of the jdbc-url, the current logic is checked if the value is not blank. Can you add more restrictions for this field? such as check the valid-URI instead of isNotBlank. https://github.com/apache/gravitino/blob/main/catalogs/catalog-jdbc-common/src/main/java/org/apache/gravitino/catalog/jdbc/config/JdbcConfig.java#L36 ### Error message and/or stacktrace https://github.com/user-attachments/assets/7a630672-ff34-4a60-834c-39a066d9ac7c";> ### How to reproduce see pic ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
jerqi commented on code in PR #4515:
URL: https://github.com/apache/gravitino/pull/4515#discussion_r1722592797
##
core/src/main/java/org/apache/gravitino/connector/authorization/UserGroupAuthorizationPlugin.java:
##
@@ -98,46 +98,15 @@ interface UserGroupAuthorizationPlugin {
Boolean onGroupAcquired(Group group) throws RuntimeException;
/**
- * After granting roles to a user from Gravitino, this method is called to
grant roles to the user
- * in the underlying system.
- *
- * @param user The entity of the User.
- * @param roles The entities of the Roles.
- * @return True if the Grant was successful, false if the Grant was failed.
- * @throws RuntimeException If granting roles to a user encounters storage
issues.
- */
- Boolean onGrantedRolesToUser(List roles, User user) throws
RuntimeException;
-
- /**
- * After revoking roles from a user from Gravitino, this method is called to
revoke roles from the
- * user in the underlying system.
- *
- * @param user The entity of the User.
- * @param roles The entities of the Roles.
- * @return True if the revoke was successfully removed, false if the revoke
failed.
- * @throws RuntimeException If revoking roles from a user encounters storage
issues.
- */
- Boolean onRevokedRolesFromUser(List roles, User user) throws
RuntimeException;
-
- /**
- * After granting roles to a group from Gravitino, this method is called to
grant roles to the
- * group in the underlying system.
- *
- * @param group The entity of the Group.
- * @param roles The entities of the Roles.
- * @return True if the revoke was successfully removed, False if the revoke
failed.
- * @throws RuntimeException If granting roles to a group encounters storage
issues.
- */
- Boolean onGrantedRolesToGroup(List roles, Group group) throws
RuntimeException;
-
- /**
- * After revoking roles from a group from Gravitino, this method is called
to revoke roles from
- * the group in the underlying system.
+ * After set a Owner to Gravitino, this method is called to set the Owner to
the underlying
+ * system.
*
- * @param group The entity of the Group.
- * @param roles The entities of the Roles.
- * @return True if the revoke was successfully removed, False if the revoke
failed.
- * @throws RuntimeException If revoking roles from a group encounters
storage issues.
+ * @param metadataObject The metadata entity.
+ * @param preOwner The previous owner.
+ * @param newOwner The new owner.
+ * @return True if the set Owner was successfully set, false if the set
Owner failed.
+ * @throws RuntimeException If adding the Group encounters storage issues.
*/
- Boolean onRevokedRolesFromGroup(List roles, Group group) throws
RuntimeException;
+ Boolean onOwnerSet(MetadataObject metadataObject, Owner preOwner, Owner
newOwner)
Review Comment:
Need I raise another pull request?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
jerqi commented on PR #4515: URL: https://github.com/apache/gravitino/pull/4515#issuecomment-2297824047 Do you have e2e tests for this pull request? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
jerqi commented on code in PR #4515: URL: https://github.com/apache/gravitino/pull/4515#discussion_r1722596783 ## authorizations/authorization-ranger/src/main/resources/META-INF/services/org.apache.gravitino.authorization.ranger.RangerAuthorization: ## @@ -0,0 +1,19 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +org.apache.gravitino.authorization.ranger.RangerAuthorization Review Comment: User can't use other authorization plugin implement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
lw-yang commented on code in PR #4515:
URL: https://github.com/apache/gravitino/pull/4515#discussion_r1722591836
##
authorizations/authorization-ranger/src/main/java/org/apache/gravitino/authorization/ranger/RangerAuthorizationPlugin.java:
##
@@ -0,0 +1,1015 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.authorization.ranger;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.errorprone.annotations.FormatMethod;
+import com.google.errorprone.annotations.FormatString;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.stream.Collectors;
+import org.apache.gravitino.MetadataObject;
+import org.apache.gravitino.authorization.Group;
+import org.apache.gravitino.authorization.Owner;
+import org.apache.gravitino.authorization.Privilege;
+import org.apache.gravitino.authorization.Role;
+import org.apache.gravitino.authorization.RoleChange;
+import org.apache.gravitino.authorization.SecurableObject;
+import org.apache.gravitino.authorization.SecurableObjects;
+import org.apache.gravitino.authorization.User;
+import org.apache.gravitino.authorization.ranger.defines.VXGroup;
+import org.apache.gravitino.authorization.ranger.defines.VXGroupList;
+import org.apache.gravitino.authorization.ranger.defines.VXUser;
+import org.apache.gravitino.authorization.ranger.defines.VXUserList;
+import org.apache.gravitino.connector.AuthorizationPropertiesMeta;
+import org.apache.gravitino.connector.authorization.AuthorizationPlugin;
+import org.apache.gravitino.exceptions.AuthorizationPluginException;
+import org.apache.ranger.RangerServiceException;
+import org.apache.ranger.plugin.model.RangerPolicy;
+import org.apache.ranger.plugin.model.RangerRole;
+import org.apache.ranger.plugin.util.GrantRevokeRoleRequest;
+import org.apache.ranger.plugin.util.SearchFilter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Ranger authorization operations plugin abstract class. */
+public abstract class RangerAuthorizationPlugin implements AuthorizationPlugin
{
+ private static final Logger LOG =
LoggerFactory.getLogger(RangerAuthorizationPlugin.class);
+
+ protected String catalogProvider;
+ protected RangerClientExt rangerClient;
+ protected String rangerServiceName;
+ /** Mapping Gravitino privilege name to the underlying authorization system
privileges. */
+ protected Map> mapPrivileges = null;
+ // The owner privileges, the owner can do anything on the metadata object
+ protected Set ownerPrivileges = null;
+
+ /**
+ * Because Ranger doesn't support the precise filter, Ranger will return the
policy meets the
+ * wildcard(*,?) conditions, just like `*.*.*` policy will match
`db1.table1.column1` So we need
+ * to manual precise filter the policies.
+ */
+ // Search Ranger policy filter keys
+ protected List policyFilterKeys = null;
+ // Search Ranger policy precise filter keys
+ protected List policyPreciseFilterKeys = null;
+
+ public static final String MANAGED_BY_GRAVITINO = "MANAGED_BY_GRAVITINO";
+
+ // TODO: Maybe need to move to the configuration in the future
+ public static final String RANGER_ADMIN_NAME = "admin";
+
+ public RangerAuthorizationPlugin(String catalogProvider, Map
config) {
+super();
+this.catalogProvider = catalogProvider;
+String rangerUrl =
config.get(AuthorizationPropertiesMeta.RANGER_ADMIN_URL);
+String authType = config.get(AuthorizationPropertiesMeta.RANGER_AUTH_TYPE);
+String username = config.get(AuthorizationPropertiesMeta.RANGER_USERNAME);
+// Apache Ranger Password should be minimum 8 characters with min one
alphabet and one numeric.
+String password = config.get(AuthorizationPropertiesMeta.RANGER_PASSWORD);
+rangerServiceName =
config.get(AuthorizationPropertiesMeta.RANGER_SERVICE_NAME);
+check(rangerUrl != null, "Ranger admin URL is requ
Re: [PR] [#4417] feat(build): Add release script to automate the release process [gravitino]
jerqi commented on PR #4518: URL: https://github.com/apache/gravitino/pull/4518#issuecomment-2297842903 Should we check the requirement of executing the scripts? For example, we should check whether users have installed git, svn. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4417] feat(build): Add release script to automate the release process [gravitino]
jerryshao commented on PR #4518: URL: https://github.com/apache/gravitino/pull/4518#issuecomment-2297845510 OK, I will add. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4072] improvement(docker-image): Transfer docker hub from datastrato to apache [gravitino]
jerryshao commented on PR #4523: URL: https://github.com/apache/gravitino/pull/4523#issuecomment-2297853476 @jerqi do you have any more feedbacks? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4277] feat(client/server): Add the `getFileLocation` interface in the Java Client/Server [gravitino]
xloya commented on PR #4281: URL: https://github.com/apache/gravitino/pull/4281#issuecomment-2297855158 > @xloya is it ready for review? Yeah, the code have been changed according to the discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4583] improvement(CI): reduce Backend Integration Test jobs [gravitino]
mchades commented on PR #4589: URL: https://github.com/apache/gravitino/pull/4589#issuecomment-2297861153 > I think we can also add some component level fine-grained control. For example, if the code change only happens in one independent component, then we can only run the related components test, what do you think? It's a good idea and we already have control in other workflows (such as Spark, Python, etc.) except Trino since the Trino CI has not been separated out yet. (I think this could be done in another PR) Currently, the only tests I can see that can be more finely controlled in the Backend Integration Test workflow are those related to catalog and client. This also depends on us first separating Trino tests from the Backend, so I suggest doing this in a separate PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4583] improvement(CI): reduce Backend Integration Test jobs [gravitino]
yuqi1129 commented on PR #4589: URL: https://github.com/apache/gravitino/pull/4589#issuecomment-2297888908 I suggest we merge several JDK versions and use JDK17 to run the GitHub CI pipeline. If we want to test all JDK versions, I think a nightly test is okay. It's more probable that the storage backend is incorrect than the JDK version. Concerning the embedded and deploy modes, I suggest keeping them as they are and both modes are useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
xunliu commented on PR #4515: URL: https://github.com/apache/gravitino/pull/4515#issuecomment-2297911224 hi @jerqi > Do you have e2e tests for this pull request? Sure, IT in the `RangerHiveIT.java` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#3963] feat(core): Apache Ranger Hive authorization pushdown [gravitino]
xunliu commented on code in PR #4515: URL: https://github.com/apache/gravitino/pull/4515#discussion_r1722651057 ## authorizations/authorization-ranger/src/main/resources/META-INF/services/org.apache.gravitino.authorization.ranger.RangerAuthorization: ## @@ -0,0 +1,19 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +org.apache.gravitino.authorization.ranger.RangerAuthorization Review Comment: OK, I fixed this problem -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
