Re: [PR] [#5985] improvement(CLI): Fix role command that supports handling multiple values [gravitino]
Abyss-lord commented on PR #5988: URL: https://github.com/apache/gravitino/pull/5988#issuecomment-2562291323 Hi @justinmclean , I’ve finished updating the code. Please take a look at the PR again when you have time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5585] improvement(bundles): Refactor bundle jars and provide core jars that does not contains hadoop-{aws,gcp,aliyun,azure} [gravitino]
jerryshao commented on code in PR #5806:
URL: https://github.com/apache/gravitino/pull/5806#discussion_r1897794542
##
docs/hadoop-catalog.md:
##
@@ -10,10 +10,8 @@ license: "This software is licensed under the Apache License
version 2."
Hadoop catalog is a fileset catalog that using Hadoop Compatible File System
(HCFS) to manage
the storage location of the fileset. Currently, it supports local filesystem
and HDFS. For
-object storage like S3, GCS, and Azure Blob Storage, you can put the hadoop
object store jar like
-hadoop-aws into the `$GRAVITINO_HOME/catalogs/hadoop/libs` directory to enable
the support.
-Gravitino itself hasn't yet tested the object storage support, so if you have
any issue,
-please create an [issue](https://github.com/apache/gravitino/issues).
+object storage like S3, GCS, Azure Blob Storage and OSS, you can put the
hadoop object store jar like
+`gravitino-aws-hadoop-bundle-{gravitino-version}.jar` into the
`$GRAVITINO_HOME/catalogs/hadoop/libs` directory to enable the support.
Review Comment:
Why do we need to use the hadoop bundle jar? I assume we already have the
Hadoop dependencies in the Hadoop catalog, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5585] improvement(bundles): Refactor bundle jars and provide core jars that does not contains hadoop-{aws,gcp,aliyun,azure} [gravitino]
yuqi1129 commented on code in PR #5806:
URL: https://github.com/apache/gravitino/pull/5806#discussion_r1897797038
##
docs/hadoop-catalog.md:
##
@@ -10,10 +10,8 @@ license: "This software is licensed under the Apache License
version 2."
Hadoop catalog is a fileset catalog that using Hadoop Compatible File System
(HCFS) to manage
the storage location of the fileset. Currently, it supports local filesystem
and HDFS. For
-object storage like S3, GCS, and Azure Blob Storage, you can put the hadoop
object store jar like
-hadoop-aws into the `$GRAVITINO_HOME/catalogs/hadoop/libs` directory to enable
the support.
-Gravitino itself hasn't yet tested the object storage support, so if you have
any issue,
-please create an [issue](https://github.com/apache/gravitino/issues).
+object storage like S3, GCS, Azure Blob Storage and OSS, you can put the
hadoop object store jar like
+`gravitino-aws-hadoop-bundle-{gravitino-version}.jar` into the
`$GRAVITINO_HOME/catalogs/hadoop/libs` directory to enable the support.
Review Comment:
hadoop bundle does not contain `hadoop-aws` though Gravitino severs already
provides a Hadoop environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5993][FOLLOWUP] Make some methods public [gravitino]
xunliu merged PR #6002: URL: https://github.com/apache/gravitino/pull/6002 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [#5872] feat(client): Add model management Java client API [gravitino]
jerryshao opened a new pull request, #6003: URL: https://github.com/apache/gravitino/pull/6003 ### What changes were proposed in this pull request? This PR adds the Java client API for model management. ### Why are the changes needed? This is a part of work of model management. Fix: #5872 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? UT added. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5585] improvement(bundles): Refactor bundle jars and provide core jars that does not contains hadoop-{aws,gcp,aliyun,azure} [gravitino]
yuqi1129 commented on code in PR #5806:
URL: https://github.com/apache/gravitino/pull/5806#discussion_r1897797038
##
docs/hadoop-catalog.md:
##
@@ -10,10 +10,8 @@ license: "This software is licensed under the Apache License
version 2."
Hadoop catalog is a fileset catalog that using Hadoop Compatible File System
(HCFS) to manage
the storage location of the fileset. Currently, it supports local filesystem
and HDFS. For
-object storage like S3, GCS, and Azure Blob Storage, you can put the hadoop
object store jar like
-hadoop-aws into the `$GRAVITINO_HOME/catalogs/hadoop/libs` directory to enable
the support.
-Gravitino itself hasn't yet tested the object storage support, so if you have
any issue,
-please create an [issue](https://github.com/apache/gravitino/issues).
+object storage like S3, GCS, Azure Blob Storage and OSS, you can put the
hadoop object store jar like
+`gravitino-aws-hadoop-bundle-{gravitino-version}.jar` into the
`$GRAVITINO_HOME/catalogs/hadoop/libs` directory to enable the support.
Review Comment:
hadoop bundle does not contain `hadoop-aws` though Gravitino severs already
provides a Hadoop environment(hadoop-client-api & hadoop-client-api-runtime).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5585] improvement(bundles): Refactor bundle jars and provide core jars that does not contains hadoop-{aws,gcp,aliyun,azure} [gravitino]
jerryshao commented on code in PR #5806:
URL: https://github.com/apache/gravitino/pull/5806#discussion_r1897798539
##
docs/hadoop-catalog.md:
##
@@ -52,7 +50,7 @@ Apart from the above properties, to access fileset like HDFS,
S3, GCS, OSS or cu
| `s3-access-key-id`| The access key of the AWS S3.
| (none) | Yes if it's a S3 fileset. |
0.7.0-incubating |
| `s3-secret-access-key`| The secret key of the AWS S3.
| (none) | Yes if it's a S3 fileset. |
0.7.0-incubating |
-At the same time, you need to place the corresponding bundle jar
[`gravitino-aws-bundle-${version}.jar`](https://repo1.maven.org/maven2/org/apache/gravitino/aws-bundle/)
in the directory `${GRAVITINO_HOME}/catalogs/hadoop/libs`.
+At the same time, you need to place the corresponding bundle jar
[`gravitino-aws-hadoop-bundle-${version}.jar`](https://repo1.maven.org/maven2/org/apache/gravitino/aws-hadoop-bundle/)
in the directory `${GRAVITINO_HOME}/catalogs/hadoop/libs`.
Review Comment:
Also here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5585] improvement(bundles): Refactor bundle jars and provide core jars that does not contains hadoop-{aws,gcp,aliyun,azure} [gravitino]
jerryshao commented on code in PR #5806:
URL: https://github.com/apache/gravitino/pull/5806#discussion_r1897799368
##
docs/how-to-use-gvfs.md:
##
@@ -77,16 +77,15 @@ Apart from the above properties, to access fileset like S3,
GCS, OSS and custom
| `s3-access-key-id` | The access key of the AWS S3.
| (none)|
Yes if it's a S3 fileset.| 0.7.0-incubating |
| `s3-secret-access-key` | The secret key of the AWS S3.
| (none)|
Yes if it's a S3 fileset.| 0.7.0-incubating |
-At the same time, you need to place the corresponding bundle jar
[`gravitino-aws-bundle-${version}.jar`](https://repo1.maven.org/maven2/org/apache/gravitino/aws-bundle/)
in the Hadoop environment(typically located in
`${HADOOP_HOME}/share/hadoop/common/lib/`).
-
+At the same time, you need to place the corresponding bundle jar
[`gravitino-aws-hadooop-bundle-${version}.jar`](https://repo1.maven.org/maven2/org/apache/gravitino/gravitino-aws-hadoop-bundle/)
in the Hadoop environment(typically located in
`${HADOOP_HOME}/share/hadoop/common/lib/`).
Review Comment:
Why using hadoop bundle, not bundle only?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5585] improvement(bundles): Refactor bundle jars and provide core jars that does not contains hadoop-{aws,gcp,aliyun,azure} [gravitino]
yuqi1129 commented on code in PR #5806:
URL: https://github.com/apache/gravitino/pull/5806#discussion_r1897801797
##
docs/hadoop-catalog.md:
##
@@ -10,10 +10,8 @@ license: "This software is licensed under the Apache License
version 2."
Hadoop catalog is a fileset catalog that using Hadoop Compatible File System
(HCFS) to manage
the storage location of the fileset. Currently, it supports local filesystem
and HDFS. For
-object storage like S3, GCS, and Azure Blob Storage, you can put the hadoop
object store jar like
-hadoop-aws into the `$GRAVITINO_HOME/catalogs/hadoop/libs` directory to enable
the support.
-Gravitino itself hasn't yet tested the object storage support, so if you have
any issue,
-please create an [issue](https://github.com/apache/gravitino/issues).
+object storage like S3, GCS, Azure Blob Storage and OSS, you can put the
hadoop object store jar like
+`gravitino-aws-hadoop-bundle-{gravitino-version}.jar` into the
`$GRAVITINO_HOME/catalogs/hadoop/libs` directory to enable the support.
Review Comment:
aws-hadoop-bundle = aws-bundle + hadoop-common + hadoop-aws.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5585] improvement(bundles): Refactor bundle jars and provide core jars that does not contains hadoop-{aws,gcp,aliyun,azure} [gravitino]
yuqi1129 commented on code in PR #5806:
URL: https://github.com/apache/gravitino/pull/5806#discussion_r1897801797
##
docs/hadoop-catalog.md:
##
@@ -10,10 +10,8 @@ license: "This software is licensed under the Apache License
version 2."
Hadoop catalog is a fileset catalog that using Hadoop Compatible File System
(HCFS) to manage
the storage location of the fileset. Currently, it supports local filesystem
and HDFS. For
-object storage like S3, GCS, and Azure Blob Storage, you can put the hadoop
object store jar like
-hadoop-aws into the `$GRAVITINO_HOME/catalogs/hadoop/libs` directory to enable
the support.
-Gravitino itself hasn't yet tested the object storage support, so if you have
any issue,
-please create an [issue](https://github.com/apache/gravitino/issues).
+object storage like S3, GCS, Azure Blob Storage and OSS, you can put the
hadoop object store jar like
+`gravitino-aws-hadoop-bundle-{gravitino-version}.jar` into the
`$GRAVITINO_HOME/catalogs/hadoop/libs` directory to enable the support.
Review Comment:
aws-hadoop-bundle = aws-bundle + hadoop-client-* + hadoop-aws.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5989] The credential type of ADLSTokenCredentialProvider is not equal to the credential type of ADLSTokenCredential [gravitino]
tengqm commented on PR #5990: URL: https://github.com/apache/gravitino/pull/5990#issuecomment-2562473280 Nice cleanup. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5987] improvement: Add more configurations for the config API [gravitino]
jerqi commented on PR #5999: URL: https://github.com/apache/gravitino/pull/5999#issuecomment-2562474621 @tengqm Do you have other suggestion? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [FEATURE] Support LDAP Authentication [gravitino]
jerqi commented on issue #6006: URL: https://github.com/apache/gravitino/issues/6006#issuecomment-2562539102 It's great to support LDAP authentication. Do you have an interest to take over this work? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4398] feat(core): support credential cache for Gravitino server [gravitino]
FANNG1 commented on code in PR #5995:
URL: https://github.com/apache/gravitino/pull/5995#discussion_r1897814886
##
catalogs/catalog-common/src/main/java/org/apache/gravitino/credential/CredentialConstants.java:
##
@@ -22,6 +22,8 @@
public class CredentialConstants {
public static final String CREDENTIAL_PROVIDER_TYPE =
"credential-provider-type";
public static final String CREDENTIAL_PROVIDERS = "credential-providers";
+ public static final String CREDENTIAL_CACHE_EXPIRE_IN_SECS =
"credential-cache-expire-in-secs";
Review Comment:
I don't think we should refresh tokens periodic, the ideal implement may
prefetch credentials according to the access rate of the credential, but this
make cache complicated, I prefer to optimize it latter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#4398] feat(core): support credential cache for Gravitino server [gravitino]
orenccl commented on code in PR #5995:
URL: https://github.com/apache/gravitino/pull/5995#discussion_r1897808881
##
core/src/main/java/org/apache/gravitino/credential/config/CredentialConfig.java:
##
@@ -21,22 +21,68 @@
import com.google.common.collect.ImmutableMap;
import java.util.Map;
+import org.apache.gravitino.Config;
+import org.apache.gravitino.config.ConfigBuilder;
+import org.apache.gravitino.config.ConfigConstants;
+import org.apache.gravitino.config.ConfigEntry;
import org.apache.gravitino.connector.PropertyEntry;
import org.apache.gravitino.credential.CredentialConstants;
-public class CredentialConfig {
+public class CredentialConfig extends Config {
+
+ private static final long DEFAULT_CREDENTIAL_CACHE_MAX_SIZE = 10_000L;
+ private static final long DEFAULT_CREDENTIAL_CACHE_EXPIRE_IN_SECS = 300;
public static final Map>
CREDENTIAL_PROPERTY_ENTRIES =
new ImmutableMap.Builder>()
.put(
CredentialConstants.CREDENTIAL_PROVIDERS,
- PropertyEntry.booleanPropertyEntry(
+ PropertyEntry.stringPropertyEntry(
CredentialConstants.CREDENTIAL_PROVIDERS,
"Credential providers for the Gravitino catalog, schema,
fileset, table, etc.",
false /* required */,
false /* immutable */,
null /* default value */,
false /* hidden */,
false /* reserved */))
+ .put(
+ CredentialConstants.CREDENTIAL_CACHE_EXPIRE_IN_SECS,
+ PropertyEntry.longPropertyEntry(
+ CredentialConstants.CREDENTIAL_CACHE_EXPIRE_IN_SECS,
+ "Max cache time for the credential.",
+ false /* required */,
+ false /* immutable */,
+ DEFAULT_CREDENTIAL_CACHE_EXPIRE_IN_SECS /* default value */,
+ false /* hidden */,
+ false /* reserved */))
+ .put(
+ CredentialConstants.CREDENTIAL_CACHE_MAX_SIZE,
+ PropertyEntry.longPropertyEntry(
+ CredentialConstants.CREDENTIAL_CACHE_MAX_SIZE,
+ "Max size for the credential cache.",
+ false /* required */,
+ false /* immutable */,
+ DEFAULT_CREDENTIAL_CACHE_MAX_SIZE /* default value */,
+ false /* hidden */,
+ false /* reserved */))
.build();
+
+ public static final ConfigEntry CREDENTIAL_CACHE_EXPIRE_IN_SECS =
+ new ConfigBuilder(CredentialConstants.CREDENTIAL_CACHE_EXPIRE_IN_SECS)
+ .doc("Max expire time for the credential cache.")
+ .version(ConfigConstants.VERSION_0_8_0)
+ .longConf()
+ .createWithDefault(DEFAULT_CREDENTIAL_CACHE_EXPIRE_IN_SECS);
+
+ public static final ConfigEntry CREDENTIAL_CACHE_MAZ_SIZE =
Review Comment:
Should be MAX?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] [Improvement] add OSS support for IcebergRESTService docker image [gravitino]
FANNG1 commented on issue #5779: URL: https://github.com/apache/gravitino/issues/5779#issuecomment-2562693546 sure, please go ahead -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [#6012] feat (gvfs-fuse): Support a Gravitino S3 fileset filesystem operation in gvfs fuse [gravitino]
diqiu50 opened a new pull request, #6013: URL: https://github.com/apache/gravitino/pull/6013 ### What changes were proposed in this pull request? Support a Gravitino S3 fileset filesystem operation in gvfs fuse, implemented by OpenDal ### Why are the changes needed? Fix: #6012 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? NO -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug report] The character “/” is illegal character, but it is not banned in fileset create and update API. [gravitino]
danhuawang opened a new issue, #6007: URL: https://github.com/apache/gravitino/issues/6007 ### Version main branch ### Describe what's wrong Currently, fileset name can be created or renamed to "ok/test2" . But “/” is illegal character in fileset name . ### Error message and/or stacktrace N/A ### How to reproduce https://github.com/user-attachments/assets/cdf5d938-56af-468e-bcae-38ffefeadf51"; /> ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [FEATURE] Support LDAP Authentication [gravitino]
nqvuong1998 commented on issue #6006: URL: https://github.com/apache/gravitino/issues/6006#issuecomment-2562413422 cc @yuqi1129 @FANNG1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4398] feat(core): support credential cache for Gravitino server [gravitino]
jerryshao commented on code in PR #5995:
URL: https://github.com/apache/gravitino/pull/5995#discussion_r1897804768
##
catalogs/catalog-common/src/main/java/org/apache/gravitino/credential/CredentialConstants.java:
##
@@ -22,6 +22,8 @@
public class CredentialConstants {
public static final String CREDENTIAL_PROVIDER_TYPE =
"credential-provider-type";
public static final String CREDENTIAL_PROVIDERS = "credential-providers";
+ public static final String CREDENTIAL_CACHE_EXPIRE_IN_SECS =
"credential-cache-expire-in-secs";
Review Comment:
I feel that this property may not so useful, you can refer to Spark's token
refresh config, set a ratio when to refresh the token.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#4398] feat(core): support credential cache for Gravitino server [gravitino]
jerryshao commented on code in PR #5995:
URL: https://github.com/apache/gravitino/pull/5995#discussion_r1897808033
##
core/src/main/java/org/apache/gravitino/credential/CredentialCache.java:
##
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.credential;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Expiry;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
+import org.apache.gravitino.credential.config.CredentialConfig;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class CredentialCache implements Closeable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CredentialCache.class);
+
+ // Calculates the credential expire time in the cache.
+ static class CredentialExpireTimeCalculator implements Expiry {
+
+private long credentialCacheExpireTimeInMs;
+
+public CredentialExpireTimeCalculator(long
credentialCacheExpireTimeInSecs) {
+ this.credentialCacheExpireTimeInMs = credentialCacheExpireTimeInSecs *
1000;
+}
+
+// Set expire time after add a credential in the cache.
+@Override
+public long expireAfterCreate(T key, Credential credential, long
currentTime) {
+ long credentialExpireTime = credential.expireTimeInMs();
+ long timeToExpire = credentialExpireTime - System.currentTimeMillis();
+ if (timeToExpire <= 0) {
+return 0;
+ }
+
+ long idealExpireTime = Math.min(timeToExpire / 2,
credentialCacheExpireTimeInMs);
+ return TimeUnit.MILLISECONDS.toNanos(idealExpireTime);
+}
+
+// Not change expire time after update credential, this should not happen.
+@Override
+public long expireAfterUpdate(T key, Credential value, long currentTime,
long currentDuration) {
+ return currentDuration;
+}
+
+// Not change expire time after read credential.
+@Override
+public long expireAfterRead(T key, Credential value, long currentTime,
long currentDuration) {
+ return currentDuration;
+}
+ }
+
+ private Cache credentialCache;
+
+ public void initialize(Map catalogProperties) {
+CredentialConfig credentialConfig = new
CredentialConfig(catalogProperties);
+long cache_size =
credentialConfig.get(CredentialConfig.CREDENTIAL_CACHE_MAZ_SIZE);
+long cache_expire_time =
credentialConfig.get(CredentialConfig.CREDENTIAL_CACHE_EXPIRE_IN_SECS);
Review Comment:
Why do you use this naming style?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#4398] feat(core): support credential cache for Gravitino server [gravitino]
FANNG1 commented on code in PR #5995:
URL: https://github.com/apache/gravitino/pull/5995#discussion_r1897883598
##
core/src/test/java/org/apache/gravitino/credential/TestCredentialCacheKey.java:
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.credential;
+
+import java.util.Set;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.testcontainers.shaded.com.google.common.collect.ImmutableSet;
+
+public class TestCredentialCacheKey {
+
+ @Test
+ void testCredentialCacheKey() {
+
+PathBasedCredentialContext context1 =
+new PathBasedCredentialContext("user1", ImmutableSet.of("path1"),
ImmutableSet.of("path2"));
+PathBasedCredentialContext context2 =
+new PathBasedCredentialContext("user2", ImmutableSet.of("path1"),
ImmutableSet.of("path2"));
+PathBasedCredentialContext context3 =
+new PathBasedCredentialContext("user3", ImmutableSet.of("path3"),
ImmutableSet.of("path4"));
Review Comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#4398] feat(core): support credential cache for Gravitino server [gravitino]
FANNG1 commented on code in PR #5995:
URL: https://github.com/apache/gravitino/pull/5995#discussion_r1897884068
##
catalogs/catalog-common/src/main/java/org/apache/gravitino/credential/CredentialConstants.java:
##
@@ -22,6 +22,8 @@
public class CredentialConstants {
public static final String CREDENTIAL_PROVIDER_TYPE =
"credential-provider-type";
public static final String CREDENTIAL_PROVIDERS = "credential-providers";
+ public static final String CREDENTIAL_CACHE_EXPIRE_IN_SECS =
"credential-cache-expire-in-secs";
Review Comment:
use `credential-cache-expire-ratio`
##
core/src/main/java/org/apache/gravitino/credential/CredentialCache.java:
##
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.credential;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Expiry;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
+import org.apache.gravitino.credential.config.CredentialConfig;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class CredentialCache implements Closeable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CredentialCache.class);
+
+ // Calculates the credential expire time in the cache.
+ static class CredentialExpireTimeCalculator implements Expiry {
+
+private long credentialCacheExpireTimeInMs;
+
+public CredentialExpireTimeCalculator(long
credentialCacheExpireTimeInSecs) {
+ this.credentialCacheExpireTimeInMs = credentialCacheExpireTimeInSecs *
1000;
+}
+
+// Set expire time after add a credential in the cache.
+@Override
+public long expireAfterCreate(T key, Credential credential, long
currentTime) {
+ long credentialExpireTime = credential.expireTimeInMs();
+ long timeToExpire = credentialExpireTime - System.currentTimeMillis();
+ if (timeToExpire <= 0) {
+return 0;
+ }
+
+ long idealExpireTime = Math.min(timeToExpire / 2,
credentialCacheExpireTimeInMs);
+ return TimeUnit.MILLISECONDS.toNanos(idealExpireTime);
+}
+
+// Not change expire time after update credential, this should not happen.
+@Override
+public long expireAfterUpdate(T key, Credential value, long currentTime,
long currentDuration) {
+ return currentDuration;
+}
+
+// Not change expire time after read credential.
+@Override
+public long expireAfterRead(T key, Credential value, long currentTime,
long currentDuration) {
+ return currentDuration;
+}
+ }
+
+ private Cache credentialCache;
+
+ public void initialize(Map catalogProperties) {
+CredentialConfig credentialConfig = new
CredentialConfig(catalogProperties);
+long cache_size =
credentialConfig.get(CredentialConfig.CREDENTIAL_CACHE_MAZ_SIZE);
+long cache_expire_time =
credentialConfig.get(CredentialConfig.CREDENTIAL_CACHE_EXPIRE_IN_SECS);
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#4398] feat(core): support credential cache for Gravitino server [gravitino]
FANNG1 commented on PR #5995: URL: https://github.com/apache/gravitino/pull/5995#issuecomment-2562602347 @jerryshao @orenccl @tengqm please help to review again -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug report] Cannot get custom config in custom catalog-backend [gravitino]
liangyouze opened a new issue, #6005: URL: https://github.com/apache/gravitino/issues/6005 ### Version main branch ### Describe what's wrong Cannot get custom config in custom catalog-backend from dynamic-config-provider ### Error message and/or stacktrace N/A ### How to reproduce 1. create an iceberg catalog in gravitino and set a custom property 2. set `catalog-config-provider` to `dynamic-config-provider` for iceberg rest catalog service 3. Create a custom catalog-backend, cannot get the custom property from the properties parameter of the initialize method ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Improvement] MODIFY_TABLE should contain the privilege to select table [gravitino]
jerqi commented on issue #6011: URL: https://github.com/apache/gravitino/issues/6011#issuecomment-2562672540 cc @xunliu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Subtask] Implement an in-memory file system for testing and validating the FUSE framework. [gravitino]
diqiu50 closed issue #5886: [Subtask] Implement an in-memory file system for testing and validating the FUSE framework. URL: https://github.com/apache/gravitino/issues/5886 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Subtask] Implement a common filesystem layer to simplify real filesystem implementations [gravitino]
diqiu50 closed issue #5877: [Subtask] Implement a common filesystem layer to simplify real filesystem implementations URL: https://github.com/apache/gravitino/issues/5877 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Improvement] add OSS support for IcebergRESTService docker image [gravitino]
TungYuChiang commented on issue #5779: URL: https://github.com/apache/gravitino/issues/5779#issuecomment-2562672842 May I take this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4398] feat(core): support credential cache for Gravitino server [gravitino]
FANNG1 commented on code in PR #5995:
URL: https://github.com/apache/gravitino/pull/5995#discussion_r1897884176
##
core/src/main/java/org/apache/gravitino/credential/config/CredentialConfig.java:
##
@@ -21,22 +21,68 @@
import com.google.common.collect.ImmutableMap;
import java.util.Map;
+import org.apache.gravitino.Config;
+import org.apache.gravitino.config.ConfigBuilder;
+import org.apache.gravitino.config.ConfigConstants;
+import org.apache.gravitino.config.ConfigEntry;
import org.apache.gravitino.connector.PropertyEntry;
import org.apache.gravitino.credential.CredentialConstants;
-public class CredentialConfig {
+public class CredentialConfig extends Config {
+
+ private static final long DEFAULT_CREDENTIAL_CACHE_MAX_SIZE = 10_000L;
+ private static final long DEFAULT_CREDENTIAL_CACHE_EXPIRE_IN_SECS = 300;
public static final Map>
CREDENTIAL_PROPERTY_ENTRIES =
new ImmutableMap.Builder>()
.put(
CredentialConstants.CREDENTIAL_PROVIDERS,
- PropertyEntry.booleanPropertyEntry(
+ PropertyEntry.stringPropertyEntry(
CredentialConstants.CREDENTIAL_PROVIDERS,
"Credential providers for the Gravitino catalog, schema,
fileset, table, etc.",
false /* required */,
false /* immutable */,
null /* default value */,
false /* hidden */,
false /* reserved */))
+ .put(
+ CredentialConstants.CREDENTIAL_CACHE_EXPIRE_IN_SECS,
+ PropertyEntry.longPropertyEntry(
+ CredentialConstants.CREDENTIAL_CACHE_EXPIRE_IN_SECS,
+ "Max cache time for the credential.",
+ false /* required */,
+ false /* immutable */,
+ DEFAULT_CREDENTIAL_CACHE_EXPIRE_IN_SECS /* default value */,
+ false /* hidden */,
+ false /* reserved */))
+ .put(
+ CredentialConstants.CREDENTIAL_CACHE_MAX_SIZE,
+ PropertyEntry.longPropertyEntry(
+ CredentialConstants.CREDENTIAL_CACHE_MAX_SIZE,
+ "Max size for the credential cache.",
+ false /* required */,
+ false /* immutable */,
+ DEFAULT_CREDENTIAL_CACHE_MAX_SIZE /* default value */,
+ false /* hidden */,
+ false /* reserved */))
.build();
+
+ public static final ConfigEntry CREDENTIAL_CACHE_EXPIRE_IN_SECS =
+ new ConfigBuilder(CredentialConstants.CREDENTIAL_CACHE_EXPIRE_IN_SECS)
+ .doc("Max expire time for the credential cache.")
+ .version(ConfigConstants.VERSION_0_8_0)
+ .longConf()
+ .createWithDefault(DEFAULT_CREDENTIAL_CACHE_EXPIRE_IN_SECS);
+
+ public static final ConfigEntry CREDENTIAL_CACHE_MAZ_SIZE =
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5585] improvement(bundles): Refactor bundle jars and provide core jars that does not contains hadoop-{aws,gcp,aliyun,azure} [gravitino]
yuqi1129 commented on PR #5806: URL: https://github.com/apache/gravitino/pull/5806#issuecomment-2562618054 > Since our bundled jar does not actually bundle anything, I would rethink the jar/module name, take aws as an example: > > * aws: jar without aws and hadoop dependencies. > * aws-bundle: jar with aws and hadoop dependencies. > > One consideration is about the compatibility, if we we rename to aws-hadoop-bundle, then it will not guarantee the compatibility, WDYT? done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [#5889] feat(client-python): Add model management Python API [gravitino]
jerryshao opened a new pull request, #6009: URL: https://github.com/apache/gravitino/pull/6009 ### What changes were proposed in this pull request? This PR proposes to add Python client API for model management. ### Why are the changes needed? This is part of work to support model management in Gravitino. Fix: #5889 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? UT added. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5987] improvement: Add more configurations for the config API [gravitino]
tengqm commented on PR #5999: URL: https://github.com/apache/gravitino/pull/5999#issuecomment-2562632081 > @tengqm Do you have other suggestion? LGTM, although I was thinking of a JSON type configuration file for this purpose. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5987] improvement: Add more configurations for the config API [gravitino]
jerqi commented on PR #5999: URL: https://github.com/apache/gravitino/pull/5999#issuecomment-2562640718 > > @tengqm Do you have other suggestion? > > LGTM, although I was thinking of a JSON type configuration file for this purpose. We can expose more config options in the future. This config option is used for front end display. It's also useful to users if they use API to get the configurations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4398] feat(core): support credential cache for Gravitino server [gravitino]
tengqm commented on PR #5995: URL: https://github.com/apache/gravitino/pull/5995#issuecomment-2562638241 Overall LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5987] improvement: Add more configurations for the config API [gravitino]
mchades merged PR #5999: URL: https://github.com/apache/gravitino/pull/5999 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Improvement] support 'gravitino.authorization.enable' value in '/configs' api [gravitino]
mchades closed issue #5987: [Improvement] support 'gravitino.authorization.enable' value in '/configs' api URL: https://github.com/apache/gravitino/issues/5987 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Improvement] support 'gravitino.authorization.enable' value in '/configs' api [gravitino]
mchades closed issue #5987: [Improvement] support 'gravitino.authorization.enable' value in '/configs' api URL: https://github.com/apache/gravitino/issues/5987 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(gravitino) branch main updated: [#5987] improvement: Add more configurations for the config API (#5999)
This is an automated email from the ASF dual-hosted git repository.
mchades pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new ba8df392c6 [#5987] improvement: Add more configurations for the config
API (#5999)
ba8df392c6 is described below
commit ba8df392c6436b01d3988caad6b5b00b90eef1ec
Author: roryqi
AuthorDate: Thu Dec 26 20:40:57 2024 +0800
[#5987] improvement: Add more configurations for the config API (#5999)
### What changes were proposed in this pull request?
Add more configurations for the config API
### Why are the changes needed?
Fix #5987
Front end can use this config option to optimize the user experience.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
---
.../apache/gravitino/server/web/ConfigServlet.java | 10 ++---
.../gravitino/server/web/TestConfigServlet.java| 46 ++
web/web/src/lib/store/auth/index.js| 2 +-
3 files changed, 51 insertions(+), 7 deletions(-)
diff --git
a/server/src/main/java/org/apache/gravitino/server/web/ConfigServlet.java
b/server/src/main/java/org/apache/gravitino/server/web/ConfigServlet.java
index cdce0a0b12..345a555cd9 100644
--- a/server/src/main/java/org/apache/gravitino/server/web/ConfigServlet.java
+++ b/server/src/main/java/org/apache/gravitino/server/web/ConfigServlet.java
@@ -42,22 +42,20 @@ public class ConfigServlet extends HttpServlet {
ImmutableSet.of(OAuthConfig.DEFAULT_SERVER_URI,
OAuthConfig.DEFAULT_TOKEN_PATH);
private static final ImmutableSet> basicConfigEntries =
- ImmutableSet.of(Configs.AUTHENTICATORS);
+ ImmutableSet.of(Configs.AUTHENTICATORS, Configs.ENABLE_AUTHORIZATION);
- private final Map configs = Maps.newHashMap();
+ private final Map configs = Maps.newHashMap();
public ConfigServlet(ServerConfig serverConfig) {
for (ConfigEntry key : basicConfigEntries) {
- String config = String.valueOf(serverConfig.get(key));
- configs.put(key.getKey(), config);
+ configs.put(key.getKey(), serverConfig.get(key));
}
if (serverConfig
.get(Configs.AUTHENTICATORS)
.contains(AuthenticatorType.OAUTH.name().toLowerCase())) {
for (ConfigEntry key : oauthConfigEntries) {
-String config = String.valueOf(serverConfig.get(key));
-configs.put(key.getKey(), config);
+configs.put(key.getKey(), serverConfig.get(key));
}
}
}
diff --git
a/server/src/test/java/org/apache/gravitino/server/web/TestConfigServlet.java
b/server/src/test/java/org/apache/gravitino/server/web/TestConfigServlet.java
new file mode 100644
index 00..c76587d039
--- /dev/null
+++
b/server/src/test/java/org/apache/gravitino/server/web/TestConfigServlet.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.server.web;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.io.PrintWriter;
+import javax.servlet.http.HttpServletResponse;
+import org.apache.gravitino.server.ServerConfig;
+import org.junit.jupiter.api.Test;
+
+public class TestConfigServlet {
+
+ @Test
+ public void testConfigServlet() throws Exception {
+ServerConfig serverConfig = new ServerConfig();
+ConfigServlet configServlet = new ConfigServlet(serverConfig);
+configServlet.init();
+HttpServletResponse res = mock(HttpServletResponse.class);
+PrintWriter writer = mock(PrintWriter.class);
+when(res.getWriter()).thenReturn(writer);
+configServlet.doGet(null, res);
+verify(writer)
+.write(
+
"{\"gravitino.authorization.enable\":false,\"gravitino.authenticators\":[\"simple\"]}");
+configServlet.destroy();
+ }
+}
diff --git a/web/web/src/lib/store/auth/index.js
b/web/web/src/lib/store/auth/index.js
index 83d58394c9..ec0f1f5212 100644
--- a/web/web/src/lib/store/auth/inde
Re: [PR] [#5902] feat: Add tag failure event to Gravitino server [gravitino]
xunliu commented on PR #5944: URL: https://github.com/apache/gravitino/pull/5944#issuecomment-2562645309 hi @FANNG1 Please help review this PR, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5192] [Feature] Support Catalog Operation DDL for paimon-catalog [gravitino]
FANNG1 commented on PR #5818: URL: https://github.com/apache/gravitino/pull/5818#issuecomment-2562769361 > Is there anything else that needs to be fixed @FANNG1 Sorry, I'm working on the issues to release 0.8, didn't have enough time to review this PR these days. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#3515] feat(flink-connector): Support flink iceberg catalog [gravitino]
FANNG1 commented on code in PR #5914:
URL: https://github.com/apache/gravitino/pull/5914#discussion_r1896596310
##
docs/flink-connector/flink-catalog-iceberg.md:
##
@@ -0,0 +1,83 @@
+---
+title: "Flink connector Iceberg catalog"
+slug: /flink-connector/flink-catalog-iceberg
+keyword: flink connector iceberg catalog
+license: "This software is licensed under the Apache License version 2."
+---
+
+The Apache Gravitino Flink connector offers the capability to read and write
Iceberg tables, with the metadata managed by the Gravitino server. To enable
the use of the Iceberg catalog within the Flink connector, you must download
the Iceberg Flink runtime JAR to the Flink classpath.
+
+## Capabilities
+
+ Supported DML and DDL operations:
+
+- `CREATE CATALOG`
+- `CREATE DATABASE`
+- `CREATE TABLE`
+- `DROP TABLE`
+- `ALTER TABLE`
+- `INSERT INTO & OVERWRITE`
+- `SELECT`
+
+ Not supported operations:
+
+- Partition operations
+- View operations
+- Metadata tables, like:
+ - `{iceberg_catalog}.{iceberg_database}.{iceberg_table}&snapshots`
+- Querying UDF
+ - `UPDATE` clause
+ - `DELETE` clause
+ - `CREATE TABLE LIKE` clause
+
+## SQL example
+```sql
+
+-- Suppose iceberg_a is the Iceberg catalog name managed by Gravitino
+
+USE iceberg_a;
+
+CREATE DATABASE IF NOT EXISTS mydatabase;
+USE mydatabase;
+
+CREATE TABLE sample (
+id BIGINT COMMENT 'unique id',
+data STRING NOT NULL
+) PARTITIONED BY (data)
+WITH ('format-version'='2');
+
+INSERT INTO sample
+VALUES (1, 'A'), (2, 'B');
+
+SELECT * FROM sample WHERE data = 'B';
+
+```
+
+## Catalog properties
+
+Gravitino Flink connector will transform the following property names defined
in catalog properties to Flink Iceberg connector configuration.
+
+| Gravitino catalog property name | Flink Iceberg connector configuration |
Description
| Since Version |
+|-|---|-|---|
+| `catalog-backend` | `catalog-type`|
Catalog backend type
| 0.8.0-incubating |
+| `uri` | `uri` |
Catalog backend URI
| 0.8.0-incubating |
+| `warehouse` | `warehouse` |
Catalog backend warehouse
| 0.8.0-incubating |
+| `io-impl` | `io-impl` |
The IO implementation for `FileIO` in Iceberg.
| 0.8.0-incubating |
+| `s3-endpoint` | `s3.endpoint` | An
alternative endpoint of the S3 service, This could be used for S3FileIO with
any S3-compatible object storage service that has a different endpoint, or
access a private S3 endpoint in a virtual private cloud. | 0.8.0-incubating |
+| `s3-region` | `client.region` |
The region of the S3 service, like `us-west-2`.
| 0.8.0-incubating |
+| `oss-endpoint` | `oss.endpoint`|
The endpoint of Aliyun OSS service.
| 0.8.0-incubating |
+
+
+## Storage
+
+### S3
Review Comment:
Have you tested `S3` `GCS` `OSS` storage?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#3515] feat(flink-connector): Support flink iceberg catalog [gravitino]
FANNG1 commented on PR #5914: URL: https://github.com/apache/gravitino/pull/5914#issuecomment-2562771463 Hi, @sunxiaojian , Sorry for the delay, I'm working on the issues to release 0.8, may doesn't have enough time to review this PR these days. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5585] improvement(bundles): Refactor bundle jars and provide core jars that does not contains hadoop-{aws,gcp,aliyun,azure} [gravitino]
yuqi1129 commented on PR #5806: URL: https://github.com/apache/gravitino/pull/5806#issuecomment-2562460198 > Since our bundled jar does not actually bundle anything. To a certain extent, it did bundle something like credential vending(contains all jars needed to provide credential services), when it comes to hadoop filesystem, `xxx-bundle.jar` contains nothing but interface `FileSystemProvider` needed by Gravitino. > One consideration is about the compatibility, if we we rename to aws-hadoop-bundle, then it will not guarantee the compatibility, WDYT? Yeah, version 0.7 and 0.8 are not compatible on this point. Considering about aspect mentioned before, I also prefer to keep the name `xxx-bundle.jar` as it is and do not introduce `-hadoop-bundle`, @FANNG1 do you have any thoughts about it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5536] Improvement(client-python): reportUndefinedVariable warning in types.py [gravitino]
jerryshao commented on PR #6001: URL: https://github.com/apache/gravitino/pull/6001#issuecomment-2562469499 Is it possible to add a lint check for such kind of Python warning in pyintrc? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Improvement] Architecture picture should be updated [gravitino]
jerqi opened a new issue, #6010: URL: https://github.com/apache/gravitino/issues/6010 ### What would you like to be improved? Architecture picture should be updated https://github.com/apache/gravitino/blob/main/docs/assets/gravitino-model-arch.png First, we should add schema. Second. We should use fileset instead of files. ### How should we improve? Draw a new picture -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Improvement] Architecture picture should be updated [gravitino]
jerqi commented on issue #6010: URL: https://github.com/apache/gravitino/issues/6010#issuecomment-2562658436 cc @jerryshao -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4398] feat(core): support credential cache for Gravitino server [gravitino]
jerryshao commented on code in PR #5995:
URL: https://github.com/apache/gravitino/pull/5995#discussion_r1897902069
##
core/src/main/java/org/apache/gravitino/credential/CredentialCache.java:
##
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.credential;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Expiry;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
+import org.apache.gravitino.credential.config.CredentialConfig;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class CredentialCache implements Closeable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CredentialCache.class);
+
+ // Calculates the credential expire time in the cache.
+ static class CredentialExpireTimeCalculator implements Expiry {
+
+private double credentialCacheExpireRatio;
+
+public CredentialExpireTimeCalculator(double credentialCacheExpireRatio) {
+ this.credentialCacheExpireRatio = credentialCacheExpireRatio;
+}
+
+// Set expire time after add a credential in the cache.
+@Override
+public long expireAfterCreate(T key, Credential credential, long
currentTime) {
Review Comment:
Is the `currentTime` here millisecond or nanosecond?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] [FEATURE] Support table format output for SchemaDetails command [gravitino]
waukin commented on issue #5957: URL: https://github.com/apache/gravitino/issues/5957#issuecomment-2562758155 > You can leave the existing method as `output(catalogs)` and just have that method output "No catalogs exist." if null is passed into it. I’ve kept the existing method as `output(catalogs)`. For the `ListCatalogs` command, if there are no catalogs, `client.listCatalogsInfo()` returns an empty array, not null. I’ve made some refactoring changes in this PR: https://github.com/apache/gravitino/pull/6015. If you have some free time, please take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5901] feat(core): support tag event to Gravitino server [gravitino]
FANNG1 commented on PR #5998: URL: https://github.com/apache/gravitino/pull/5998#issuecomment-2562758704 will review the PR until #5944 is merged for some duplicated logic, please wait for a while. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Subtask] Propose a draft design to add Hbase JDBC Catalog Support [gravitino]
flaming-archer commented on issue #6020: URL: https://github.com/apache/gravitino/issues/6020#issuecomment-2562905575 Here is the design document. Please feel free to leave your comments. https://docs.google.com/document/d/10KFe0ubyBzYoI_Csqw5vstdBJvAoVYZ2T7irk0BeoQE/edit?usp=sharing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Subtask] Propose a draft design to add Hbase JDBC Catalog Support [gravitino]
flaming-archer opened a new issue, #6020: URL: https://github.com/apache/gravitino/issues/6020 ### Describe the subtask Propose a draft design to add Hbase JDBC Catalog Support ### Parent issue https://github.com/apache/gravitino/issues/6019 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Feature] Support Hbase catalog [gravitino]
flaming-archer closed issue #5865: [Feature] Support Hbase catalog URL: https://github.com/apache/gravitino/issues/5865 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Feature] Support Clickhouse catalog [gravitino]
flaming-archer commented on issue #3888: URL: https://github.com/apache/gravitino/issues/3888#issuecomment-2562910741 closed due to https://github.com/apache/gravitino/issues/6017 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Feature] Support Hbase catalog [gravitino]
flaming-archer commented on issue #5865: URL: https://github.com/apache/gravitino/issues/5865#issuecomment-2562910186 closed due to https://github.com/apache/gravitino/issues/6019 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Improvement] Refactor CLI output methods for no data hints [gravitino]
waukin opened a new issue, #6014: URL: https://github.com/apache/gravitino/issues/6014 ### What would you like to be improved? In the `ListMetalakes` and `ListCatalogs` methods, if no data is returned, the CLI should provide a hint, such as "No metalakes exist." Currently, the implementation uses two methods: `System.out.println` and `output`, which feels inconsistent. My goal is to combine the implementation into a single method. ### How should we improve? _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Improvement] Refactor CLI output methods for no data hints [gravitino]
waukin commented on issue #6014: URL: https://github.com/apache/gravitino/issues/6014#issuecomment-2562704506 I am working on this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [#6014] refactor: CLI output methods for no data hints [gravitino]
waukin opened a new pull request, #6015: URL: https://github.com/apache/gravitino/pull/6015 ### What changes were proposed in this pull request? In the `ListMetalakes` and `ListCatalogs` methods, retain the use of `output(metalakes)` and `output(catalogs)`. If metalakes or catalogs are empty arrays, they will be handled by the `output` method in `PlainFormat` and `TableFormat`. ### Why are the changes needed? Issue: https://github.com/apache/gravitino/issues/6014 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4398] feat(core): support credential cache for Gravitino server [gravitino]
FANNG1 commented on code in PR #5995:
URL: https://github.com/apache/gravitino/pull/5995#discussion_r1897933562
##
core/src/main/java/org/apache/gravitino/credential/CredentialCache.java:
##
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.credential;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Expiry;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
+import org.apache.gravitino.credential.config.CredentialConfig;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class CredentialCache implements Closeable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CredentialCache.class);
+
+ // Calculates the credential expire time in the cache.
+ static class CredentialExpireTimeCalculator implements Expiry {
+
+private double credentialCacheExpireRatio;
+
+public CredentialExpireTimeCalculator(double credentialCacheExpireRatio) {
+ this.credentialCacheExpireRatio = credentialCacheExpireRatio;
+}
+
+// Set expire time after add a credential in the cache.
+@Override
+public long expireAfterCreate(T key, Credential credential, long
currentTime) {
Review Comment:
According the API it's nanosecond. `currentTime – the current time, in
nanoseconds`
In test, the value of currentTime is odd, and according to the API, seems we
should calculate the time by ourself.
```
Note: The currentTime is supplied by the configured Ticker and by default
does not relate to system or wall-clock time. When calculating the duration
based on a timestamp, the current time should be obtained independently.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#4398] feat(core): support credential cache for Gravitino server [gravitino]
FANNG1 commented on code in PR #5995:
URL: https://github.com/apache/gravitino/pull/5995#discussion_r1897933562
##
core/src/main/java/org/apache/gravitino/credential/CredentialCache.java:
##
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.credential;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Expiry;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
+import org.apache.gravitino.credential.config.CredentialConfig;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class CredentialCache implements Closeable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CredentialCache.class);
+
+ // Calculates the credential expire time in the cache.
+ static class CredentialExpireTimeCalculator implements Expiry {
+
+private double credentialCacheExpireRatio;
+
+public CredentialExpireTimeCalculator(double credentialCacheExpireRatio) {
+ this.credentialCacheExpireRatio = credentialCacheExpireRatio;
+}
+
+// Set expire time after add a credential in the cache.
+@Override
+public long expireAfterCreate(T key, Credential credential, long
currentTime) {
Review Comment:
According the API it's nanosecond. `currentTime – the current time, in
nanoseconds`
The value of currentTime is odd, and according to the API, seems we should
calculate the time by ourself.
```
Note: The currentTime is supplied by the configured Ticker and by default
does not relate to system or wall-clock time. When calculating the duration
based on a timestamp, the current time should be obtained independently.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5536] Improvement(client-python): reportUndefinedVariable warning in types.py [gravitino]
orenccl commented on PR #6001: URL: https://github.com/apache/gravitino/pull/6001#issuecomment-2562883570 @jerryshao I think currently pylint can not detect forward reference https://peps.python.org/pep-0484/#forward-references -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5585] improvement(bundles): Refactor bundle jars and provide core jars that does not contains hadoop-{aws,gcp,aliyun,azure} [gravitino]
jerryshao commented on PR #5806: URL: https://github.com/apache/gravitino/pull/5806#issuecomment-2562400221 Since our bundled jar does not actually bundle anything, I would rethink the jar/module name, take aws as an example: - aws: jar without aws and hadoop dependencies. - aws-bundle: jar with aws and hadoop dependencies. One consideration is about the compatibility, if we we rename to aws-hadoop-bundle, then it will not guarantee the compatibility, WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5985] improvement(CLI): Fix role command that supports handling multiple values [gravitino]
tengqm commented on code in PR #5988:
URL: https://github.com/apache/gravitino/pull/5988#discussion_r1897848602
##
clients/cli/src/main/java/org/apache/gravitino/cli/GravitinoCommandLine.java:
##
@@ -762,6 +775,12 @@ protected void handleRoleCommand() {
}
}
+ private String getOneRole(String[] roles, String command) {
+Preconditions.checkArgument(
+roles.length == 1, command + " requires only one role, but multiple
are currently passed.");
Review Comment:
I think what Justin suggested was that to check if this actually happens.
If the `--role` option is supposed to be specified only once, would it be
possible for a users to specify that option for multiple times?
If the answer is no, we don't need this check.
It the answer is yes, we have a serious problem regarding command line
option checking. I mean, we have too many cases to check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] [Bug report] IRCS cannot get AKSK via dynamic-config-provider [gravitino]
liangyouze closed issue #6000: [Bug report] IRCS cannot get AKSK via dynamic-config-provider URL: https://github.com/apache/gravitino/issues/6000 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4398] feat(core): support credential cache for Gravitino server [gravitino]
orenccl commented on code in PR #5995:
URL: https://github.com/apache/gravitino/pull/5995#discussion_r1897826816
##
core/src/test/java/org/apache/gravitino/credential/TestCredentialCacheKey.java:
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.credential;
+
+import java.util.Set;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.testcontainers.shaded.com.google.common.collect.ImmutableSet;
+
+public class TestCredentialCacheKey {
+
+ @Test
+ void testCredentialCacheKey() {
+
+PathBasedCredentialContext context1 =
+new PathBasedCredentialContext("user1", ImmutableSet.of("path1"),
ImmutableSet.of("path2"));
+PathBasedCredentialContext context2 =
+new PathBasedCredentialContext("user2", ImmutableSet.of("path1"),
ImmutableSet.of("path2"));
+PathBasedCredentialContext context3 =
+new PathBasedCredentialContext("user3", ImmutableSet.of("path3"),
ImmutableSet.of("path4"));
Review Comment:
Would it be better to set `user1` in context3 to test only different path?
Would it be better to name them `context`, `contextWithDiffUser`, and
`contextWithDiffPath`?
This approach seems much clearer and eliminates the need for comments.
However, I’m fine with the current approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[I] [Improvement] Remove useless `CONTRIBUTING.md` [gravitino]
jerqi opened a new issue, #6008: URL: https://github.com/apache/gravitino/issues/6008 ### What would you like to be improved? In the directory `.github`, there is an extra CONTRIBUTING.md. ### How should we improve? We should remove this file -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Improvement] consider the cross time zone problems for credential [gravitino]
FANNG1 opened a new issue, #6016: URL: https://github.com/apache/gravitino/issues/6016 ### What would you like to be improved? credential use epoch time to represent expire time, could it works well when Gravitino server and client side locates in different time zone regions? ### How should we improve? _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#4398] feat(core): support credential cache for Gravitino server [gravitino]
jerryshao commented on code in PR #5995:
URL: https://github.com/apache/gravitino/pull/5995#discussion_r1897946356
##
core/src/main/java/org/apache/gravitino/credential/CredentialCache.java:
##
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.credential;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Expiry;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
+import org.apache.gravitino.credential.config.CredentialConfig;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class CredentialCache implements Closeable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CredentialCache.class);
+
+ // Calculates the credential expire time in the cache.
+ static class CredentialExpireTimeCalculator implements Expiry {
+
+private double credentialCacheExpireRatio;
+
+public CredentialExpireTimeCalculator(double credentialCacheExpireRatio) {
+ this.credentialCacheExpireRatio = credentialCacheExpireRatio;
+}
+
+// Set expire time after add a credential in the cache.
+@Override
+public long expireAfterCreate(T key, Credential credential, long
currentTime) {
Review Comment:
I see, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [#ISSUE-5993][FOLLOWUP] Make some methods public [gravitino]
jerqi opened a new pull request, #6002: URL: https://github.com/apache/gravitino/pull/6002 ### What changes were proposed in this pull request? Make some methods public, let some classes to reuse some methods. ### Why are the changes needed? This is a follow-up PR. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? No need. Just refactor. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Improvement] Sample command in CLI document may report error [gravitino]
justinmclean commented on issue #5541: URL: https://github.com/apache/gravitino/issues/5541#issuecomment-2562309559 Perhaps, but they can also try out the command and see what it does, and that's how most people learn. The text would be "Some basic information about a metalake, like its name and descriptive comment." However, adding text like this to all commands, doesn't solve this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Subtask] Authentication and authorization support for Iceberg REST server [gravitino]
nqvuong1998 commented on issue #4063: URL: https://github.com/apache/gravitino/issues/4063#issuecomment-2562326301 Hi @FANNG1 , any updates for this task? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Subtask] Authentication and authorization support for Iceberg REST server [gravitino]
nqvuong1998 commented on issue #4063: URL: https://github.com/apache/gravitino/issues/4063#issuecomment-2562329191 The Graviton Iceberg REST service would benefit from supporting the following features: - Authentication: Integration with LDAP and OAuth2 (e.g., Client Credentials Flow, Authorization Code Flow, etc.). - Authorization: Compatibility with tools like Apache Ranger. - Audit: Event storage in systems such as Kafka, S3, or Elasticsearch. - Impersonation: Functionality similar to HiveMetastore. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Subtask] Propose a draft design to add ClickHouse JDBC Catalog Support [gravitino]
flaming-archer commented on issue #6018: URL: https://github.com/apache/gravitino/issues/6018#issuecomment-2562902404 Here is the design document. Please feel free to leave your comments. https://docs.google.com/document/d/1oj1oG_PzVBX_ngRzacupG9lshnH7UjDsso1c6LrhFog/edit?usp=sharing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [EPIC] Support Hbase catalog [gravitino]
flaming-archer opened a new issue, #6019: URL: https://github.com/apache/gravitino/issues/6019 ### Describe the proposal [Apache](https://www.apache.org/) [HBase®](https://hbase.apache.org/) is the [Hadoop](https://hadoop.apache.org/) database, a distributed, scalable, big data store. Use Apache HBase® when you need random, realtime read/write access to your Big Data. This project's goal is the hosting of very large tables -- billions of rows X millions of columns -- atop clusters of commodity hardware. Apache HBase® is an open-source, distributed, versioned, non-relational database modeled after Google's [Bigtable: A Distributed Storage System for Structured Data](https://research.google.com/archive/bigtable.html) by Chang et al. Just as Bigtable leverages the distributed data storage provided by the Google File System, Apache HBase® provides Bigtable-like capabilities on top of Hadoop and HDFS . Gravitino currently supports[ MySQL](https://www.mysql.com/) ,[ PostgreSQL](https://www.postgresql.org/) and[ Apache Doris](https://doris.apache.org/) metadata management but does not support Phoenix yet. This proposal is to add Phoenix JDBC catalog support in Gravitino so that users can have more options for managing database metadata. ### Task list - [ ] @flaming-archer - [ ] -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#6012] feat (gvfs-fuse): Support Gravitino S3 fileset filesystem operation in gvfs fuse [gravitino]
tengqm commented on code in PR #6013:
URL: https://github.com/apache/gravitino/pull/6013#discussion_r1898152704
##
clients/filesystem-fuse/conf/gvfs_fuse.toml:
##
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# fuse settings
+[fuse]
+default_mask = 0o600
+fs_type = "memory"
+
+[fuse.properties]
+key1 = "value1"
+key2 = "value2"
Review Comment:
?
##
clients/filesystem-fuse/src/config.rs:
##
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+use crate::error::ErrorCode::{ConfigNotFound, InvalidConfig};
+use crate::utils::GvfsResult;
+use config::{builder, Config};
+use log::{info, warn};
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fs;
+
+const FUSE_DEFAULT_FILE_MASK: ConfigEntity = ConfigEntity::new(
+FuseConfig::MODULE_NAME,
+"default_file_mask",
+"The default file mask for the FUSE filesystem",
+0o600,
+);
+
+const FUSE_DEFAULT_DIR_MASK: ConfigEntity = ConfigEntity::new(
+FuseConfig::MODULE_NAME,
+"default_dir_mask",
+"The default directory mask for the FUSE filesystem",
+0o700,
+);
+
+const FUSE_FS_TYPE: ConfigEntity<&'static str> = ConfigEntity::new(
Review Comment:
For naming conventions, I'd suggest we remove `FUSE_` from the constant
names and add `DEFAULT_` to all default values for clarity.
##
clients/filesystem-fuse/src/gravitino_client.rs:
##
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+use crate::config::GravitinoConfig;
+use crate::error::{ErrorCode, GvfsError};
+use reqwest::Client;
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fmt::Debug;
+use urlencoding::encode;
+
+#[derive(Debug, Deserialize)]
+pub(crate) struct Fileset {
+pub(crate) name: String,
+#[serde(rename = "type")]
+pub(crate) fileset_type: String,
+comment: String,
+#[serde(rename = "storageLocation")]
+pub(crate) storage_location: String,
+properties: HashMap,
+}
+
+#[derive(Debug, Deserialize)]
+struct FilesetResponse {
+code: i32,
+fileset: Fileset,
+}
+
+#[derive(Debug, Deserialize)]
+struct FileLocationResponse {
+code: i32,
Review Comment:
here too
##
clients/filesystem-fuse/src/config.rs:
##
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You ma
Re: [PR] [#6012] feat (gvfs-fuse): Support Gravitino S3 fileset filesystem operation in gvfs fuse [gravitino]
diqiu50 commented on code in PR #6013:
URL: https://github.com/apache/gravitino/pull/6013#discussion_r1898216648
##
clients/filesystem-fuse/src/gravitino_client.rs:
##
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+use crate::config::GravitinoConfig;
+use crate::error::{ErrorCode, GvfsError};
+use reqwest::Client;
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fmt::Debug;
+use urlencoding::encode;
+
+#[derive(Debug, Deserialize)]
+pub(crate) struct Fileset {
+pub(crate) name: String,
+#[serde(rename = "type")]
+pub(crate) fileset_type: String,
+comment: String,
+#[serde(rename = "storageLocation")]
+pub(crate) storage_location: String,
+properties: HashMap,
+}
+
+#[derive(Debug, Deserialize)]
+struct FilesetResponse {
+code: i32,
Review Comment:
u32 or int32 are both acceptable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5982] feat (gvfs-fuse): Implement Gravitino fileset file system [gravitino]
FANNG1 commented on code in PR #5984:
URL: https://github.com/apache/gravitino/pull/5984#discussion_r1898333523
##
clients/filesystem-fuse/src/gvfs_fileset_fs.rs:
##
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+use crate::config::AppConfig;
+use crate::filesystem::{FileStat, FileSystemCapacity, FileSystemContext,
PathFileSystem, Result};
+use crate::gravitino_client::GravitinoClient;
+use crate::opened_file::{OpenFileFlags, OpenedFile};
+use async_trait::async_trait;
+use fuse3::Errno;
+use std::path::{Path, PathBuf};
+
+pub(crate) struct GravitinoFileSystemConfig {}
+
+/// GravitinoFileSystem is a filesystem that is associated with a fileset in
Gravitino.
+/// It mapping the fileset path to the original data storage path. and
delegate the operation
+/// to the inner filesystem like S3 GCS, JuiceFS.
+pub(crate) struct GvfsFilesetFs {
+fs: Box,
+client: GravitinoClient,
+fileset_location: PathBuf,
+}
+
+impl GvfsFilesetFs {
+pub async fn new(
+fs: Box,
+location: &Path,
+client: GravitinoClient,
+_config: &AppConfig,
+_context: &FileSystemContext,
+) -> Self {
+Self {
+fs: fs,
+client: client,
+fileset_location: location.into(),
+}
+}
+
+fn map_fileset_path_to_raw_path(&self, path: &Path) -> PathBuf {
Review Comment:
the method name seems too long, how about `to_raw_path` and
`to_fileset_path`?
##
clients/filesystem-fuse/src/gravitino_client.rs:
##
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+use crate::config::GravitinoConfig;
+use crate::error::{ErrorCode, GvfsError};
+use reqwest::Client;
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fmt::Debug;
+use urlencoding::encode;
+
+#[derive(Debug, Deserialize)]
+pub(crate) struct Fileset {
+pub(crate) name: String,
+#[serde(rename = "type")]
+pub(crate) fileset_type: String,
+comment: String,
+#[serde(rename = "storageLocation")]
+pub(crate) storage_location: String,
+properties: HashMap,
+}
+
+#[derive(Debug, Deserialize)]
+struct FilesetResponse {
+code: u32,
+fileset: Fileset,
+}
+
+#[derive(Debug, Deserialize)]
+struct FileLocationResponse {
+code: u32,
+#[serde(rename = "fileLocation")]
+location: String,
+}
+
+pub(crate) struct GravitinoClient {
+gravitino_uri: String,
+metalake: String,
+
+client: Client,
+}
+
+impl GravitinoClient {
Review Comment:
The client is binded to fileset, Could you plan to add some abstractions
like `loadCatalog` like java client for more extensiable.
##
clients/filesystem-fuse/src/gvfs_fileset_fs.rs:
##
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND,
Re: [PR] [#5982] feat (gvfs-fuse): Implement Gravitino fileset file system [gravitino]
FANNG1 commented on code in PR #5984:
URL: https://github.com/apache/gravitino/pull/5984#discussion_r1898328151
##
clients/filesystem-fuse/src/gravitino_client.rs:
##
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+use crate::config::GravitinoConfig;
+use crate::error::{ErrorCode, GvfsError};
+use reqwest::Client;
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fmt::Debug;
+use urlencoding::encode;
+
+#[derive(Debug, Deserialize)]
+pub(crate) struct Fileset {
+pub(crate) name: String,
+#[serde(rename = "type")]
+pub(crate) fileset_type: String,
+comment: String,
+#[serde(rename = "storageLocation")]
+pub(crate) storage_location: String,
+properties: HashMap,
+}
+
+#[derive(Debug, Deserialize)]
+struct FilesetResponse {
+code: u32,
+fileset: Fileset,
+}
+
+#[derive(Debug, Deserialize)]
+struct FileLocationResponse {
+code: u32,
+#[serde(rename = "fileLocation")]
+location: String,
+}
+
+pub(crate) struct GravitinoClient {
+gravitino_uri: String,
+metalake: String,
+
+client: Client,
+}
+
+impl GravitinoClient {
Review Comment:
The client is binded to fileset, Could you add some abstractions like
`loadCatalog` like java client for more extensiable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#5902] feat: Add tag failure event to Gravitino server [gravitino]
FANNG1 commented on code in PR #5944:
URL: https://github.com/apache/gravitino/pull/5944#discussion_r1898338425
##
core/src/main/java/org/apache/gravitino/listener/api/event/AlterTagFailureEvent.java:
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.listener.api.event;
+
+import org.apache.gravitino.annotation.DeveloperApi;
+import org.apache.gravitino.tag.TagChange;
+import org.apache.gravitino.tag.TagManager;
+
+/**
+ * Represents an event triggered when an attempt to alter a tag in the
database fails due to an
+ * exception.
+ */
+@DeveloperApi
+public class AlterTagFailureEvent extends TagFailureEvent {
+ private final String metalake;
+ private final String name;
+ private final TagChange[] changes;
+
+ /**
+ * Constructs a new AlterTagFailureEvent.
+ *
+ * @param user the user who attempted to alter the tag
+ * @param metalake the metalake identifier
+ * @param name the name of the tag
+ * @param changes the changes attempted to be made to the tag
+ * @param exception the exception that caused the failure
+ */
+ public AlterTagFailureEvent(
+ String user, String metalake, String name, TagChange[] changes,
Exception exception) {
+super(user, TagManager.ofTagIdent(metalake, name), exception);
Review Comment:
It's odd to refer `TagManager` here, how about move `ofTagIdent` to
`NameIdentifierUtil`?
##
core/src/main/java/org/apache/gravitino/listener/api/event/AlterTagFailureEvent.java:
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.listener.api.event;
+
+import org.apache.gravitino.annotation.DeveloperApi;
+import org.apache.gravitino.tag.TagChange;
+import org.apache.gravitino.tag.TagManager;
+
+/**
+ * Represents an event triggered when an attempt to alter a tag in the
database fails due to an
+ * exception.
+ */
+@DeveloperApi
+public class AlterTagFailureEvent extends TagFailureEvent {
+ private final String metalake;
+ private final String name;
+ private final TagChange[] changes;
+
+ /**
+ * Constructs a new AlterTagFailureEvent.
+ *
+ * @param user the user who attempted to alter the tag
+ * @param metalake the metalake identifier
+ * @param name the name of the tag
+ * @param changes the changes attempted to be made to the tag
+ * @param exception the exception that caused the failure
+ */
+ public AlterTagFailureEvent(
+ String user, String metalake, String name, TagChange[] changes,
Exception exception) {
+super(user, TagManager.ofTagIdent(metalake, name), exception);
+this.name = name;
+this.metalake = metalake;
+this.changes = changes;
+ }
+
+ /**
+ * Returns the name of the tag.
+ *
+ * @return the name of the tag
+ */
+ public String name() {
Review Comment:
please remove `name()` and `metalake()`, because they are not temp variables
to generate tag identifier
##
core/src/main/java/org/apache/gravitino/listener/api/event/AssociateTagsForMetadataObjectFailureEvent.java:
##
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you m
Re: [PR] [#5902] feat: Add tag failure event to Gravitino server [gravitino]
FANNG1 commented on PR #5944: URL: https://github.com/apache/gravitino/pull/5944#issuecomment-2563431271 Could you add related document in `gravitino-server-config.md`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Subtask] Add audit CLI command for Model [gravitino]
VigneshSK17 commented on issue #5962: URL: https://github.com/apache/gravitino/issues/5962#issuecomment-2563160833 No worries, definitely make sure to enjoy your time off! I did have a question about this command. Should this audit command also be auditing the ModelVersion, not do so, or should I be making a separate command for ModelVersion? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5536] Improvement(client-python): reportUndefinedVariable warning in types.py [gravitino]
orenccl commented on PR #6001: URL: https://github.com/apache/gravitino/pull/6001#issuecomment-2563309985 Sure, I will update. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5585] improvement(bundles): Refactor bundle jars and provide core jars that does not contains hadoop-{aws,gcp,aliyun,azure} [gravitino]
yuqi1129 commented on code in PR #5806:
URL: https://github.com/apache/gravitino/pull/5806#discussion_r1898259764
##
docs/how-to-use-gvfs.md:
##
@@ -77,16 +77,15 @@ Apart from the above properties, to access fileset like S3,
GCS, OSS and custom
| `s3-access-key-id` | The access key of the AWS S3.
| (none)|
Yes if it's a S3 fileset.| 0.7.0-incubating |
| `s3-secret-access-key` | The secret key of the AWS S3.
| (none)|
Yes if it's a S3 fileset.| 0.7.0-incubating |
-At the same time, you need to place the corresponding bundle jar
[`gravitino-aws-bundle-${version}.jar`](https://repo1.maven.org/maven2/org/apache/gravitino/aws-bundle/)
in the Hadoop environment(typically located in
`${HADOOP_HOME}/share/hadoop/common/lib/`).
-
+At the same time, you need to place the corresponding bundle jar
[`gravitino-aws-bundle-${version}.jar`](https://repo1.maven.org/maven2/org/apache/gravitino/gravitino-aws-bundle/)
in the Hadoop environment(typically located in
`${HADOOP_HOME}/share/hadoop/common/lib/`).
GCS fileset
| Configuration item | Description
| Default value |
Required | Since version|
||--|---|---|--|
| `gcs-service-account-file` | The path of GCS service account JSON file.
| (none)|
Yes if it's a GCS fileset.| 0.7.0-incubating |
-In the meantime, you need to place the corresponding bundle jar
[`gravitino-gcp-bundle-${version}.jar`](https://repo1.maven.org/maven2/org/apache/gravitino/gcp-bundle/)
in the Hadoop environment(typically located in
`${HADOOP_HOME}/share/hadoop/common/lib/`).
+In the meantime, you need to place the corresponding bundle jar
[`gravitino-gcp-bundle-${version}.jar`](https://repo1.maven.org/maven2/org/apache/gravitino/gravitino-gcp-bundle/)
in the Hadoop environment(typically located in
`${HADOOP_HOME}/share/hadoop/common/lib/`).
Review Comment:
done
##
docs/how-to-use-gvfs.md:
##
@@ -137,8 +137,13 @@ You can configure these properties in two ways:
```
:::note
-If you want to access the S3, GCS, OSS or custom fileset through GVFS, apart
from the above properties, you need to place the corresponding bundle jar in
the Hadoop environment.
-For example if you want to access the S3 fileset, you need to place the S3
bundle jar
[`gravitino-aws-bundle-${version}.jar`](https://repo1.maven.org/maven2/org/apache/gravitino/aws-bundle/)
in the Hadoop environment(typically located in
`${HADOOP_HOME}/share/hadoop/common/lib/`) or add it to the classpath.
+If you want to access the S3, GCS, OSS or custom fileset through GVFS, apart
from the above properties, you need to place the corresponding bundle jars in
the Hadoop environment.
+For example, if you want to access the S3 fileset, you need to place
+1. The S3 hadoop bundle jar
[`gravitino-aws-bundle-${gravitino-version}.jar`](https://repo1.maven.org/maven2/org/apache/gravitino/gravitino-aws-bundle/)
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#6012] feat (gvfs-fuse): Support Gravitino S3 fileset filesystem operation in gvfs fuse [gravitino]
diqiu50 commented on code in PR #6013:
URL: https://github.com/apache/gravitino/pull/6013#discussion_r1898185731
##
clients/filesystem-fuse/src/config.rs:
##
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+use crate::error::ErrorCode::{ConfigNotFound, InvalidConfig};
+use crate::utils::GvfsResult;
+use config::{builder, Config};
+use log::{info, warn};
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fs;
+
+const FUSE_DEFAULT_FILE_MASK: ConfigEntity = ConfigEntity::new(
+FuseConfig::MODULE_NAME,
+"default_file_mask",
+"The default file mask for the FUSE filesystem",
+0o600,
+);
+
+const FUSE_DEFAULT_DIR_MASK: ConfigEntity = ConfigEntity::new(
+FuseConfig::MODULE_NAME,
+"default_dir_mask",
+"The default directory mask for the FUSE filesystem",
+0o700,
+);
+
+const FUSE_FS_TYPE: ConfigEntity<&'static str> = ConfigEntity::new(
+FuseConfig::MODULE_NAME,
+"fs_type",
+"The type of the FUSE filesystem",
+"memory",
+);
+
+const FUSE_CONFIG_PATH: ConfigEntity<&'static str> = ConfigEntity::new(
+FuseConfig::MODULE_NAME,
+"config_path",
+"The path of the FUSE configuration file",
+"/etc/gvfs/gvfs.toml",
Review Comment:
There's no need to write the file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#6012] feat (gvfs-fuse): Support Gravitino S3 fileset filesystem operation in gvfs fuse [gravitino]
diqiu50 commented on code in PR #6013:
URL: https://github.com/apache/gravitino/pull/6013#discussion_r1898216993
##
clients/filesystem-fuse/src/gravitino_client.rs:
##
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+use crate::config::GravitinoConfig;
+use crate::error::{ErrorCode, GvfsError};
+use reqwest::Client;
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fmt::Debug;
+use urlencoding::encode;
+
+#[derive(Debug, Deserialize)]
+pub(crate) struct Fileset {
+pub(crate) name: String,
+#[serde(rename = "type")]
+pub(crate) fileset_type: String,
+comment: String,
+#[serde(rename = "storageLocation")]
+pub(crate) storage_location: String,
+properties: HashMap,
+}
+
+#[derive(Debug, Deserialize)]
+struct FilesetResponse {
+code: i32,
+fileset: Fileset,
+}
+
+#[derive(Debug, Deserialize)]
+struct FileLocationResponse {
+code: i32,
+#[serde(rename = "fileLocation")]
+location: String,
+}
+
+pub(crate) struct GravitinoClient {
+gravitino_uri: String,
+metalake: String,
+
+http_client: Client,
+}
+
+impl GravitinoClient {
+pub fn new(config: &GravitinoConfig) -> Self {
+Self {
+gravitino_uri: config.gravitino_url.clone(),
+metalake: config.metalake.clone(),
+http_client: Client::new(),
+}
+}
+
+pub fn init(&self) {}
+
+pub fn do_post(&self, path: &str, data: &str) {
+println!("POST request to {} with data: {}", path, data);
+}
+
+pub fn request(&self, _path: &str, _data: &str) -> Result<(), GvfsError> {
+todo!()
+}
+
+pub fn list_schema(&self) -> Result<(), GvfsError> {
+todo!()
+}
+
+pub fn list_fileset(&self) -> Result<(), GvfsError> {
+todo!()
+}
+
+fn get_fileset_url(&self, catalog_name: &str, schema_name: &str,
fileset_name: &str) -> String {
+format!(
+"{}/api/metalakes/{}/catalogs/{}/schemas/{}/filesets/{}",
+self.gravitino_uri, self.metalake, catalog_name, schema_name,
fileset_name
+)
+}
+
+async fn do_get(&self, url: &str) -> Result
+where
+T: for<'de> Deserialize<'de>,
+{
+let http_resp =
+self.http_client.get(url).send().await.map_err(|e| {
+GvfsError::RestError(format!("Failed to send request to {}",
url), e)
+})?;
+
+let res = http_resp.json::().await.map_err(|e| {
+GvfsError::RestError(format!("Failed to parse response from {}",
url), e)
+})?;
+
+Ok(res)
+}
+
+pub async fn get_fileset(
+&self,
+catalog_name: &str,
+schema_name: &str,
+fileset_name: &str,
+) -> Result {
+let url = self.get_fileset_url(catalog_name, schema_name,
fileset_name);
+let res = self.do_get::(&url).await?;
+
+if res.code != 0 {
Review Comment:
Yes, it gravitino error code
##
clients/filesystem-fuse/src/gravitino_client.rs:
##
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+use crate::config::GravitinoConfig;
+use crate::error::{ErrorCode, GvfsError};
+use reqwest::Client;
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fmt::Debug;
+use urlencoding::encode;
+
+#[derive(Debug, Deserialize)]
+pub(crate) struct Fileset {
+pub(crate) name: String,
+#[serde(rename = "type")]
+
Re: [PR] [#6012] feat (gvfs-fuse): Support Gravitino S3 fileset filesystem operation in gvfs fuse [gravitino]
diqiu50 commented on code in PR #6013:
URL: https://github.com/apache/gravitino/pull/6013#discussion_r1898217225
##
clients/filesystem-fuse/src/gvfs_fuse.rs:
##
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+use crate::config::AppConfig;
+use crate::default_raw_filesystem::DefaultRawFileSystem;
+use crate::error::ErrorCode::UnSupportedFilesystem;
+use crate::filesystem::FileSystemContext;
+use crate::fuse_api_handle::FuseApiHandle;
+use crate::fuse_server::FuseServer;
+use crate::gvfs_creator::create_gvfs_filesystem;
+use crate::gvfs_fileset_fs::GvfsFilesetFs;
+use crate::memory_filesystem::MemoryFileSystem;
+use crate::utils::GvfsResult;
+use log::info;
+use once_cell::sync::Lazy;
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+static SERVER: Lazy>>> = Lazy::new(||
Mutex::new(None));
+
+pub(crate) enum CreateFsResult {
+Memory(MemoryFileSystem),
+Gvfs(GvfsFilesetFs),
+FuseMemoryFs(FuseApiHandle>),
+FuseGvfs(FuseApiHandle>),
+None,
+}
+
+pub enum FileSystemSchema {
+S3,
+}
+
+pub async fn mount(mount_to: &str, mount_from: &str, config: &AppConfig) ->
GvfsResult<()> {
+info!("Starting gvfs-fuse server...");
+let svr = Arc::new(FuseServer::new(mount_to));
+{
+let mut server = SERVER.lock().await;
+*server = Some(svr.clone());
+}
+let fs = create_fuse_fs(mount_from, config).await?;
+match fs {
+CreateFsResult::FuseMemoryFs(vfs) => svr.start(vfs).await?,
+CreateFsResult::FuseGvfs(vfs) => svr.start(vfs).await?,
+_ => return Err(UnSupportedFilesystem.to_error("Unsupported filesystem
type".to_string())),
+}
+Ok(())
+}
+
+pub async fn unmount() -> GvfsResult<()> {
+info!("Stop gvfs-fuse server...");
+let svr = {
+let mut server = SERVER.lock().await;
+if server.is_none() {
+info!("Server is already stopped.");
+return Ok(());
+}
+server.take().unwrap()
+};
+svr.stop().await
+}
+
+pub(crate) async fn create_fuse_fs(
+mount_from: &str,
+config: &AppConfig,
+) -> GvfsResult {
+let uid = unsafe { libc::getuid() };
+let gid = unsafe { libc::getgid() };
+let fs_context = FileSystemContext {
+uid: uid,
+gid: gid,
+default_file_perm: 0o644,
+default_dir_perm: 0o755,
Review Comment:
fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#6012] feat (gvfs-fuse): Support Gravitino S3 fileset filesystem operation in gvfs fuse [gravitino]
diqiu50 commented on code in PR #6013:
URL: https://github.com/apache/gravitino/pull/6013#discussion_r1898216870
##
clients/filesystem-fuse/src/gravitino_client.rs:
##
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+use crate::config::GravitinoConfig;
+use crate::error::{ErrorCode, GvfsError};
+use reqwest::Client;
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fmt::Debug;
+use urlencoding::encode;
+
+#[derive(Debug, Deserialize)]
+pub(crate) struct Fileset {
+pub(crate) name: String,
+#[serde(rename = "type")]
+pub(crate) fileset_type: String,
+comment: String,
+#[serde(rename = "storageLocation")]
+pub(crate) storage_location: String,
+properties: HashMap,
+}
+
+#[derive(Debug, Deserialize)]
+struct FilesetResponse {
+code: i32,
+fileset: Fileset,
+}
+
+#[derive(Debug, Deserialize)]
+struct FileLocationResponse {
+code: i32,
+#[serde(rename = "fileLocation")]
+location: String,
+}
+
+pub(crate) struct GravitinoClient {
+gravitino_uri: String,
+metalake: String,
+
+http_client: Client,
Review Comment:
fix
##
clients/filesystem-fuse/src/gravitino_client.rs:
##
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+use crate::config::GravitinoConfig;
+use crate::error::{ErrorCode, GvfsError};
+use reqwest::Client;
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fmt::Debug;
+use urlencoding::encode;
+
+#[derive(Debug, Deserialize)]
+pub(crate) struct Fileset {
+pub(crate) name: String,
+#[serde(rename = "type")]
+pub(crate) fileset_type: String,
+comment: String,
+#[serde(rename = "storageLocation")]
+pub(crate) storage_location: String,
+properties: HashMap,
+}
+
+#[derive(Debug, Deserialize)]
+struct FilesetResponse {
+code: i32,
+fileset: Fileset,
+}
+
+#[derive(Debug, Deserialize)]
+struct FileLocationResponse {
+code: i32,
+#[serde(rename = "fileLocation")]
+location: String,
+}
+
+pub(crate) struct GravitinoClient {
+gravitino_uri: String,
+metalake: String,
+
+http_client: Client,
+}
+
+impl GravitinoClient {
+pub fn new(config: &GravitinoConfig) -> Self {
+Self {
+gravitino_uri: config.gravitino_url.clone(),
+metalake: config.metalake.clone(),
+http_client: Client::new(),
+}
+}
+
+pub fn init(&self) {}
+
+pub fn do_post(&self, path: &str, data: &str) {
+println!("POST request to {} with data: {}", path, data);
Review Comment:
Yes, add todo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#6012] feat (gvfs-fuse): Support Gravitino S3 fileset filesystem operation in gvfs fuse [gravitino]
diqiu50 commented on code in PR #6013:
URL: https://github.com/apache/gravitino/pull/6013#discussion_r1898218949
##
clients/filesystem-fuse/src/open_dal_filesystem.rs:
##
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+use crate::config::AppConfig;
+use crate::error::ErrorCode::OpenDalError;
+use crate::filesystem::{
+FileReader, FileStat, FileSystemCapacity, FileSystemContext, FileWriter,
PathFileSystem, Result,
+};
+use crate::gvfs_fuse::FileSystemSchema;
+use crate::opened_file::{OpenFileFlags, OpenedFile};
+use crate::utils::GvfsResult;
+use async_trait::async_trait;
+use bytes::Bytes;
+use fuse3::FileType::{Directory, RegularFile};
+use fuse3::{Errno, FileType, Timestamp};
+use log::{debug, error};
+use opendal::layers::LoggingLayer;
+use opendal::services::S3;
+use opendal::{Builder, EntryMode, ErrorKind, Metadata, Operator};
+use std::path::Path;
+use std::time::SystemTime;
+
+pub(crate) struct OpenDalFileSystem {
+op: Operator,
+}
+
+impl OpenDalFileSystem {}
+
+impl OpenDalFileSystem {
+fn new(op: Operator, _config: &AppConfig, _fs_context: &FileSystemContext)
-> Self {
+Self { op: op }
+}
+
+pub(crate) fn create_file_system(
+schema: &FileSystemSchema,
+config: &AppConfig,
+fs_context: &FileSystemContext,
+) -> GvfsResult> {
+match schema {
+FileSystemSchema::S3 => {
+let builder = S3::from_map(config.extent_config.clone());
+
+let op = Operator::new(builder);
+if let Err(e) = op {
+error!("opendal create failed: {:?}", e);
+return Err(OpenDalError.to_error(format!("opendal create
failed: {:?}", e)));
+}
+let op = op.unwrap().layer(LoggingLayer::default()).finish();
+Ok(Box::new(OpenDalFileSystem::new(op, config, fs_context)))
+}
+}
+}
+
+fn opendal_meta_to_file_stat(&self, meta: &Metadata, file_stat: &mut
FileStat) {
+let now = SystemTime::now();
+let mtime = meta.last_modified().map(|x| x.into()).unwrap_or(now);
+
+file_stat.size = meta.content_length();
+file_stat.kind = opendal_filemode_to_filetype(meta.mode());
+file_stat.ctime = Timestamp::from(mtime);
+file_stat.atime = Timestamp::from(now);
+file_stat.mtime = Timestamp::from(mtime);
+}
+}
+
+#[async_trait]
+impl PathFileSystem for OpenDalFileSystem {
+async fn init(&self) -> Result<()> {
+Ok(())
+}
+
+async fn stat(&self, path: &Path) -> Result {
+let file_name = path.to_string_lossy().to_string();
+let meta = self
+.op
+.stat_with(&file_name)
+.await
+.map_err(opendal_error_to_errno)?;
+
+let mut file_stat = FileStat::new_file_filestat_with_path(path, 0);
+self.opendal_meta_to_file_stat(&meta, &mut file_stat);
+Ok(file_stat)
+}
+
+async fn read_dir(&self, path: &Path) -> Result> {
+let file_name = path.to_string_lossy().to_string();
Review Comment:
That‘s impossible.
Actually, It is impossible to distinguish a file from a directory based on
the `path` alone. If the wrong type is passed, an error will occur later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#6012] feat (gvfs-fuse): Support Gravitino S3 fileset filesystem operation in gvfs fuse [gravitino]
diqiu50 commented on code in PR #6013:
URL: https://github.com/apache/gravitino/pull/6013#discussion_r1898218180
##
clients/filesystem-fuse/src/main.rs:
##
@@ -16,18 +16,32 @@
* specific language governing permissions and limitations
* under the License.
*/
+use fuse3::Errno;
+use gvfs_fuse::config::AppConfig;
use gvfs_fuse::{gvfs_mount, gvfs_unmount};
-use log::info;
+use log::{error, info};
use tokio::signal;
#[tokio::main]
async fn main() -> fuse3::Result<()> {
tracing_subscriber::fmt().init();
-tokio::spawn(async { gvfs_mount("gvfs").await });
+
+let config = AppConfig::from_file(Some("conf/gvfs_fuse.toml)"));
Review Comment:
That's not implemented. we need to read config file path from app arguments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#6012] feat (gvfs-fuse): Support Gravitino S3 fileset filesystem operation in gvfs fuse [gravitino]
diqiu50 commented on code in PR #6013: URL: https://github.com/apache/gravitino/pull/6013#discussion_r1898220362 ## clients/filesystem-fuse/conf/gvfs_fuse.toml: ## @@ -0,0 +1,39 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# fuse settings +[fuse] +default_mask = 0o600 +fs_type = "memory" + +[fuse.properties] +key1 = "value1" +key2 = "value2" + +# filesystem settings +[filesystem] +block_size = 8192 + +# Gravitino settings +[gravitino] +gravitino_url = "http://localhost:8090"; +metalake = "test" + +# extent settings +[extent_config] +access_key = "XXX_access_key" +secret_key = "XXX_secret_key" Review Comment: fix -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5536] Improvement(client-python): reportUndefinedVariable warning in types.py [gravitino]
jerryshao commented on PR #6001: URL: https://github.com/apache/gravitino/pull/6001#issuecomment-2563285454 I see, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5536] Improvement(client-python): reportUndefinedVariable warning in types.py [gravitino]
jerryshao commented on PR #6001: URL: https://github.com/apache/gravitino/pull/6001#issuecomment-2563285824 Can you please check all the python codes to see if there's similar issue in other files, I think we can fix it all in this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug report] When the authentication is configured to Kerberos, the server fails to start and hangs. [gravitino]
Aireed opened a new issue, #6022:
URL: https://github.com/apache/gravitino/issues/6022
### Version
main branch
### Describe what's wrong
The service did not start properly.
### Error message and/or stacktrace
gravitino-server.log:
```
2024-12-27 11:57:16.442 INFO [main]
[org.apache.gravitino.server.GravitinoServer.main(GravitinoServer.java:163)] -
Starting Gravitino Server
2024-12-27 11:57:16.530 INFO [main]
[org.apache.gravitino.GravitinoEnv.initializeFullComponents(GravitinoEnv.java:167)]
- Initializing Gravitino full environment...
2024-12-27 11:57:16.567 INFO [main]
[org.apache.gravitino.metrics.MetricsSystem.register(MetricsSystem.java:78)] -
Register jvm to metrics system
2024-12-27 11:57:16.599 INFO [main]
[org.apache.gravitino.audit.AuditLogManager.init(AuditLogManager.java:47)] -
Audit log is not enabled
2024-12-27 11:57:17.118 INFO [main]
[org.apache.gravitino.auxiliary.AuxiliaryServiceManager.registerAuxService(AuxiliaryServiceManager.java:138)]
- AuxService name:iceberg-rest, config:{jdbc.password=3221631,
jdbc.user=readwrite, io-impl=org.apache.iceberg.aws.s3.S3FileIO,
jdbc-driver=com.mysql.cj.jdbc.Driver,
s3-secret-access-key=q3XuiBJFrw6NpY64teJInWn70wxuBjY21sIaI8PE,
classpath=iceberg-rest-server/libs, iceberg-rest-server/conf, httpPort=9001,
warehouse=s3://demo1/warehouse/amoro,
uri=jdbc:mysql://10.122.173.167:3306/jdbc_iceberg_catalog?useUnicode=true&characterEncoding=UTF8&autoReconnect=true&useAffectedRows=true&useSSL=false,
s3-endpoint=http://10.45.132.189:9000, catalog-backend=jdbc,
s3-access-key-id=Py5fhy8vrmkDYYqPlT8G, catalog-backend-name=jdbc, host=0.0.0.0,
s3-region=amoro-dev, authType=simple}, valid
classpath:[/home/arctic/gravitino/gravitino-0.8.0-incubating-SNAPSHOT-bin/iceberg-rest-server/libs,
/home/arctic/gravitino/gravitino-0.8.0-incubating-SNAPSHOT-bin/icebe
rg-rest-server/conf]
2024-12-27 11:57:17.191 INFO [main]
[org.apache.gravitino.auxiliary.AuxiliaryServiceManager.registerAuxService(AuxiliaryServiceManager.java:151)]
- AuxService:iceberg-rest registered successfully
2024-12-27 11:57:17.219 INFO [main]
[org.eclipse.jetty.util.log.Log.initialized(Log.java:170)] - Logging
initialized @1754ms to org.eclipse.jetty.util.log.Slf4jLog
2024-12-27 11:57:17.446 INFO [main]
[org.apache.gravitino.metrics.MetricsSystem.register(MetricsSystem.java:78)] -
Register iceberg-rest-server to metrics system
2024-12-27 11:57:17.451 INFO [main]
[org.apache.gravitino.iceberg.service.provider.IcebergConfigProviderFactory.create(IcebergConfigProviderFactory.java:44)]
- Load Iceberg catalog provider:
org.apache.gravitino.iceberg.service.provider.StaticIcebergConfigProvider.
2024-12-27 11:57:17.643 INFO [main]
[org.apache.gravitino.iceberg.service.metrics.IcebergMetricsManager.loadIcebergMetricsStore(IcebergMetricsManager.java:176)]
- Load Iceberg metrics store:
org.apache.gravitino.iceberg.service.metrics.DummyMetricsStore.
2024-12-27 11:57:17.746 INFO [main]
[org.apache.gravitino.iceberg.RESTService.serviceInit(RESTService.java:138)] -
Iceberg REST service init.
2024-12-27 11:57:17.752 INFO [tree-lock-dead-lock-checker-0]
[org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:114)]
- Start to check the dead lock...
2024-12-27 11:57:17.753 INFO [tree-lock-dead-lock-checker-0]
[org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:116)]
- Finish to check the dead lock...
2024-12-27 11:57:17.755 INFO [main]
[org.apache.gravitino.GravitinoEnv.initializeFullComponents(GravitinoEnv.java:172)]
- Gravitino full environment is initialized.
2024-12-27 11:57:17.798 INFO [main]
[org.apache.gravitino.server.web.JettyServer.initializeWebAppServletContextHandler(JettyServer.java:309)]
- Gravitino Webapp path: /tmp/GravitinoWar4666932943606397997
```
### How to reproduce
set `gravitino.authenticators` to `kerberos`
gravatino.conf
```
gravitino.authorization.enable = true
gravitino.authenticators = kerberos
gravitino.authenticator.kerberos.principal = hive/h...@xxx.com
gravitino.authenticator.kerberos.keytab =
/etc/security/keytabs/hive/hive.service.keytab
```
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [#6012] feat (gvfs-fuse): Support Gravitino S3 fileset filesystem operation in gvfs fuse [gravitino]
diqiu50 commented on code in PR #6013: URL: https://github.com/apache/gravitino/pull/6013#discussion_r1898220290 ## clients/filesystem-fuse/conf/gvfs_fuse.toml: ## @@ -0,0 +1,39 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# fuse settings +[fuse] +default_mask = 0o600 +fs_type = "memory" + +[fuse.properties] +key1 = "value1" +key2 = "value2" Review Comment: remove ## clients/filesystem-fuse/conf/gvfs_fuse.toml: ## @@ -0,0 +1,39 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# fuse settings +[fuse] +default_mask = 0o600 +fs_type = "memory" + +[fuse.properties] +key1 = "value1" +key2 = "value2" + +# filesystem settings +[filesystem] +block_size = 8192 + +# Gravitino settings +[gravitino] +gravitino_url = "http://localhost:8090"; +metalake = "test" Review Comment: fix -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [#5968] fix(server-common): The owner of the catalog is incorrect when using Basic Auth and Password is empty [gravitino]
frankvicky opened a new pull request, #6023: URL: https://github.com/apache/gravitino/pull/6023 ### Why are the changes needed? Current implementation of `SimpleAuthenticator` doesn't comply with HTTP Basic Authentication specification, which allows username-only or username with empty password formats. Fix: #5968 ### Does this PR introduce _any_ user-facing change? n/a ### How was this patch tested? Unit test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5968] fix(server-common): The owner of the catalog is incorrect when using Basic Auth and Password is empty [gravitino]
frankvicky commented on PR #6023: URL: https://github.com/apache/gravitino/pull/6023#issuecomment-2563290844 Hi @jerqi Could you please take a look ? Many thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5968] fix(server-common): The owner of the catalog is incorrect when using Basic Auth and Password is empty [gravitino]
jerqi commented on PR #6023: URL: https://github.com/apache/gravitino/pull/6023#issuecomment-2563293656 Could you add more cases? like`user1:`,`:password`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#6012] feat (gvfs-fuse): Support Gravitino S3 fileset filesystem operation in gvfs fuse [gravitino]
diqiu50 commented on code in PR #6013:
URL: https://github.com/apache/gravitino/pull/6013#discussion_r1898191633
##
clients/filesystem-fuse/src/config.rs:
##
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+use crate::error::ErrorCode::{ConfigNotFound, InvalidConfig};
+use crate::utils::GvfsResult;
+use config::{builder, Config};
+use log::{info, warn};
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fs;
+
+const FUSE_DEFAULT_FILE_MASK: ConfigEntity = ConfigEntity::new(
+FuseConfig::MODULE_NAME,
+"default_file_mask",
+"The default file mask for the FUSE filesystem",
+0o600,
+);
+
+const FUSE_DEFAULT_DIR_MASK: ConfigEntity = ConfigEntity::new(
+FuseConfig::MODULE_NAME,
+"default_dir_mask",
+"The default directory mask for the FUSE filesystem",
+0o700,
+);
+
+const FUSE_FS_TYPE: ConfigEntity<&'static str> = ConfigEntity::new(
+FuseConfig::MODULE_NAME,
+"fs_type",
+"The type of the FUSE filesystem",
+"memory",
+);
+
+const FUSE_CONFIG_PATH: ConfigEntity<&'static str> = ConfigEntity::new(
+FuseConfig::MODULE_NAME,
+"config_path",
+"The path of the FUSE configuration file",
+"/etc/gvfs/gvfs.toml",
+);
+
+const FILESYSTEM_BLOCK_SIZE: ConfigEntity = ConfigEntity::new(
+FilesystemConfig::MODULE_NAME,
+"block_size",
+"The block size of the gvfs fuse filesystem",
+4096,
+);
+
+const GRAVITINO_URL: ConfigEntity<&'static str> = ConfigEntity::new(
+GravitinoConfig::MODULE_NAME,
+"gravitino_url",
+"The URL of the Gravitino server",
+"http://localhost:8090";,
+);
+
+const GRAVITINO_METALAKE: ConfigEntity<&'static str> = ConfigEntity::new(
+GravitinoConfig::MODULE_NAME,
+"metalake",
+"The metalake of the Gravitino server",
+"",
+);
+
+struct ConfigEntity {
+module: &'static str,
+name: &'static str,
+description: &'static str,
+default: T,
+}
+
+impl ConfigEntity {
+const fn new(
+module: &'static str,
+name: &'static str,
+description: &'static str,
+default: T,
+) -> Self {
+ConfigEntity {
+module: module,
+name: name,
+description: description,
+default: default,
+}
+}
+}
+
+enum ConfigValue {
+I32(ConfigEntity),
+U32(ConfigEntity),
+String(ConfigEntity<&'static str>),
+Bool(ConfigEntity),
+Float(ConfigEntity),
+}
+
+struct DefaultConfig {
+configs: HashMap,
+}
+
+impl Default for DefaultConfig {
+fn default() -> Self {
+let mut configs = HashMap::new();
+
+configs.insert(
+Self::compose_key(FUSE_DEFAULT_FILE_MASK),
+ConfigValue::U32(FUSE_DEFAULT_FILE_MASK),
+);
+configs.insert(
+Self::compose_key(FUSE_DEFAULT_DIR_MASK),
+ConfigValue::U32(FUSE_DEFAULT_DIR_MASK),
+);
+configs.insert(
+Self::compose_key(FUSE_FS_TYPE),
+ConfigValue::String(FUSE_FS_TYPE),
+);
+configs.insert(
+Self::compose_key(FUSE_CONFIG_PATH),
+ConfigValue::String(FUSE_CONFIG_PATH),
+);
+configs.insert(
+Self::compose_key(GRAVITINO_URL),
+ConfigValue::String(GRAVITINO_URL),
+);
+configs.insert(
+Self::compose_key(GRAVITINO_METALAKE),
+ConfigValue::String(GRAVITINO_METALAKE),
+);
+configs.insert(
+Self::compose_key(FILESYSTEM_BLOCK_SIZE),
+ConfigValue::U32(FILESYSTEM_BLOCK_SIZE),
+);
+
+DefaultConfig { configs }
+}
+}
+
+impl DefaultConfig {
+fn compose_key(entity: ConfigEntity) -> String {
+format!("{}.{}", entity.module, entity.name)
Review Comment:
There is no such restriction. In our configuration class, we can set the
configuration name corresponding to the variable name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructu
Re: [PR] [#6014] refactor: CLI output methods for no data hints [gravitino]
tengqm commented on PR #6015: URL: https://github.com/apache/gravitino/pull/6015#issuecomment-2563179429 lgtm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#5889] feat(client-python): Add model management Python API [gravitino]
tengqm commented on code in PR #6009: URL: https://github.com/apache/gravitino/pull/6009#discussion_r1898161883 ## clients/client-python/gravitino/client/generic_model_catalog.py: ## @@ -0,0 +1,534 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +from typing import Dict, List + +from gravitino.name_identifier import NameIdentifier +from gravitino.api.catalog import Catalog +from gravitino.api.model import Model +from gravitino.api.model_version import ModelVersion +from gravitino.client.base_schema_catalog import BaseSchemaCatalog +from gravitino.client.generic_model import GenericModel +from gravitino.client.generic_model_version import GenericModelVersion +from gravitino.dto.audit_dto import AuditDTO +from gravitino.dto.requests.model_register_request import ModelRegisterRequest +from gravitino.dto.requests.model_version_link_request import ModelVersionLinkRequest +from gravitino.dto.responses.base_response import BaseResponse +from gravitino.dto.responses.drop_response import DropResponse +from gravitino.dto.responses.entity_list_response import EntityListResponse +from gravitino.dto.responses.model_response import ModelResponse +from gravitino.dto.responses.model_version_list_response import ModelVersionListResponse +from gravitino.dto.responses.model_vesion_response import ModelVersionResponse +from gravitino.exceptions.base import NoSuchModelException, NoSuchModelVersionException +from gravitino.exceptions.handlers.model_error_handler import MODEL_ERROR_HANDLER +from gravitino.namespace import Namespace +from gravitino.rest.rest_utils import encode_string +from gravitino.utils import HTTPClient + + +class GenericModelCatalog(BaseSchemaCatalog): +""" +The generic model catalog is a catalog that supports model and model version operations, +for example, model register, model version link, model and model version list, etc. +A model catalog is under the metalake. +""" + +def __init__( +self, +namespace: Namespace, +name: str = None, +catalog_type: Catalog.Type = Catalog.Type.UNSUPPORTED, +provider: str = None, +comment: str = None, +properties: Dict[str, str] = None, +audit: AuditDTO = None, +rest_client: HTTPClient = None, +): +super().__init__( +namespace, +name, +catalog_type, +provider, +comment, +properties, +audit, +rest_client, +) + +def as_model_catalog(self): +return self + +def list_models(self, namespace: Namespace) -> List[NameIdentifier]: +"""List the models in a schema namespace from the catalog. + +Args: +namespace: The namespace of the schema. + +Raises: +NoSuchSchemaException: If the schema does not exist. + +Returns: +A list of NameIdentifier of models under the given namespace. +""" +self._check_model_namespace(namespace) + +model_full_ns = self._model_full_namespace(namespace) +resp = self.rest_client.get( +self._format_model_request_path(model_full_ns), +error_handler=MODEL_ERROR_HANDLER, +) Review Comment: Sounds a little bit dangerous if the GET doesn't work as expected, or the return is not `application/json`. ## clients/client-python/gravitino/api/model_version.py: ## @@ -0,0 +1,85 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions
