yuqi1129 commented on code in PR #5020:
URL: https://github.com/apache/gravitino/pull/5020#discussion_r1801226391
##########
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/HadoopCatalogOperations.java:
##########
@@ -119,26 +125,35 @@ public void initialize(
Map<String, String> config, CatalogInfo info, HasPropertyMetadata
propertiesMetadata)
throws RuntimeException {
this.propertiesMetadata = propertiesMetadata;
- // Initialize Hadoop Configuration.
- this.conf = config;
- this.hadoopConf = new Configuration();
this.catalogInfo = info;
- Map<String, String> bypassConfigs =
- config.entrySet().stream()
- .filter(e -> e.getKey().startsWith(CATALOG_BYPASS_PREFIX))
- .collect(
- Collectors.toMap(
- e -> e.getKey().substring(CATALOG_BYPASS_PREFIX.length()),
- Map.Entry::getValue));
- bypassConfigs.forEach(hadoopConf::set);
+
+ this.conf = config;
+
+ String fileSystemProviders =
+ (String)
+ propertiesMetadata
+ .catalogPropertiesMetadata()
+ .getOrDefault(
+ config,
HadoopCatalogPropertiesMetadata.FILESYSTEM_PROVIDERS_CLASSNAMES);
+ FileSystemUtils.initFileSystemProviders(fileSystemProviders,
fileSystemProvidersMap);
+
+ String defaultFileSystemProviderClassName =
+ (String)
+ propertiesMetadata
+ .catalogPropertiesMetadata()
+ .getOrDefault(
+ config,
HadoopCatalogPropertiesMetadata.DEFAULT_FS_PROVIDER_CLASSNAME);
+ this.defaultFileSystemProviderScheme =
+ StringUtils.isNotBlank(defaultFileSystemProviderClassName)
+ ? FileSystemUtils.getSchemeByFileSystemProvider(
+ defaultFileSystemProviderClassName, fileSystemProvidersMap)
+ : LOCAL_FILE_SCHEME;
Review Comment:
If we use the service load mechanism to load file system providers, I can't
get why we need to provide the configuration `filesystem-providers`, please see
below
```java
// fileSystemProviders is useless if we use ServiceLoader
public static Map<String, FileSystemProvider>
getFileSystemProviders(String fileSystemProviders) {
Map<String, FileSystemProvider> resultMap = Maps.newHashMap();
ServiceLoader<FileSystemProvider> fileSystemProvidersLoader =
ServiceLoader.load(FileSystemProvider.class);
fileSystemProvidersLoader.forEach(
fileSystemProvider -> resultMap.put(fileSystemProvider.name(),
fileSystemProvider));
return resultMap;
}
```
then the only configuration we need to provide is
`default-filesystem-provider`, @jerryshao what do you think about it, is it
okay for you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gravitino.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
