CalvinKirs opened a new pull request, #64695: URL: https://github.com/apache/doris/pull/64695
## What This PR contains two related fe-filesystem changes: ### 1. Migrate kernel HDFS properties into fe-filesystem The HDFS SPI module previously passed raw user properties straight into a Hadoop `Configuration` without any translation. When fed raw properties it silently dropped the typed authentication params — most importantly `hdfs.authentication.kerberos.principal/keytab` were never mapped to `hadoop.kerberos.principal/keytab`, so Kerberos was not recognized and the client fell back to simple auth without any error. This change ports the kernel `HdfsProperties` inheritance chain into a new package `org.apache.doris.filesystem.hdfs.properties`, preserving the parameter translation, xml-resource loading, and default injection, while keeping the module free of fe-core / fe-common (only the already-allowed `fe-foundation` dependency is added): - Copies `ConnectionProperties`, `StorageProperties` (minus the provider-registry static factory, which referenced every storage provider), `HdfsCompatibleProperties` (minus the `hadoopAuthenticator` field that depended on fe-common security), `HdfsProperties`, and `HdfsPropertiesUtils`. - Adds a minimal local `UserException` stub (the original drags in `ErrorCode`/`InternalErrorCode`) and a local `HdfsConfigFileLoader` replacing `CatalogConfigFileUtils`. - `HdfsFileSystemProvider.create()` now resolves raw properties through `HdfsProperties` before constructing `DFSFileSystem`. - The FE-side hadoop config directory is passed down as a system-injected context key `_HADOOP_CONFIG_DIR_` (set by `StoragePropertiesConverter`), so the filesystem module can resolve `hadoop.config.resources` without depending on fe-core `Config`. ### 2. OSS listing / exception / URI fixes - Fix OSS recursive list truncation by falling back to the last returned key when `NextMarker` is absent. - Catch `OSSException` (a sibling of `ClientException`) so server-side OSS errors are translated to `IOException` instead of leaking. - Consolidate object-storage URI parsing into a single shared, scheme-agnostic parser and align path-style handling across S3/OSS/COS/OBS. ## Tests - `fe-filesystem-hdfs`: full module test suite passes, including the new `HdfsPropertiesTest` (Kerberos translation, simple-auth fallback default, user-overridden keys preserved, xml load from injected/absolute paths, idempotency). 0 checkstyle violations. - A focused `StoragePropertiesConverter` test asserting `_HADOOP_CONFIG_DIR_` injection is added on the fe-core side. - The fe-core module could not be fully compiled in the local environment; CI validates that side. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
