CalvinKirs opened a new pull request, #66315:
URL: https://github.com/apache/doris/pull/66315
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #65586 (the branch-4.1 change this ports)
Problem Summary:
Hadoop's `FileSystem.CACHE` keys entries only by `(scheme, authority, UGI)`.
Different catalogs/TVFs accessing the same bucket/namenode with different
credentials could receive each other's cached `FileSystem` instance
(cross-credential contamination), so Doris forced
`fs.<schema>.impl.disable.cache=true` everywhere — which makes every access
(e.g. every JNI scanner split) create a brand-new `FileSystem` instance
(closing/releasing depends on GC).
This PR makes the cache credential-aware instead of disabling it. It is the
master counterpart of #65586; the 4.1 code no longer exists here (the
connector-plugin refactor moved storage properties into the `fe-filesystem/*`
SPI plugins and replaced the typed `StorageProperties` hierarchy with
`StorageAdapter`), so this is the same design re-layered rather than a
cherry-pick.
**BE/FE shared patch (hadoop-deps):**
- Ship a patched copy of `org.apache.hadoop.fs.FileSystem` (verbatim from
hadoop-common 3.4.2, hunks marked `DORIS-PATCH`): `Cache.Key` carries one extra
dimension read from the reserved conf property `doris.fs.cache.key`.
Absent/empty value = byte-for-byte vanilla behavior.
- maven-enforcer pins `hadoop.version=3.4.2`: upgrading hadoop fails the
build until the patched file is re-synced.
**Load-order guarantees (three classpaths):**
- BE loads `be/lib/hadoop_hdfs/hadoop-deps.jar` before `lib/*.jar`
(existing).
- FE declares `hadoop-deps` as fe-core's first dependency (all transitives
excluded) and `start_fe.sh` prepends the jar explicitly.
- **Plugins.** Filesystem plugins already resolve `org.apache.hadoop.*`
parent-first (`FileSystemPluginManager.FS_PARENT_FIRST_PREFIXES`), so they see
the FE copy. Connector plugins (iceberg/paimon/hudi/hive) deliberately bundle
their own `hadoop-common` and load it **child-first**, so `hadoop-deps` is
added to those four plugins and `DirectoryPluginRuntimeManager` promotes any
`hadoop-deps*.jar` to the front of the plugin classpath. Neither directory
position nor `collectJars()`'s lexicographic order delivers that
(`"hadoop-common-x.y.z.jar"` sorts *before* `"hadoop-deps-*.jar"`), and the
promoted jar is excluded from the service-discovery set — discovery narrows to
root-level jars whenever any exist, so a jar sitting at the plugin root that
registers no factory would otherwise starve it. Both failure modes are covered
by `DirectoryPluginRuntimeManagerPatchedHadoopJarTest`.
**FE fingerprint injection:**
- `FsCacheKeys` (fe-filesystem-api) computes a stable 32-hex SHA-256
fingerprint from the concrete class name + `matchedProperties` (the
user-supplied storage definition, credentials included), exposed as
`StorageProperties.fsCacheFingerprint()`.
- `StorageAdapter` injects it into `getBackendConfigProperties()` (now a
defensive copy) and into `hadoopStorageConfig`; the SPI providers inject it
into their own hadoop/backend maps in place of the removed disable-cache keys.
- Every multi-storage merge site (`CredentialUtils`,
iceberg/paimon/hudi/hive scan + write + catalog config builders) replaces the
last-wins key with an order-independent combined fingerprint. REST vended
credentials chain into the fingerprint, so per-session credentials never share
a slot and the static storage identity is not dropped.
- The blanket `fs.<schema>.impl.disable.cache=true` defaults are removed;
explicit user-provided values are still honored as an escape hatch. The HDFS
plugin's `HdfsConfigBuilder` keeps its per-instance disable: it exists for
manual lifecycle management (`DFSFileSystem.close`), not credential isolation.
`OutFileClause` no longer mutates the returned backend map (it is a copy
now); it injects `fs.defaultFS` into the source properties and rebuilds the
descriptor.
**Notes:**
- Rolling upgrade: upgrade BE before FE (standard order). Old FE + new BE
stays safe because old FE still sends the disable flags; new FE + old BE would
re-expose the vanilla cache.
- `FileSystem.CACHE` has no eviction, so rotating vended credentials
accumulate entries over time — still strictly better than today's
new-instance-per-access behavior.
### Release note
Hadoop FileSystem cache is now enabled and keyed by a per-storage credential
fingerprint (`doris.fs.cache.key`) instead of being disabled globally; explicit
`fs.<schema>.impl.disable.cache` settings are still honored.
### Check List (For Author)
- Test
- [x] Unit test:
- `DorisFileSystemCacheKeyTest` — patched class loaded, vanilla
default behavior, distinct keys isolate, equal keys share
- `StorageAdapterFsCacheFingerprintTest` — fingerprint stability /
credential sensitivity, both injection channels, defensive copy, no blanket
disable + explicit override honored, combined + overlay fingerprint semantics,
patched `Cache.Key` on the FE classpath
- `DirectoryPluginRuntimeManagerPatchedHadoopJarTest` — patched jar
wins over a plugin's bundled hadoop-common, and does not starve service
discovery
- Updated HDFS / OSS-HDFS / JFS SPI parity goldens and
`CredentialUtilsTest`
- [ ] Regression test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test.
- Behavior changed:
- [ ] No.
- [x] Yes. Hadoop FileSystem caching is enabled again with
credential-aware keys; blanket cache disabling removed.
- Does this need documentation?
- [ ] No.
- [x] Yes. `doris.fs.cache.key` is a reserved property; upgrade order
note (BE before FE).
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
