CalvinKirs opened a new pull request, #65586:
URL: https://github.com/apache/doris/pull/65586
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Hadoop's `FileSystem.CACHE` keys entries only by `(scheme, authority, UGI)`.
Different catalogs/TVFs accessing the same bucket/namenode with different
credentials could receive each other's cached `FileSystem` instance
(cross-credential contamination), so Doris forced
`fs.<schema>.impl.disable.cache=true` everywhere — which makes every access
(e.g. every JNI scanner split) create a brand-new `FileSystem` instance that is
never closed: slow and leaky.
This PR makes the cache credential-aware instead of disabling it:
**BE/FE shared patch (hadoop-deps):**
- Ship a patched copy of `org.apache.hadoop.fs.FileSystem` (verbatim from
hadoop-common 3.4.2, hunks marked `DORIS-PATCH`): `Cache.Key` carries one extra
dimension read from the reserved conf property `doris.fs.cache.key`.
Absent/empty value = byte-for-byte vanilla behavior.
- Load-order guarantees: BE loads `be/lib/hadoop_hdfs/hadoop-deps.jar`
before `lib/*.jar` (existing); FE now declares `hadoop-deps` as fe-core's first
dependency (all transitives excluded) and `start_fe.sh` prepends the jar
explicitly, so both JVMs (including JNI scanners and libhdfs) see the patched
class.
- maven-enforcer pins `hadoop.version=3.4.2`: upgrading hadoop fails the
build until the patched file is re-synced.
**FE fingerprint injection:**
- `StorageProperties` computes a stable 32-hex SHA-256 fingerprint from the
concrete class name + `matchedProperties` (the user-supplied storage
definition, credentials included) and injects it as `doris.fs.cache.key`
through a final `getBackendConfigProperties()` wrapper (subclasses now
implement `doGetBackendConfigProperties()`) and through `hadoopStorageConfig`.
- Runtime/vended credentials chain into the fingerprint so per-session
credentials never share a slot.
- Every multi-storage merge site (CatalogProperty backend/hadoop maps,
CredentialUtils, Iceberg/Paimon metastore `Configuration` builders, Iceberg
table/delete/merge sinks) replaces the last-wins key with an order-independent
combined fingerprint via `StorageProperties.setCombinedFsCacheKey`.
- The blanket `fs.<schema>.impl.disable.cache=true` defaults are removed;
explicit user-provided values are still honored as an escape hatch. FE
`DFSFileSystem` keeps its per-instance disable: it exists for manual lifecycle
management (RemoteFSPhantomManager closes evicted instances), not credential
isolation.
**Notes:**
- Rolling upgrade: upgrade BE before FE (standard order). Old FE + new BE
stays safe because old FE still sends the disable flags; new FE + old BE would
re-expose the vanilla cache.
- `FileSystem.CACHE` has no eviction, so rotating vended credentials
accumulate entries over time — still strictly better than today's
new-instance-per-access behavior.
### Release note
Hadoop FileSystem cache is now enabled and keyed by a per-storage credential
fingerprint (`doris.fs.cache.key`) instead of being disabled globally; explicit
`fs.<schema>.impl.disable.cache` settings are still honored.
### Check List (For Author)
- Test
- [x] Unit test: `DorisFileSystemCacheKeyTest` (patched class loaded,
vanilla default behavior, distinct keys isolate, equal keys share),
`StoragePropertiesFsCacheFingerprintTest` (stability, credential sensitivity,
both injection channels, combined fingerprint semantics), updated
GCS/S3/COS/OBS/OSS property tests
- [ ] Regression test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test.
- Behavior changed:
- [ ] No.
- [x] Yes. Hadoop FileSystem caching is enabled again with
credential-aware keys; blanket cache disabling removed.
- Does this need documentation?
- [ ] No.
- [x] Yes. `doris.fs.cache.key` is a reserved property; upgrade order
note (BE before FE).
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
