CalvinKirs opened a new pull request, #66004:
URL: https://github.com/apache/doris/pull/66004
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #65834
Problem Summary:
fe-core still carries its own typed storage-properties hierarchy
(`datasource/property/storage/*`, `fs/SchemaTypeMapper`,
`fs/StoragePropertiesConverter`) in parallel with the fe-filesystem SPI
plugins. Every storage consumer in fe-core is coupled to those typed classes,
which blocks deleting the duplicated hierarchy and keeps storage knowledge
split across two stacks.
This PR migrates every fe-core main-source consumer onto the fe-filesystem
SPI, behind a single fe-core facade. After this PR, fe-core `src/main` has
**zero references** to the legacy package (gate: `grep -rE
"datasource\.property\.storage\.|fs\.SchemaTypeMapper|StoragePropertiesConverter"
fe/fe-core/src/main/java --include=*.java | grep -v
"/datasource/property/storage/" | grep -v SchemaTypeMapper.java | grep -v
StoragePropertiesConverter.java` returns empty). The legacy classes are
intentionally kept in-tree as golden-test oracles; a follow-up PR removes them.
**Approach**
- **SPI readiness**: HDFS-family typed properties implement the SPI
interfaces; auth is unified on a foundation-level `ExecutionAuthenticator`
(doAs abstraction, no Hadoop types) with UGI holders staying in impl modules;
all 14 providers implement `supportsExplicit`/`supportsGuess`;
`bindPrimary`/`bindAll` mirror the legacy `createPrimary`/`createAll` routing
(priority order, explicit `fs.<x>.support` flags disabling guesses,
OSS-HDFS/OSS exclusivity, default-HDFS fallback at index 0).
- **fe-core facade** (`datasource/storage`): `StorageAdapter` wraps one SPI
binding with the legacy consumer surface (backend map, Hadoop conf, storage
name, schemas, type id, URI validation, broker params, AWS credentials
providers); `StorageTypeId`/`StorageSchemeRegistry` replace the Type enum and
`SchemaTypeMapper`; `S3ThriftAdapter` (`TS3StorageParam`) and
`CloudObjectStoreAdapter` (`ObjectStoreInfoPB`) are verbatim moves of the wire
glue; `S3ResourceCompat` is the verbatim home for the image/DDL-bearing S3
constants and map utilities so `S3Resource`/`AzureResource` logic stays
untouched.
- **Consumers**: CatalogProperty, LocationPath, BrokerDesc/StorageDesc, all
TVFs, backup/restore, resources/vaults/policy, connectivity testers, vended
credentials, iceberg/paimon/hive integrations and cloud copy-into are re-typed
mechanically; behavior is byte-frozen by ~150 golden parity tests that compare
the new path against the in-tree legacy classes as oracles (thrift/PB structs
compared as whole objects).
- Known behavior deltas are deliberate and test-locked: S3-family no longer
silently defaults region to us-east-1 (same error as fe-core raised), OSS
non-standard endpoints are rewritten like fe-core did, multi-provider
backend/Hadoop map merge order is pinned to binding order (legacy iterated an
enum-keyed HashMap), and `validateAndNormalizeUri` failures surface as
unchecked `StoragePropertiesException` with unchanged messages.
Gson/image shapes, EditLog payloads, BE code and thrift IDL are unchanged
(verified field-by-field for every `@SerializedName` bearer in the diff).
### Release note
None
### Check List (For Author)
- Test
- [x] Unit test: ~150 new golden parity tests (legacy classes as
oracles) covering typed-property conversion, bind routing, thrift/PB wire
structs, persisted S3Resource maps, URI normalization and credential
resolution; full fe-core UT suite run.
- Behavior changed:
- [x] No.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
