morrySnow commented on code in PR #66315:
URL: https://github.com/apache/doris/pull/66315#discussion_r3773645431
##########
fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java:
##########
@@ -82,8 +82,46 @@ public class ConnectorPluginManager {
// Connector SPI and filesystem SPI classes must be parent-first so that
all
// instances of shared interfaces/classes are loaded by a single
ClassLoader.
- private static final List<String> CONNECTOR_PARENT_FIRST_PREFIXES =
- Arrays.asList("org.apache.doris.connector.",
"org.apache.doris.filesystem.");
+ //
+ // org.apache.hadoop. is parent-first so that the Doris-patched
org.apache.hadoop.fs.FileSystem
+ // (hadoop-deps; its Cache.Key carries doris.fs.cache.key.<scheme>)
reaches EVERY connector
+ // plugin, including a third-party or previous-release one that bundles
vanilla hadoop-common.
+ // FE no longer sends the blanket fs.<scheme>.impl.disable.cache=true, and
could not send it
+ // selectively either -- storage property maps are built before anyone
knows which plugin
+ // consumes them -- so a plugin on an unpatched FileSystem would silently
hand one cached client
+ // to catalogs that differ only in credentials.
+ //
+ // The prefix covers the namespace, not just that one class, because the
JVM requires it: a
+ // plugin-loaded subclass such as
org.apache.hadoop.hdfs.DistributedFileSystem overrides
+ // FileSystem.initialize(URI, Configuration), and both loaders must then
resolve Configuration to
+ // the same Class or startup dies with "loader constraint violation".
+ //
+ // Parent-first is a delegation ORDER, not an exclusive claim:
ChildFirstClassLoader falls back
+ // to the plugin's own jars for anything the parent lacks. So
org.apache.hadoop.hbase.* (hudi)
+ // and org.apache.hadoop.hive.* still come from the plugin -- FE carries
hive-exec:core, the
+ // plugins carry hive-metastore, and the class names do not intersect.
Everything else the
+ // plugins bundle under this namespace does change provider:
hadoop-common/auth/annotations/
+ // hdfs-client/aws, hadoop-shaded-guava and -protobuf, and the huaweicloud
fs.obs.* classes
+ // (paimon), which the FE kernel ships too. All of them are the same
artifact at the same
+ // version on both sides, and both versions are pinned in fe/pom.xml --
hadoop.version is
+ // additionally held by the maven-enforcer rule in
be-java-extensions/hadoop-deps, huaweiobs
+ // .version only by dependencyManagement, so bumping either for one side
alone silently hands
+ // the plugins the kernel's copy.
+ //
+ // The static state hanging off these now-shared classes matters as much
as the classes: see the
+ // DORIS-PATCH in hadoop-deps' FileSystem.loadFileSystems, which binds the
ServiceLoader scan to
+ // the class's own loader so that FileSystem.SERVICE_FILE_SYSTEMS -- a
process-wide, first-caller
+ // -wins registry -- cannot be frozen by whichever plugin's context loader
happens to touch it
+ // first.
+ //
+ // NOTE: the intended end state is an FE kernel with no hadoop classes at
all, every plugin
+ // bringing its own. At that point the fallback above takes over on its
own, and the plugin
+ // becomes responsible for shipping a patched FileSystem the same way the
kernel does today.
+ //
+ // Package-private so ConnectorPluginHadoopPatchTest asserts against this
list, not a copy of it.
+ static final List<String> CONNECTOR_PARENT_FIRST_PREFIXES =
+ Arrays.asList("org.apache.doris.connector.",
"org.apache.doris.filesystem.",
+ "org.apache.hadoop.");
Review Comment:
Adding `org.apache.hadoop.` to the parent-first allowlist silently overrides
the connector plugins' carefully-engineered **child-first** hadoop bundling.
The four connector plugins bundle their own hadoop artifacts (hadoop-common
/ hadoop-hdfs-client / hadoop-aws / hadoop-huaweicloud) child-first on purpose,
and this is stated explicitly in their poms — e.g.
`fe-connector-paimon/pom.xml`: "The plugin runs child-first (org.apache.hadoop
is not parent-first)". `build.sh`'s RC-4 block (lines ~1219-1233) copies the
jindofs jars into the paimon plugin lib *because* "The plugin runs child-first,
so without its OWN copy JindoOssFileSystem resolves from the parent 'app'
classloader and cannot be cast to the plugin's child-loaded
org.apache.hadoop.fs.FileSystem". Making the whole namespace parent-first
shadows every one of those bundled jars with the FE kernel's copy, so those
comments are now wrong.
Because the delegation order still falls back to the child for anything the
parent lacks, the practical risk is confined to artifacts the kernel ships at a
*different* revision than the plugin bundles. The clearest example is paimon:
it bundles the `hadoop-huaweicloud-hw-46` fat jar that self-contains `esdk-obs`
(`com/obs/*`), while the kernel ships `${huaweiobs.version}` hadoop-huaweicloud
plus a separate `esdk-obs-java-bundle`. Under parent-first, paimon's OBS
(`fs.obs.*`) classes now resolve to the kernel's copies. This also applies to
any third-party or previous-release connector bundling a non-3.4.2 hadoop.
Could you confirm each bundled connector resolves every hadoop class to the
identical artifact/version on both sides, and update the now-stale child-first
comments in the connector poms and in `build.sh`'s RC-4 block?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
