pzhdfy commented on code in PR #69493:
URL: https://github.com/apache/doris/pull/69493#discussion_r4360683823
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/AbstractPaimonProperties.java:
##########
@@ -123,6 +124,34 @@ protected void appendUserHadoopConfig(Configuration conf)
{
normalizeS3Config().forEach(conf::set);
}
+ /**
+ * Mirrors the S3 credentials Doris assembled into the Hadoop configuration
+ * onto the catalog options, so the pinned Paimon 2.1 SDK sees them.
+ *
+ * <p>Doris stores a catalog's storage credentials (fs.s3a.access.key,
fs.s3a.secret.key,
+ * fs.s3a.endpoint, ...) only in the Hadoop Configuration handed to
+ * CatalogContext.create(options, conf) — the paimon 1.4 FileIO consulted
exactly
+ * that conf. The pinned 2.1 S3FileIO instead reads its credentials
exclusively from
+ * CatalogContext.options() (S3FileIO#configure ->
loadHadoopConfigFromContext copies
+ * the s3./s3a./fs.s3a.-prefixed option entries and never looks at the
conf), so
+ * without this mirror every s3-warehouse catalog initializes the SDK with
no
+ * credentials and the first file operation fails on the ambient AWS
provider chain
+ * ("Unable to load credentials from system settings"). Mirroring the
whole fs.s3a.*
+ * set keeps user extras (path style, connection limits) flowing as well;
entries
+ * outside the prefix are not the SDK's concern.
+ */
+ protected void mirrorHadoopS3OptionsToCatalogOptions(Configuration conf) {
+ if (conf == null || catalogOptions == null) {
+ return;
+ }
+ conf.forEach(entry -> {
+ String key = entry.getKey();
+ if (key.startsWith(HADOOP_S3A_PREFIX) &&
StringUtils.isNotBlank(entry.getValue())) {
+ catalogOptions.set(key, entry.getValue());
Review Comment:
Confirmed and fixed in af5c7cc: the Hadoop `fs.s3a.*` mirror now fills only
options the catalog has not already set, so an explicit assumed-role
credentials provider (set via a `paimon.fs.s3a.*` property) keeps precedence
over the SimpleAWSCredentialsProvider the generated Hadoop configuration
carries; absent defaults are still filled. Covered by
`testHadoopS3MirrorDoesNotOverwriteExplicitProvider` (13/13 properties UTs
green). This affects the filesystem, HMS and JDBC paimon catalog variants
alike.
### Carried-over notes
- The per-level `file.format.per.level='0:orc'` residual (a Parquet-default
table with level-0 ORC files carrying LTZ escaping the schema-history
check) is acknowledged as the remaining known edge of the ORC-era guard;
it joins the stored-NULL item as a documented limitation with the same
follow-up shape (parse the per-level option in the pre-flight).
- All other round-2 findings (nanosecond pre-flight, reserved materializer
names, distinct-distance exemption, redundant heartbeat bit) were
confirmed addressed by this round and are closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
