sunchao commented on code in PR #7026:
URL: https://github.com/apache/datafusion-comet/pull/7026#discussion_r4331657673
##########
spark/src/main/scala/org/apache/comet/objectstore/NativeConfig.scala:
##########
@@ -428,6 +444,12 @@ object NativeConfig {
"Native scan does not support ABFS files from more than one
container or account " +
s"(${first.getRawAuthority} and ${uri.getRawAuthority})")
}
+ collectS3(uri)
+ }
+ val s3Reason =
+ if (s3Uris.isEmpty) None else S3ConfigGate.declineReason(hadoopConf,
s3Uris.values.toSeq)
Review Comment:
[P2] Preserve supported SPI-backed scans when applying the gate here.
Configure
`fs.s3a.comet.credential.provider.class=org.apache.comet.cloud.s3.HadoopS3ACredentialProviderAdapter`,
`fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider`,
and credentials only under `fs.s3a.bucket.<bucket>.fs.s3a.access.key` and
`.secret.key`. The real Comet adapter resolves these long-form aliases exactly
as Hadoop does, so the existing native scan can use the same credentials as
Spark. However, `s3KeyDivergenceReason` still compares against
`nativeShortThenGlobal`, which cannot see those aliases, and this new
invocation returns `Left`, forcing regular Parquet/CSV scans back to Spark.
This disables native scanning for a previously supported configuration. Model
the active SPI credential-resolution path in the equivalence check while
retaining the store-level checks, and add an adapter-backed regression test.
Evidence: An offline probe compiled the exact-head `S3ConfigGate`,
`HadoopS3ACredentialProviderAdapter` and supporting classes, then compared
adapter credentials with the real Hadoop `SimpleAWSCredentialsProvider` after
`S3AUtils.propagateBucketOptions`. With synthetic global credentials, both
providers matched and the gate returned `None`. With the same synthetic
credentials stored only in long-form bucket aliases, both providers still
matched, but the gate returned `Some(Native scan cannot forward
fs.s3a.access.key for review-test-bucket ...)`. The probe exited successfully
after asserting credential equality. Source:
`/tmp/comet7026-review/AdapterGateProbe.scala`; output:
`/tmp/comet7026-review/probe.log`. Native `s3.rs::create_store` selects the SPI
bridge instead of `build_credential_provider` when this adapter is configured.
The base version of `extractScanObjectStoreOptions` did not invoke the gate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
