dwsmith1983 commented on PR #5365: URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5389347576
> ### [P2] Also handle plain long-form S3 bucket credentials > The long-form JCEKS cases are fixed, but the same compatibility gap remains for **plain configuration values** at `514f71e0`. > > With `fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider` and only these plain keys configured: > > ``` > fs.s3a.bucket.mybucket.fs.s3a.access.key > fs.s3a.bucket.mybucket.fs.s3a.secret.key > ``` > > [`bucketCredentialAliasReason`](https://github.com/apache/datafusion-comet/blob/514f71e0d09f4b0e85964014afae17363fa2dba1/contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala#L712-L713) returns `None` because no credential-provider path is set. Hadoop resolves these keys, but `NativeConfig` forwards them unchanged and the [native lookup](https://github.com/apache/datafusion-comet/blob/514f71e0d09f4b0e85964014afae17363fa2dba1/native/core/src/parquet/objectstore/s3.rs#L299-L318) reads only short bucket keys and globals. The native Simple provider therefore has no credentials. If different plain global credentials are also configured, Hadoop uses the long bucket aliases while native uses those globals instead. Claiming the Delta scan exposes this mismatch to reads that previously used Hadoop successfully. > > I verified both cases with actual Hadoop 3.4.1 `SimpleAWSCredentialsProvider` and the unchanged current admission/extraction methods, before and after `S3AUtils.propagateBucketOptions`, plus exact-source Rust lookup tests. Short-only and global-only controls agree. This was not a full Comet/JNI scan or a live S3 request. > > Could we conservatively decline these configurations, or normalize the effective credentials using Hadoop's precedence, and add plain-long-only and global-shadowing regressions? > > This is a remaining case of the [existing S3-authentication P2](https://github.com/apache/datafusion-comet/pull/5365#discussion_r3835076674), posted here for visibility rather than as another inline thread. @sunchao Fixed at 6d20ad69f. The gate now also compares plain values: for each bucket it computes Hadoop's effective credentials (short overrides long, then long, then global) against what the native lookup would use (short then global) and declines whenever a long-form key is set and the two diverge. Pure conf-map reads, so still zero I/O. Added plain-long-only and global-shadowing regressions plus controls for short-overrides-long and long-equals-global. Also rebased onto latest main. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
