dwsmith1983 opened a new issue, #5662: URL: https://github.com/apache/datafusion-comet/issues/5662
### Describe the bug Raised by @viirya in the #5365 review. `NativeConfig.extractObjectStoreOptions` forwards `fs.s3a.*` options to the native object store client with no divergence check, but Comet's native client resolves S3 configuration differently from Hadoop's S3AFileSystem in several ways (lookupPassword bucket precedence, JCEKS credential provider aliases, clear text fallback, assumed role session policies, provider class semantics). A plain Parquet table read through the regular native scan can therefore read under a different effective identity or endpoint than Spark would, silently. The Delta contrib in #5365 carries a divergence comparator that detects these cases and declines to Spark, but the regular native scan path has no equivalent gate. #5658 tracks extracting that comparator into a shared helper; this issue tracks actually applying it (or an equivalent check) to the regular native scan so the gap is closed for every native read, not only Delta. ### Steps to reproduce Configure any setting the two resolvers treat differently, for example a JCEKS-backed fs.s3a.secret.key with hadoop.security.credential.clear-text-fallback=false plus a clear text value, and read a plain Parquet table with the native scan enabled. ### Expected behavior The scan declines to Spark when native would resolve different S3 configuration than Hadoop, or resolves identically. ### Additional context No response -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
