kazuyukitanimura commented on code in PR #7026: URL: https://github.com/apache/datafusion-comet/pull/7026#discussion_r4339685037
########## docs/source/user-guide/latest/datasources.md: ########## @@ -424,7 +424,28 @@ Comet's S3 support has the following limitations: 2. **Custom credential providers**: Custom credential provider classes named in `fs.s3a.aws.credentials.provider` are not supported; only the standard providers listed in the table above are. To route credential requests through your own Java code, implement Comet's `CometS3CredentialProvider` SPI; see [S3 Credential Providers](s3-credential-providers.md). Broader Hadoop S3A integration is tracked in [#1829](https://github.com/apache/datafusion-comet/issues/1829). -3. **Settings that Hadoop's S3A client resolves differently**: The native scans resolve storage settings with their own client rather than with Hadoop's. Where the two disagree, a native read can use a different identity or route than Spark and fail where Spark succeeds. Examples are JCEKS-backed secrets and the precedence of per-bucket settings, which the native Parquet scan forwards without checking for a difference ([#5662](https://github.com/apache/datafusion-comet/issues/5662)). +3. **S3 settings that Hadoop and Comet resolve differently**: Comet does not run Hadoop's `S3AFileSystem`, so some + settings take effect differently in the native client than they do for Spark's reader. To keep a scan from + silently reading under a different identity or endpoint than Spark would, Comet declines the scan, and Spark Review Comment: Limitations item 3 now names the native Parquet, CSV and Delta scans and notes the libhdfs and SPI exemptions. It also says the Iceberg scan, native writes and S3-compliant aliases are still unchecked, with a link back to #5662. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
