andygrove opened a new issue, #6958: URL: https://github.com/apache/datafusion-comet/issues/6958
### What is the problem the feature request solves? The native writer opens only `file`, `memory`, `s3`, `s3a` and `gs` data locations, and a `gs` location only through `GCSFileIO`. These fall back to iceberg-java: - **Azure: `abfs`, `abfss`, `wasb` and `wasbs`.** `native/Cargo.toml` already builds `iceberg-storage-opendal` with `opendal-azdls`, whose resolver accepts all four schemes, and iceberg-rust's `OpenDalStorageFactory::Azdls` reads `adls.*` properties, but `builtin_storage_schemes` lists none of them for reads or writes. #6058 covers the scan. - **`oss`** is read-only in `builtin_storage_schemes`, because no `oss.*` property is forwarded and no test covers a write. - **S3-compatible scheme aliases** opted in through `fs.comet.s3Compliant.schemes` are scan-only. The write path does not forward the opt-in. - **`gs` through `HadoopFileIO`** (#5637, fixed by falling back). `HadoopFileIO` takes its GCS settings from `fs.gs.*` in the Hadoop configuration, and only `fs.s3a.*` is translated for the native writer. ### Describe the potential solution - Azure: admit the schemes for writes once the property translation from #6058 exists. iceberg-java's `ADLSFileIO` keys (`adls.sas-token.<account>`, `adls.connection-string.<account>`, `adls.auth.shared-key.account.name` / `.key`, `adls.token`) differ from iceberg-rust's `adls.*` keys, and `HadoopFileIO` needs `fs.azure.*` translated. Admit only the authentication modes the translation covers. - `oss`: map iceberg-java's Aliyun `OSSFileIO` properties (`oss.endpoint`, `client.access-key-id`, `client.access-key-secret`) to iceberg-rust's `oss.endpoint`, `oss.access-key-id` and `oss.access-key-secret`, fall back on the rest (for example `client.security-token`), and add a write test. - Aliases: forward `fs.comet.s3Compliant.schemes` into the write's catalog properties as the scan does, and admit the opted-in schemes in `requireSupportedStorageScheme`. - `gs` through `HadoopFileIO`: translate the GCS connector settings that have an iceberg-rust equivalent (`fs.gs.project.id`, `fs.gs.storage.root.url`, an `fs.gs.auth.type` of application default or unauthenticated, a service account JSON key file, requester pays), and keep falling back on the rest, such as an access token provider class or service account impersonation. Each needs a `CometIcebergWriteDetectionSuite` case and a write through both writers, against an emulator such as Azurite or fake-gcs-server where CI can run one. ### Additional context Part of #5643 and the native write epic #5649. `hdfs` is #5894, which covers the write gate too and waits on apache/iceberg-rust#3111. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
