mixermt opened a new issue, #5894: URL: https://github.com/apache/datafusion-comet/issues/5894
### What is the problem the feature request solves? Comet's native Iceberg scan cannot read a table whose data lives on HDFS. `hdfs://` is not in the scheme allowlist that mirrors `storage_factory_for`, so any Iceberg table on HDFS falls back to the JVM reader and gets no native acceleration. This is a gap relative to the plain-Parquet native scan, which does read HDFS (through libhdfs/JNI, via `fs.comet.libhdfs.schemes`). On-premise Iceberg deployments are commonly HDFS-backed, so the Iceberg acceleration Comet offers is unavailable to them today. ### Describe the potential solution Route `hdfs://` to iceberg-rust's `hdfs-native` OpenDAL backend, a pure-Rust HDFS RPC client that needs no JNI or libhdfs. That backend is being added upstream in apache/iceberg-rust#3111; this issue tracks the Comet-side work once it lands. Comet-side work: - Add an `hdfs` arm to `storage_factory_for` and admit the scheme in the JVM gates (`CometScanRule.icebergReadableSchemes`, and `CometIcebergNativeWrite.SupportedStorageSchemes` for the write path). - Forward the `hdfs.`/`hadoop.` property prefixes to the native `FileIO`. - Resolve the NameNode endpoints for HA clusters. opendal's `HdfsNativeBuilder` never dials the authority written in the path: it builds one client against a synthetic authority and synthesizes the HA config from the comma-separated `hdfs.name-node` value. iceberg-rust falls back to the path authority only when that property is absent, which is correct just for a real `host:port`. An HA location reads `hdfs://<nameservice>/...`, and a nameservice is not a routable host, so the endpoints have to be derived from `dfs.ha.namenodes.<ns>` / `dfs.namenode.rpc-address.<ns>.<nn>` in the session Hadoop configuration. - Decline a scan whose data/delete files span more than one HDFS authority: one `hdfs.name-node` per scan overrides the authority of every path the FileIO opens, so a second nameservice would be read from the first one's NameNode at the same relative path. ### Additional context Note that this is a second, independent HDFS client in the same process: the plain-Parquet path reaches HDFS through libhdfs/JNI while an Iceberg table would be opened over pure-Rust RPC. The two read the same `$HADOOP_CONF_DIR` XML but hold separate connections and separate Kerberos state. Blocked on apache/iceberg-rust#3111. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
