parthchandra commented on PR #1992: URL: https://github.com/apache/datafusion-comet/pull/1992#issuecomment-3059721586
Update on this: 1) fs-hdfs includes its [own version of libhdfs](https://github.com/datafusion-contrib/fs-hdfs/tree/hadoop-3.1.4/c_src/libhdfs) as part of its build so we do not need to worry about that 2) fs-hdfs recognizes only [file, hdfs, and viewfs](https://github.com/datafusion-contrib/fs-hdfs/blob/8c03c5ef0942b75abc79ed673931355fa9552131/src/hdfs.rs#L841) as schemes. This is not a showstopper as it is easily addressed. With the above addressed and with appropriate modifications to Comet to always use the hdfs object store, I was able to access an `s3a` uri. The read failed but it may be unrelated (I will investigate). I'll verify with a custom credentials provider as well. So it appears that using the `datafusion-comet-objectstore-hdfs` object store is a viable path forward. @drexler-sky I will agree with @comphead and @Kontinuation so if you want to go ahead with your proposed plan, please go ahead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
