sunchao commented on PR #5365: URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5322185992
Hi @dwsmith1983 Thanks for putting this together! We are also actively looking at Delta support for Comet, and it'd be great if we can collaborate on this effort! Since #4952 is already approved and close to landing, what do you think about using it as the shared foundation for this work? Ideally, the same `contrib` infrastructure could support both JVM-planned Delta scans and the Rust Kernel-based approach, with this PR providing the JVM-planned path. We have related work in progress, so it would be good to converge on one implementation. In addition, would it also make sense to land this in smaller pieces, for easier review and iterating? For example: - Basic native Delta reads, including time travel and fallback for unsupported features - Column mapping and schema evolution - Deletion vectors - Row tracking - Change Data Feed Starting to support this in Spark 4 & Delta 4 would be a useful first milestone. Curious how you see the relationship between the two PRs and whether that direction makes sense to you. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
