andygrove opened a new issue, #414: URL: https://github.com/apache/datafusion-comet/issues/414
### What is the problem the feature request solves? We have our first PR up that works around an issue with Comet working with AWS Spark (https://github.com/apache/datafusion-comet/pull/412). I think we need to carefully consider our stance on supporting closed-source forks of Spark from the cloud service providers. Supporting closed-source Spark versions is challenging for many reasons: - There are often custom operators and expressions that Comet will not be aware of, and, therefore will not be able to map to native versions - Software updates can be pushed out at any time, potentially breaking compatibility with Comet - It is difficult to debug issues because source is not available (and reverse engineering is prohibited) - We would ideally need CI to test against all supported CSP versions - It increases the maintenance burden for all contributors, and not all contributors will have access to the CSP versions for testing If the community desires to maintain Comet versions that can work with CSP Spark versions, then I think we would need to find an approach that allows those contributors to extend the "core" Comet project and add CSP support without adding maintenance burden for the core project. One idea, for example, would be to keep the core `datafusion-comet` project compatible with OSS Apache Spark, and then have specific downstream repositories such as `datafusion-comet-aws` that extend the project to support a specific CSP. ### Describe the potential solution _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
