SimonC-Audigent commented on issue #1451: URL: https://github.com/apache/datafusion-comet/issues/1451#issuecomment-2687398695
> > I am getting the following error when running a job on EMR 7.6 with spark code also compiled with version 3.5.3 and using comet: 3.5_2.12-0.6.0 > > Did you compile Comet against EMR's proprietary fork of Spark or Apache Spark? It looks like EMR does not provide ShuffleExchangeLike, but I can't say for sure because I do not have access to the EMR source code. > > We have only tested Comet against Apache Spark. I have compiled my project with Spark 3.5.3 the open source one (but the deps are defined as provided as the jars are already installed on the VMs). I am not aware of AWS having using proprietary spark code on EMR (unlike AWS Glue which use proprietary code), in fact if it was the case I imagine I would have faced a lot more issue in the past. I do believe they use the open source one, they specify what version of which lib is installed in their release notes: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-760-release.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
