sunchao commented on code in PR #5538:
URL: https://github.com/apache/datafusion-comet/pull/5538#discussion_r4125143461
##########
spark/pom.xml:
##########
@@ -168,6 +168,25 @@ under the License.
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-c-data</artifactId>
</dependency>
+ <!-- Direct native broadcast IPC uses the same codec-prefixed formats as
native shuffle. -->
+ <dependency>
+ <groupId>org.lz4</groupId>
Review Comment:
[P2] Select the LZ4 coordinates per Spark profile. With `-Pspark-4.2`, Spark
supplies `at.yawk.lz4:lz4-java:1.11.0`. This unconditional dependency adds
`org.lz4:lz4-java:1.8.0` alongside it. Maven retains both because their group
IDs differ, but they contain differing `net.jpountz.*` classes. Spark 4.2
should continue to build with direct reads disabled, yet root-reactor
`validate` now fails at `BanDuplicateClasses`, blocking builds and tests for
that profile regardless of the feature flag. Please use Spark 4.2's LZ4 fork
for that profile and retain the older coordinates only for profiles that
require them.
Evidence: Reproduced on the unchanged head with Maven 3.9.6/JDK 17 using
root-reactor `mvn -Pspark-4.2 validate`. `comet-spark-spark4.2_2.13` fails with
`Rule 2: org.codehaus.mojo.extraenforcer.dependencies.BanDuplicateClasses`,
naming both LZ4 artifacts and duplicates including `LZ4FrameOutputStream.class`
and `LZ4Factory.class`. An isolated reproduction using the published jars and
the repository's exact rule also fails. Its control containing only
`at.yawk.lz4:lz4-java:1.11.0` passes. Spark v4.2.0's root and core POMs confirm
the fork dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
