byronwang123 commented on code in PR #7636:
URL: https://github.com/apache/kyuubi/pull/7636#discussion_r3805822952
##########
externals/kyuubi-spark-sql-engine/pom.xml:
##########
@@ -405,6 +430,17 @@
<pattern>net.bytebuddy</pattern>
<shadedPattern>${kyuubi.shade.packageName}.net.bytebuddy</shadedPattern>
</relocation>
+ <!--
+ Relocate the bundled arrow-compression classes so
that Kyuubi's copy
+ never conflicts with the arrow-compression jar
shipped by Spark 4.1+.
+ The relocated codec classes still resolve the
unrelocated
+ org.apache.arrow.vector.* and
com.github.luben.zstd.* types, which are
Review Comment:
I checked this further. With the current implementation, the none path still
references CommonsCompressionFactory in the slice path, so simply removing the
bundled arrow-compression would cause a runtime class-not-found issue on Spark
3.5/4.0.
One alternative is to keep the original one-argument
VectorLoader/VectorUnloader for none, and only use the compression-aware path
for zstd. That would make arrow-compression truly optional on the engine side
and avoid the half-shading.
Do you prefer this approach over keeping the relocated bundled dependency?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
