byronwang123 commented on code in PR #7636:
URL: https://github.com/apache/kyuubi/pull/7636#discussion_r3804900691
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/sql/execution/arrow/KyuubiArrowConverters.scala:
##########
@@ -39,6 +41,28 @@ object KyuubiArrowConverters extends SQLConfHelper with
Logging {
type Batch = (Array[Byte], Long)
+ /**
+ * Create the Arrow compression codec for the given codec name. Returns the
no-compression codec
+ * when the codec name is "none" or null.
+ *
+ * The zstd codec is constructed directly with the configured compression
level rather than
+ * through a [[CompressionCodec.Factory]]: the factory overloads built from
the codec type enum
+ * cannot carry a compression level and silently fall back to the default
one, losing the
+ * user-configured level. This follows the latest Spark upstream behavior
+ * (ArrowCompressionUtils).
+ */
+ private def createCodec(codecName: String, zstdLevel: Int): CompressionCodec
= {
+ if (codecName == null || codecName == "none") {
+ NoCompressionCodec.INSTANCE
+ } else {
+ codecName match {
+ case "zstd" => new ZstdCompressionCodec(zstdLevel)
+ case other =>
Review Comment:
Addressed. Merged the branches and made unsupported codecs fail with a clear
error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
