viirya commented on code in PR #6480:
URL: https://github.com/apache/datafusion-comet/pull/6480#discussion_r4150084603
##########
spark/src/main/scala/org/apache/comet/serde/strings.scala:
##########
@@ -218,7 +218,78 @@ object CometStringReplace
object CometSubstring extends CometScalarFunction[Substring]("substring")
-object CometSubstringIndex extends CometExpressionSerde[SubstringIndex] {
+/**
+ * Support level for native string kernels that search, trim or compare
strings by raw bytes
+ * (`instr`, `substring_index`, `trim` with a trim string, `greatest`,
`least`). They ignore a
+ * non-UTF8_BINARY collation, for example UTF8_LCASE where 'a' equals 'A', and
give wrong answers.
+ * Reporting Incompatible routes the expression through the JVM codegen
dispatcher, which runs
+ * Spark's collation-aware implementation.
+ */
+private[serde] object StringCollationSupport extends CometTypeShim {
+ def collationReason(name: String): String =
+ "Spark evaluates non-UTF8_BINARY collated string input under its
collation, while " +
+ s"Comet's native $name compares raw bytes"
+
+ def getSupportLevel(name: String, operands: Seq[Expression]): SupportLevel =
+ if (operands.exists(op => hasNonDefaultStringCollation(op.dataType))) {
+ Incompatible(Some(collationReason(name)))
+ } else {
+ Compatible()
+ }
+}
+
+object CometStringInstr
+ extends CometScalarFunction[StringInstr]("instr")
+ with CodegenDispatchFallback {
Review Comment:
Done in 98d902308. I regenerated the column with GenerateDocs to confirm
these seven rows now classify as Hybrid, updated them, and added notes that
collated input (with a trim string, for the trim family) routes through the
codegen dispatcher.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
