sunchao commented on code in PR #5416:
URL: https://github.com/apache/datafusion-comet/pull/5416#discussion_r4125964347
##########
native/spark-expr/src/string_funcs/split.rs:
##########
@@ -68,6 +67,11 @@ pub fn spark_split(
-1
};
+ // Spark's UTF8String.split remaps limit == 0 to -1 before calling Java's
+ // String.split, specifically to avoid Java's "drop trailing empty strings"
+ // behavior. Normalize here so every helper below shares Spark semantics.
+ let limit = if limit == 0 { -1 } else { limit };
Review Comment:
[P2] Preserve the single empty field when normalizing a zero limit. With
`spark.comet.expression.StringSplit.allowIncompatible=true`, a persisted row
containing `s = ''` and `SELECT split(s, '', 0) FROM t` previously returned
`[""]`, matching Spark, but now returns `["", ""]`. Patterns `'^'` and `'.*'`
reproduce the same regression. Normalization exposes the Rust iterators'
empty-input boundary matches without Spark's handling, changing array
cardinality and causing downstream `explode` to emit an extra row. Please
retain the trailing-empty correction while emitting one empty field for empty
input after pattern validation, with scalar and column regression coverage.
Evidence: A fresh harness compiled the unchanged exact-base and exact-head
split/cache modules against locked Arrow 59.3.0, DataFusion 55.1.0, and regex
1.13.1. Calling `spark_split` with an empty Utf8 scalar, pattern `''`, and
Int32 limit `0` returned one empty element at base and two at head. A nullable
StringArray reproduced the same change for its empty row. Scalar patterns `'^'`
and `'.*'` also changed from one element to two. Spark's
`RegexpExpressionsSuite` explicitly expects `Seq("")` for
`StringSplit(Literal(""), Literal(""), 0)` across all inspected versions.
Remote-tag verification confirmed the reference UTF8String sources, and a Java
probe of their delegated `"".split(pattern, -1)` operation returned one element
for each pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
