tlfua opened a new issue, #7452:
URL: https://github.com/apache/datafusion-comet/issues/7452
### Describe the bug
`lpad` and `rpad` fail natively when the length argument evaluates to a NULL
scalar. The serde passes any `len` expression through, and a scalar subquery
that returns no rows evaluates to `ScalarValue::Int32(None)` on the native side
(`subquery.rs`). The `read_side_padding` kernel only matches
`Scalar(Int32(Some(_)))` for a scalar length, so the call falls through to the
error arm:
```
Unsupported arguments [Array(StringArray
[
"ab",
]), Scalar(Int32(NULL)), Scalar(Utf8("x"))] for function
rpad/lpad/read_side_padding
```
A literal `NULL` length doesn't hit this, because Spark's optimizer folds
the whole call to `NULL` first.
### Steps to reproduce
```scala
Seq((1, "ab"), (2, "cd")).toDF("id",
"s").repartition(1).write.parquet("/tmp/strs")
spark.read.parquet("/tmp/strs").createOrReplaceTempView("strs")
sql("SELECT id, rpad(s, (SELECT max(id) FROM strs WHERE id > 100), 'x') FROM
strs").collect()
```
`lpad`, and the two-argument form `rpad(s, <null subquery>)`, should behave
the same way.
I reproduced this at the kernel level, calling `spark_rpad` with
`[Array(["ab"]), Scalar(Int32(None)), Scalar(Utf8("x"))]` on `main` at
644a87088. I haven't run the SQL above end to end; it goes through the same
scalar-subquery path as #7245.
### Expected behavior
Spark's `StringRPad`/`StringLPad` are null-intolerant, so every row is
`NULL`.
### Additional context
Found while working on #7245 (PR #7450), which handles a scalar *string*
argument. A NULL *length* array is already handled; only the NULL scalar length
is missing. The fix is probably a new arm that returns an all-NULL result,
sized to the string argument, when the length is `Scalar(Int32(None))`.
### Willingness to contribute
I can contribute a fix for this bug independently
AI disclosure: this issue was investigated and written with an AI coding
assistant, and reviewed by me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
