andygrove opened a new pull request, #357:
URL: https://github.com/apache/datafusion-spark/pull/357
## Which issue does this PR close?
- Part of #353.
## Rationale for this change
Spark's `ArrayRepeat` takes an INT count. Here the count was coerced to
BIGINT, or to `UInt64` for an unsigned type, which DataFusion's `array_repeat`
then cannot read:
| Query | Before | This PR |
|---|---|---|
| `array_repeat('a', arrow_cast(2, 'UInt32'))` | `Internal error: could not
cast array of type UInt64 to ...Int64Type` | `[a, a]` |
| `array_repeat('a', 3000000000)` | `array_repeat: offset 3000000000 exceeds
the maximum value for offset type` | a cast error, since the count is outside
the INT range |
| `array_repeat('a', '2')` | `count must be an integer type` | ``
[DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] The second parameter of
`array_repeat` requires the "INT" type, however it has the type "STRING".
SQLSTATE: 42K09 `` |
## What changes are included in this PR?
- `coerce_types` casts an integer count of any type to INT, and rejects
other types with Spark's `UNEXPECTED_INPUT_TYPE` error.
- The count is cast back to BIGINT before calling DataFusion's
`array_repeat`, which reads a BIGINT count.
Spark rejects a TINYINT, SMALLINT or BIGINT count, rather than casting it.
DataFusion types a bare integer literal such as `2` as BIGINT, so rejecting
BIGINT would break `array_repeat('a', 2)`. This follows `array_insert` and
`stack`, which accept every integer type for their INT arguments for the same
reason. I wrote in #353 that `array_repeat` could use `SparkInputTypes` with
`[Any, Integer]`, but that would also cast strings and decimals, which Spark
rejects.
## Are these changes tested?
Yes. New records in `array/array_repeat.slt` cover each integer type, an
unsigned count, a column with a NULL count, a count outside the INT range, the
errors for other types, and ANSI mode. Most of them fail without this change. I
checked the file against Spark 4.2.0 with the PySpark runner. Every record
matches except the three TINYINT, SMALLINT and BIGINT counts, which Spark
rejects as described above. The runner does not compare error messages, so I
checked them against Spark by hand.
## Are there any user-facing changes?
`array_repeat` accepts an unsigned integer count, a count outside the INT
range is an error, and a count of another type fails with Spark's error.
This PR was prepared with AI assistance (Claude Code) and reviewed by me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]