hanke580 opened a new issue, #27503:
URL: https://github.com/apache/datafusion/issues/27503
### Describe the bug
`array_slice` (and `list_slice`, and the `[from:to:stride]` syntax) panics
when the 0-based start
index plus the stride exceeds the maximum of the list's offset type. For
`List` that is
`i32::MAX`, and a stride of `2147483647` already triggers it. For
`LargeList` it is `i64::MAX`.
`compute_slice_plan` (`datafusion/functions-nested/src/extract.rs`, lines
549-553 on `main` `1841fb6e`;
577-583 in 55.2.0) builds the strided indices in the offset type `O` with an
unchecked addition:
```rust
let mut index = from;
while index <= to {
indices.push(index);
index += stride;
}
```
`1 + 2147483647` wraps to `-2147483648` in a release build. That value still
satisfies
`index <= to`, so a negative relative index is pushed. `general_array_slice`
then panics on
`(start + rel_index).to_usize().unwrap()` (line 655 on `main`, 682 in
55.2.0). `datafusion-cli` built from `main`:
```
thread 'main' (764241) panicked at
datafusion/functions-nested/src/extract.rs:655:73:
called `Option::unwrap()` on a `None` value
```
`general_list_view_array_slice` has the same `unwrap` (line 744 on `main`).
### To Reproduce
```sql
SELECT array_slice(make_array(1, 2, 3, 4, 5), 2, 2, 2147483647);
-- panic: called `Option::unwrap()` on a `None` value
SELECT make_array(1, 2, 3, 4, 5)[2:2:2147483647];
-- panic
SELECT list_slice(['h', 'e', 'l', 'l', 'o'], 2, 4, 2147483647);
-- panic
SELECT array_slice(arrow_cast(make_array(1, 2, 3, 4, 5),
'LargeList(Int64)'), 2, 2, 9223372036854775807);
-- panic
SELECT array_slice(a, 2, 2, s) FROM (VALUES (make_array(1, 2, 3, 4, 5),
2147483647)) t(a, s);
-- panic
```
For comparison:
```sql
SELECT array_slice(make_array(1, 2, 3, 4, 5), 2, 2, 2147483646); -- [2]
SELECT array_slice(make_array(1, 2, 3, 4, 5), 1, 5, 2147483647); -- [1]
(0 + stride does not overflow)
SELECT array_slice(make_array(1, 2, 3, 4, 5), 2, 2, 2147483648);
-- Internal error: array_slice got invalid stride: 2147483648.
-- This issue was likely caused by a bug in DataFusion's code. ...
```
### Expected behavior
`[2]` (and `[2]` for the other statements, `['e']` for the `list_slice`
one): a stride past the
end of the list keeps only the first element. That is the result for stride
`2147483646`, and it is
DuckDB's result for all of these statements. DataFusion's slice semantics
follow DuckDB, and DuckDB
returns `[2]` for `list_slice([1,2,3,4,5], 2, 2, 2147483647)`, for the same
call with
`9223372036854775807`, and for `[1,2,3,4,5][2:2:2147483647]`.
Computing the next index with `checked_add` (stop when it overflows), or in
`i64`/`i128`, would fix
the panic. Clamping the stride to the list length would also remove the
`Internal error` for
strides that do not fit in `i32` on a `List`. The signature accepts any
`Int64` stride.
### Additional context
- Version: `main` at `1841fb6e` (`datafusion-cli` 57.0.0-dev, `SELECT
version()` = `Apache DataFusion
57.0.0-dev, x86_64 on linux`), built with `cargo build --release -p
datafusion-cli`. Also DataFusion 55.2.0, the
latest release, through the Python bindings `datafusion` 55.0.0, where the
panic surfaces as
`pyo3_runtime.PanicException`.
- The negative-stride loop cannot overflow, because it requires `index >= to
>= 0`.
- Found with a width-boundary test generator that tries integer arguments at
and around 2^31 and
2^63.
### Willingness to contribute
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]