hanke580 opened a new issue, #27503:
URL: https://github.com/apache/datafusion/issues/27503

   ### Describe the bug
   
   
   `array_slice` (and `list_slice`, and the `[from:to:stride]` syntax) panics 
when the 0-based start
   index plus the stride exceeds the maximum of the list's offset type. For 
`List` that is
   `i32::MAX`, and a stride of `2147483647` already triggers it. For 
`LargeList` it is `i64::MAX`.
   
   `compute_slice_plan` (`datafusion/functions-nested/src/extract.rs`, lines 
549-553 on `main` `1841fb6e`;
   577-583 in 55.2.0) builds the strided indices in the offset type `O` with an 
unchecked addition:
   
   ```rust
   let mut index = from;
   while index <= to {
       indices.push(index);
       index += stride;
   }
   ```
   
   `1 + 2147483647` wraps to `-2147483648` in a release build. That value still 
satisfies
   `index <= to`, so a negative relative index is pushed. `general_array_slice` 
then panics on
   `(start + rel_index).to_usize().unwrap()` (line 655 on `main`, 682 in 
55.2.0). `datafusion-cli` built from `main`:
   
   ```
   thread 'main' (764241) panicked at 
datafusion/functions-nested/src/extract.rs:655:73:
   called `Option::unwrap()` on a `None` value
   ```
   
   `general_list_view_array_slice` has the same `unwrap` (line 744 on `main`).
   
   ### To Reproduce
   
   ```sql
   SELECT array_slice(make_array(1, 2, 3, 4, 5), 2, 2, 2147483647);
   -- panic: called `Option::unwrap()` on a `None` value
   
   SELECT make_array(1, 2, 3, 4, 5)[2:2:2147483647];
   -- panic
   
   SELECT list_slice(['h', 'e', 'l', 'l', 'o'], 2, 4, 2147483647);
   -- panic
   
   SELECT array_slice(arrow_cast(make_array(1, 2, 3, 4, 5), 
'LargeList(Int64)'), 2, 2, 9223372036854775807);
   -- panic
   
   SELECT array_slice(a, 2, 2, s) FROM (VALUES (make_array(1, 2, 3, 4, 5), 
2147483647)) t(a, s);
   -- panic
   ```
   
   For comparison:
   
   ```sql
   SELECT array_slice(make_array(1, 2, 3, 4, 5), 2, 2, 2147483646);   -- [2]
   SELECT array_slice(make_array(1, 2, 3, 4, 5), 1, 5, 2147483647);   -- [1]  
(0 + stride does not overflow)
   SELECT array_slice(make_array(1, 2, 3, 4, 5), 2, 2, 2147483648);
   -- Internal error: array_slice got invalid stride: 2147483648.
   -- This issue was likely caused by a bug in DataFusion's code. ...
   ```
   
   ### Expected behavior
   
   
   `[2]` (and `[2]` for the other statements, `['e']` for the `list_slice` 
one): a stride past the
   end of the list keeps only the first element. That is the result for stride 
`2147483646`, and it is
   DuckDB's result for all of these statements. DataFusion's slice semantics 
follow DuckDB, and DuckDB
   returns `[2]` for `list_slice([1,2,3,4,5], 2, 2, 2147483647)`, for the same 
call with
   `9223372036854775807`, and for `[1,2,3,4,5][2:2:2147483647]`.
   
   Computing the next index with `checked_add` (stop when it overflows), or in 
`i64`/`i128`, would fix
   the panic. Clamping the stride to the list length would also remove the 
`Internal error` for
   strides that do not fit in `i32` on a `List`. The signature accepts any 
`Int64` stride.
   
   ### Additional context
   
   
   - Version: `main` at `1841fb6e` (`datafusion-cli` 57.0.0-dev, `SELECT 
version()` = `Apache DataFusion
     57.0.0-dev, x86_64 on linux`), built with `cargo build --release -p 
datafusion-cli`. Also DataFusion 55.2.0, the
     latest release, through the Python bindings `datafusion` 55.0.0, where the 
panic surfaces as
     `pyo3_runtime.PanicException`.
   - The negative-stride loop cannot overflow, because it requires `index >= to 
>= 0`.
   - Found with a width-boundary test generator that tries integer arguments at 
and around 2^31 and
     2^63.
   
   ### Willingness to contribute
   
   None


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to