chrikrah opened a new issue, #55295:
URL: https://github.com/apache/arrow/issues/55295
`Dataset.head`, `Scanner.head` and `Fragment.head` accept a negative
`num_rows` and return a `Table` that fails `Table.validate()` with `Array
length is negative`, the check in `ValidateLayout`
(`cpp/src/arrow/array/validate.cc:477`). `len()` on it raises `SystemError`.
```python
import os, tempfile
import pyarrow as pa
import pyarrow.dataset as ds
import pyarrow.parquet as pq
print(pa.__version__)
d = tempfile.mkdtemp()
pq.write_table(pa.table({"a": [1, 2, 3]}), os.path.join(d, "x.parquet"))
dataset = ds.dataset(d)
t = dataset.head(-1)
print(t.num_rows)
try:
t.validate()
except pa.ArrowInvalid as e:
print(type(e).__name__, e)
try:
len(t)
except BaseException as e:
print(type(e).__name__, e)
print(dataset.scanner().head(-2).num_rows)
```
```
$ python -I head_negative.py # pyarrow 26.0.0 wheel, Python 3.12, Linux
x86_64
26.0.0
-1
ArrowInvalid Column 0: In chunk 0: Invalid: Array length is negative
SystemError <built-in function len> returned NULL without setting an
exception
-2
```
The R binding already rejects this: `head.Scanner` in `r/R/dataset-scan.R`
runs `assert_that(n >= 0)`, with the comment that a negative n would need
`nrow(x)` and so a full scan. That rules out the pandas reading of `head(-n)`.
On the same wheel `pa.array([1, 2, 3]).slice(0, -1)` raises `ValueError: Length
must be non-negative`.
The Python wrappers forward `num_rows` to `Scanner::Head`.
`AsyncScanner::Head` (`cpp/src/arrow/dataset/scanner.cc:595`, `9eb748d`) only
special-cases `0`, then calls `Slice(0, num_rows)` on each batch, so a negative
count yields batches of negative length. I read the source at `9eb748d` and ran
the 26.0.0 wheel, not a source build.
Expected: `ArrowInvalid` for `num_rows < 0`, raised in `AsyncScanner::Head`
so Python and C++ callers both get it. Since `ArrowInvalid` subclasses
`ValueError`, the Cython layer needs no separate check.
I can open a PR with that check and a test in `test_dataset.py`. Would you
take it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
