lmmx opened a new issue, #51639: URL: https://github.com/apache/arrow/issues/51639
### Describe the bug, including details regarding any error messages, version, and platform. I was developing a routine to 'compact' my parquet files down to some different row group sizes (etc) with PyArrow, and along the way used a `RecordBatch.cast` which failed. I got a coding agent (Claude Opus 5.5) to dig into this and bisect it down to a minimal repro and it identified the cause as follows. --- **Repro files:** [matrix.py](https://github.com/user-attachments/files/32807938/matrix.py) <details><summary>Details</summary> <p> ```py """Which casts fail, and the length of the null-typed child in each result. Run: python matrix.py (needs only pyarrow). Prints one line per case, ok or FAIL. """ import pyarrow as pa import pyarrow.compute as pc print("pyarrow", pa.__version__) s = pa.struct([("a", pa.int64()), ("n", pa.null())]) s_int32 = pa.struct([("a", pa.int32()), ("n", pa.null())]) s_no_null = pa.struct([("a", pa.int64()), ("n", pa.int64())]) L = pa.list_ cases = [ # Lists, not sliced: fails when the list has fewer rows than structs ("list [[x]]: 1 row, 1 struct", pa.array([[{"a": 1}]], L(s)), L(s)), ("list [[x, x]]: 1 row, 2 structs", pa.array([[{"a": 1}, {"a": 2}]], L(s)), L(s)), ("list [[x], [x]]: 2 rows, 2 structs", pa.array([[{"a": 1}], [{"a": 2}]], L(s)), L(s)), ("list [None, [x, x]]: 2 rows, 2 structs", pa.array([None, [{"a": 1}, {"a": 2}]], L(s)), L(s)), ("list [[x, x], [x]]: 2 rows, 3 structs", pa.array([[{"a": 1}, {"a": 2}], [{"a": 3}]], L(s)), L(s)), ("large_list [[x, x]]", pa.array([[{"a": 1}, {"a": 2}]], pa.large_list(s)), pa.large_list(s)), ("list_view [[x, x]]", pa.array([[{"a": 1}, {"a": 2}]], pa.list_view(s)), pa.list_view(s)), ("list [[x, x]] with n: int64 (control)", pa.array([[{"a": 1}, {"a": 2}]], L(s_no_null)), L(s_no_null)), # Casts to another type ("list [[x, x]] -> large_list<same struct>", pa.array([[{"a": 1}, {"a": 2}]], L(s)), pa.large_list(s)), ("list [[x, x]] -> list<struct<a: int32, n: null>>", pa.array([[{"a": 1}, {"a": 2}]], L(s)), L(s_int32)), # Structs, not in a list ("struct, 2 rows, not sliced", pa.array([{"a": 0}, {"a": 1}], s), s), ("struct, 2 rows, slice(1)", pa.array([{"a": 0}, {"a": 1}], s).slice(1), s), ("struct, 2 rows, slice(1) -> struct<a: int32, n: null>", pa.array([{"a": 0}, {"a": 1}], s).slice(1), s_int32), ] for label, arr, to in cases: arr.validate(full=True) out = pc.cast(arr, to) try: out.validate(full=True) print(f"ok {label}") except pa.ArrowInvalid as e: print(f"FAIL {label}: input length {len(arr)}, {e}") # Array.cast and ChunkedArray.cast give the same invalid result, also without raising arr = pa.array([[{"a": 1}, {"a": 2}]], L(s)) for label, out in [ ("Array.cast", arr.cast(arr.type)), ("ChunkedArray.cast", pa.chunked_array([arr]).cast(arr.type)), ]: try: out.validate(full=True) print(f"ok {label} of list [[x, x]]") except pa.ArrowInvalid as e: print(f"FAIL {label} of list [[x, x]]: {e}") ``` </p> </details> [repro.py](https://github.com/user-attachments/files/32807937/repro.py) <details><summary>Details</summary> <p> ```py """Casting a struct with a null-typed field, nested in a list or sliced, to its own type returns an invalid array, without raising. Run: python repro.py (needs only pyarrow). Exits non-zero while the bug is present. """ import sys import pyarrow as pa import pyarrow.compute as pc print("pyarrow", pa.__version__) s = pa.struct([("a", pa.int64()), ("n", pa.null())]) failed = False # 1. A list of 1 row holding 2 structs arr = pa.array([[{"a": 1}, {"a": 2}]], type=pa.list_(s)) arr.validate(full=True) # the input is valid out = pc.cast(arr, arr.type) # does not raise print( f"list: {len(arr)} row, struct length {len(out.values)}," f" child a length {len(out.values.field(0))}, child n length {len(out.values.field(1))}" ) try: out.validate(full=True) except pa.ArrowInvalid as e: print(" BUG: the cast result is invalid:", e) failed = True # 2. A struct array sliced to its second row arr = pa.array([{"a": 0}, {"a": 1}], type=s).slice(1) arr.validate(full=True) out = pc.cast(arr, s) print(f"sliced struct: offset {out.offset}, length {len(out)}") try: out.validate(full=True) except pa.ArrowInvalid as e: print(" BUG: the cast result is invalid:", e) failed = True sys.exit(1 if failed else 0) ``` </p> </details> [repro_parquet.py](https://github.com/user-attachments/files/32807939/repro_parquet.py) <details><summary>Details</summary> <p> ```py """The same bug reached through Parquet: batches from ParquetFile.iter_batches are slices, so RecordBatch.cast to the file's own schema fails once a batch starts past row 0. Run: python repro_parquet.py (needs only pyarrow). Writes repro.parquet next to itself. """ from pathlib import Path import pyarrow as pa import pyarrow.parquet as pq print("pyarrow", pa.__version__) path = Path(__file__).with_name("repro.parquet") t = pa.list_(pa.struct([("a", pa.int64()), ("n", pa.null())])) pq.write_table(pa.table({"c": pa.array([None, [{"a": 1}, {"a": 2}]], type=t)}), path) f = pq.ParquetFile(path) for i, batch in enumerate(f.iter_batches(batch_size=1)): batch.validate(full=True) # every batch is valid try: batch.cast(f.schema_arrow) print(f"batch {i}: RecordBatch.cast to the file's schema ok") except pa.ArrowInvalid as e: print(f"batch {i}: RecordBatch.cast to the file's schema FAILS: {e}") # Reading the whole file in one batch, nothing is sliced and the cast succeeds pq.read_table(path).cast(f.schema_arrow) print("read_table(...).cast(schema) ok") ``` </p> </details> **Description:** Take a struct with a field of type `null`, nested in a list holding more structs than the list has rows, or sliced. Cast it to its own type, and the result is invalid. The cast does not raise: the result's null-typed child comes back with the length of the array passed to `cast`, not the struct's own length. It fails `validate()`, and `to_pylist()` on it can raise. - **Silent:** `Array.cast`, `ChunkedArray.cast` and `pyarrow.compute.cast` return the invalid array without raising. - **Raises:** `RecordBatch.cast` raises `ArrowInvalid`, because it validates its result. Reproduced on pyarrow **25.0.1**, the latest release on PyPI on 2026-09-29, on Linux x86_64 with Python 3.11. A search of apache/arrow issues on 2026-09-29 found no report of it. The nearest, [GH-50515](https://github.com/apache/arrow/pull/50546) ("Respect parent validity bitmap when casting nested structs with non-nullable fields"), is a different struct-cast bug. ## Minimal repro [`repro.py`](repro.py), which needs only pyarrow: ```python import pyarrow as pa s = pa.struct([("a", pa.int64()), ("n", pa.null())]) arr = pa.array([[{"a": 1}, {"a": 2}]], type=pa.list_(s)) # 1 row, 2 structs arr.validate(full=True) # the input is valid out = arr.cast(arr.type) # does not raise len(out.values) # 2 len(out.values.field(0)) # 2, the int64 child len(out.values.field(1)) # 1, the null child: the list's length, not the struct's out.validate(full=True) # pyarrow.lib.ArrowInvalid: List child array invalid: Invalid: Struct child array #1 has length # smaller than expected for struct array (1 < 2) ``` A sliced struct array, not in a list, fails the same way: ```python arr = pa.array([{"a": 0}, {"a": 1}], type=s).slice(1) pa.compute.cast(arr, s).validate(full=True) # pyarrow.lib.ArrowInvalid: Struct child array #1 has length smaller than expected for struct array (1 < 2) ``` Output of `python repro.py` (exits 1 while the bug is present): ``` pyarrow 25.0.1 list: 1 row, struct length 2, child a length 2, child n length 1 BUG: the cast result is invalid: List child array invalid: Invalid: Struct child array #1 has length smaller than expected for struct array (1 < 2) sliced struct: offset 1, length 1 BUG: the cast result is invalid: Struct child array #1 has length smaller than expected for struct array (1 < 2) ``` ## Which casts fail [`matrix.py`](matrix.py) validates each cast's result. Output on 25.0.1: ``` ok list [[x]]: 1 row, 1 struct FAIL list [[x, x]]: 1 row, 2 structs: input length 1, List child array invalid: Invalid: Struct child array #1 has length smaller than expected for struct array (1 < 2) ok list [[x], [x]]: 2 rows, 2 structs ok list [None, [x, x]]: 2 rows, 2 structs FAIL list [[x, x], [x]]: 2 rows, 3 structs: input length 2, List child array invalid: Invalid: Struct child array #1 has length smaller than expected for struct array (2 < 3) FAIL large_list [[x, x]]: input length 1, List child array invalid: Invalid: Struct child array #1 has length smaller than expected for struct array (1 < 2) FAIL list_view [[x, x]]: input length 1, List-view child array is invalid: Invalid: Struct child array #1 has length smaller than expected for struct array (1 < 2) ok list [[x, x]] with n: int64 (control) ok list [[x, x]] -> large_list<same struct> ok list [[x, x]] -> list<struct<a: int32, n: null>> ok struct, 2 rows, not sliced FAIL struct, 2 rows, slice(1): input length 1, Struct child array #1 has length smaller than expected for struct array (1 < 2) ok struct, 2 rows, slice(1) -> struct<a: int32, n: null> FAIL Array.cast of list [[x, x]]: List child array invalid: Invalid: Struct child array #1 has length smaller than expected for struct array (1 < 2) FAIL ChunkedArray.cast of list [[x, x]]: In chunk 0: Invalid: List child array invalid: Invalid: Struct child array #1 has length smaller than expected for struct array (1 < 2) ``` - Every failure is a struct whose length differs from the length of the array passed to `cast`: a list with more structs than rows (`list`, `large_list` and `list_view` alike), or a sliced struct. The error's first number is the input's length, the second the length the struct needs. - Lists whose row count equals their struct count cast correctly, including one with a null row. - The same struct with the field as `int64` casts correctly. - Casting to a different type, one that changes another field (`a` to `int32`) or the list type (`list` to `large_list`), gives a valid result. Only a cast to the array's own type fails. ## Through Parquet [`repro_parquet.py`](repro_parquet.py) shows how the bug arises in ordinary use. It writes `[None, [x, x]]` to Parquet and reads it back in 1-row batches. The second batch is `[[x, x]]`, a list with 1 row and 2 structs. So `RecordBatch.cast` to the file's own schema fails on it, although the batch passes `validate(full=True)`: ``` pyarrow 25.0.1 batch 0: RecordBatch.cast to the file's schema ok batch 1: RecordBatch.cast to the file's schema FAILS: In column 0: Invalid: List child array invalid: Invalid: Struct child array #1 has length smaller than expected for struct array (1 < 2) read_table(...).cast(schema) ok ``` Read whole, the file has 2 rows and 2 structs, and the cast succeeds. ### Component(s) Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
