pearu opened a new issue, #50842:
URL: https://github.com/apache/arrow/issues/50842
### Describe the enhancement requested
Once #50840 is fixed (#50841), converting a dictionary array whose decoded
form exceeds `INT32_MAX` bytes raises cleanly instead of crashing:
```python
import numpy as np, pyarrow as pa
arr = pa.DictionaryArray.from_arrays(
np.zeros(50_000_000, dtype=np.int16),
pa.array(["a" * 50]),
)
np.asarray(arr)
# pyarrow.lib.ArrowInvalid: Take operation overflowed binary array capacity
```
That is a strict improvement over a segfault, but the conversion still fails
on data that is entirely representable in the output. The result of
`np.asarray` here is a **numpy object array of Python `str`** — a format with
no 2 GiB limit. The failure comes purely from an intermediate step.
#### Why it fails
`Array.to_numpy(zero_copy_only=False)` sets `decode_dictionaries=True`. In
`arrow_to_pandas.cc`, `ConvertChunkedArrayToPandas()` then decodes by casting
to the dictionary's **value type**:
```cpp
const auto& dense_type =
checked_cast<const DictionaryType&>(*arr->type()).value_type(); //
string, int32 offsets
RETURN_NOT_OK(DecodeDictionaries(options.pool, dense_type, &arr));
```
So a 50M × 50-byte decode has to fit in a 32-bit-offset `string` array. It
cannot — hence the error. But that dense array is a pure implementation
artifact; nothing downstream needs it to be a `string` array specifically.
#### Approach (a): widen the intermediate type
Use `large_string`/`large_binary` as the decode target instead of
`string`/`binary`.
**Pros**
- Very small, local change — essentially picking a different `dense_type`
- Low risk; reuses the existing cast/take machinery unchanged
- Invisible to users: the final output is an object array either way
- Fixes the `to_numpy` and `to_pandas` paths together
**Cons**
- Still materializes the entire dense array (2.5 GB in the example) purely
as scratch
- Still creates one Python object per element — 50M separate `str` objects
for 50M indices — so peak memory stays very large even though the input holds a
single distinct value
- Slightly increases memory for the common, non-overflowing case (8-byte
offsets), unless a size pre-check is added to decide when to promote
- Treats the symptom rather than the redundancy
#### Approach (b): skip the dense intermediate entirely
For the object-output path, convert each **dictionary value** to a
`PyObject` once, then walk the indices and `Py_INCREF` the corresponding object
into the output array.
**Pros**
- No dense intermediate at all — the 2.5 GB scratch buffer disappears
- For the example, one Python string is allocated instead of 50M; memory and
time drop by orders of magnitude
- Removes the size ceiling entirely rather than raising it
- Also speeds up the ordinary, non-overflowing dictionary → object
conversion, which is the common categorical/pandas path
- Output is semantically identical: object arrays hold references, and
Python strings are immutable, so sharing is unobservable
**Cons**
- Larger change: needs a dedicated dictionary branch in the object-writer
instead of reusing `DecodeDictionaries`
- Null indices and index bounds must be handled explicitly rather than
inherited from the cast
- Only covers the object-output path; any path that genuinely needs a dense
Arrow array (e.g. `strings_to_categorical`) would still want (a)
- Callers relying on distinct object identity per element would see shared
references — not a documented guarantee, and already untrue for interned
strings, but worth noting
#### Recommendation
**(b)**, because it addresses the actual redundancy rather than raising a
ceiling. Dictionary-encoded data is used precisely when values repeat, so
materializing one Python object per *index* rather than per *dictionary entry*
is wasted work in every case — the overflow is just where it becomes fatal. It
also improves the common path, not only the pathological one.
(a) remains worth keeping in reserve for any remaining code path that must
produce a dense array; the two are complementary rather than exclusive.
I'm happy to implement (b), pending agreement on the direction.
### Component(s)
Python
---
_🤖 Drafted by Claude Code (an AI agent) and reviewed & approved by pearu._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
