chrikrah opened a new issue, #51696:
URL: https://github.com/apache/arrow/issues/51696
<!--
TITLE: [Python] DataFrame interchange: from_dataframe rejects every pandas
float16 column
Form: Bug Report. Component(s): Python. Companion patch:
.drafts/p9b-patches/arrow_new_float16_from_dataframe.patch, title GH-<n> once
this has a number.
-->
### Describe the bug, including details regarding any error messages,
version, and platform.
`pa.interchange.from_dataframe` raises `NotImplementedError` for a pandas
`float16` column, with or without a NaN in it. pandas reports NaN as the null
kind for every float column, and `validity_buffer_nan_sentinel` in
`python/pyarrow/interchange/from_dataframe.py:549` refuses `float16` with that
null kind. The comment there says `pyarrow.compute.is_nan` has no `float16`
kernel. That kernel arrived in #46866 (GH-45083), so the guard is stale.
```
$ python -c 'import numpy as np, pandas as pd, pyarrow as pa
from pyarrow.interchange import from_dataframe
print(from_dataframe(pd.DataFrame({"h": np.array([1.5], dtype="<f2")})))'
NotImplementedError: (<DtypeKind.FLOAT: 2>, 16, 'e', '=') with 1 is not yet
supported.
# pyarrow 26.0.0.dev350 nightly wheel, pandas 3.2.0.dev0, CPython 3.12,
Linux x86_64
```
With the `float16` branch removed, so the NaN mask goes through `pc.is_nan`
like `float32` and `float64`:
```
$ python -m pytest -q pyarrow/tests/interchange/
# main e85e181
3045 passed, 2 skipped
# branch removed, test_pandas_to_pyarrow_float16_with_missing changed to
expect [0, None, 2] as float16
3045 passed, 2 skipped
# that test change alone, from_dataframe.py at main
1 failed, 3044 passed, 2 skipped
```
The test `test_pandas_to_pyarrow_float16_with_missing` pins the old
`NotImplementedError`, so it changes with the fix.
### Component(s)
Python
I saw #49629. Until the protocol is removed, `pa.table(df)` imports the same
column:
```
$ python -c 'import numpy as np, pandas as pd, pyarrow as pa
print(pa.table(pd.DataFrame({"h": np.array([1.5, np.nan], dtype="<f2")})))'
pyarrow.Table
h: halffloat
----
h: [[1.5,null]]
# pyarrow 26.0.0.dev350 nightly wheel, pandas 3.2.0.dev0
```
I have the patch, 14 lines added and 20 removed across those two files. Is
it worth opening against this issue, or would you rather close this against
#49629?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
