alexandrefimov opened a new issue, #51231:
URL: https://github.com/apache/arrow/issues/51231
Could Acero preserve input nullability when a `project` only references
top-level input fields? Currently, such a projection changes a required input
field to nullable in the output schema. A `table_source` and `Table.select`
preserve the same field's nullability. The projected values are unchanged.
This also affects Substrait `RelCommon.emit`: even an identity mapping `[0,
1]` changes a required field to nullable, while a bare read preserves it.
### To reproduce
Install `pyarrow==25.0.1` and run:
```python
import pyarrow as pa
import pyarrow.acero as ac
import pyarrow.compute as pc
schema = pa.schema([
pa.field("r", pa.int64(), nullable=False),
pa.field("n", pa.int64(), nullable=True),
])
table = pa.Table.from_arrays([
pa.array([1, 2], type=pa.int64()),
pa.array([None, 3], type=pa.int64()),
], schema=schema)
source = ac.Declaration("table_source", ac.TableSourceNodeOptions(table))
projected = ac.Declaration(
"project",
ac.ProjectNodeOptions([pc.field("r"), pc.field("n")], ["r", "n"]),
inputs=[source],
).to_table(use_threads=False)
for name, value in [
("source", source.to_table(use_threads=False)),
("select", table.select([0, 1])),
("project", projected),
]:
print(name, [field.nullable for field in value.schema])
print("values unchanged:", projected.to_pylist() == table.to_pylist())
```
Output:
```text
source [False, True]
select [False, True]
project [True, True]
values unchanged: True
```
I expected the direct-reference projection to retain `[False, True]`. It
only selects the two input fields; there are no functions or casts, and the
required input field contains no nulls.
### Schema compatibility impact
Writing the projected table to an IPC stream created with the original
schema fails. Run this after the example above:
```python
import pyarrow.ipc as ipc
with ipc.new_stream(pa.BufferOutputStream(), schema) as writer:
writer.write_table(projected)
```
This raises `ArrowInvalid: Tried to write record batch with different
schema`. Replacing `projected` with `table` or `table.select([0, 1])` succeeds.
The values and column order are identical; the schema differs only in
nullability.
### Where it happens
In
[ProjectNode::Make](https://github.com/apache/arrow/blob/apache-arrow-25.0.1/cpp/src/arrow/acero/project_node.cc#L70),
output fields are created with the expression name and type, without carrying
input nullability. The same construction is present at main commit
`397b3d0023030f6c6bc69d214ea7b27a687256f8`; the runtime reproduction above is
against 25.0.1.
Substrait's
[ProcessEmit](https://github.com/apache/arrow/blob/apache-arrow-25.0.1/cpp/src/arrow/engine/substrait/relation_internal.cc#L136)
adds a field-reference project, which explains the identity emit result.
Related issue #20108 concerns Substrait type serialization and deserialization;
the native Acero example above reproduces this behavior without either step.
The [Acero Substrait
documentation](https://github.com/apache/arrow/blob/apache-arrow-25.0.1/docs/source/cpp/acero/substrait.rst#types)
notes incomplete support for non-nullable types. I have not found an explicit
guarantee that `ProjectNode` preserves input nullability, so I am requesting a
schema-preservation improvement here. This request is limited to direct
references to top-level input fields; general expression nullability inference
is outside its scope.
Four related Substrait cases, including identity and reordered emit with
controls, are [in the
corpus](https://github.com/alexandrefimov/substrait-conformance-cases/tree/6f5af2ed59502c38189a4e4afeff79732df53626/probe/structural-cases/acero).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
