chicagobuss commented on issue #15812:
URL: https://github.com/apache/datafusion/issues/15812#issuecomment-6463260117
I can reproduce this with DataFusion engine 55.2.0, the latest stable crate
version on 2026-10-28. Here is a four-row PyArrow reproduction, including
equality with NaN and the settings needed to recover the rows.
Environment: Linux x86_64, kernel 7.0.0-38-generic, glibc 2.43; CPython
3.14.4; PyArrow 26.0.0. The PyPI package is `datafusion==55.0.0`; `SELECT
version()` reports `Apache DataFusion 55.2.0, x86_64 on linux`. The standalone
`datafusion-cli 55.2.0` gives the same control results below.
Save this as `repro_datafusion_pruning.py`. In a fresh virtual environment
install `datafusion==55.0.0 pyarrow==26.0.0`, then run `python
repro_datafusion_pruning.py`. The script creates its own input beside itself.
```python
from pathlib import Path
import datafusion as df
import pyarrow as pa
import pyarrow.parquet as pq
path = Path(__file__).with_name('df_nan.parquet')
pq.write_table(pa.table({'x': [float('nan'), 0.0, 0.25, 1.0]}), path,
row_group_size=3, use_dictionary=False,
write_page_index=False)
print('datafusion Python:', df.__version__, 'pyarrow:', pa.__version__)
print(df.SessionContext().sql('SELECT version()').collect())
for pruning, statistics, page_index in [(True, True, True), (False, True,
True),
(True, False, True), (True, True,
False),
(False, False, False)]:
config = df.SessionConfig().with_target_partitions(1)
config.set('datafusion.execution.collect_statistics',
str(statistics).lower())
config.set('datafusion.execution.parquet.pruning', str(pruning).lower())
config.set('datafusion.execution.parquet.enable_page_index',
str(page_index).lower())
ctx = df.SessionContext(config)
# register_parquet's explicit pruning option overrides the session
setting.
ctx.register_parquet('t', path, parquet_pruning=pruning)
print('pruning/statistics/page_index:', pruning, statistics, page_index)
for query, expected in [
('SELECT count(*) AS n FROM t WHERE x >= 0.5', 2),
('SELECT count(*) AS n FROM t WHERE x > 0.5', 2),
("SELECT count(*) AS n FROM t WHERE x = 'NaN'::DOUBLE", 1),
('SELECT max(x) AS m FROM t', 'NaN'),
('SELECT max(x+0) AS m FROM t', 'NaN'),
]:
observed = [r for b in ctx.sql(query).collect() for r in
b.to_pylist()]
print(query, 'observed:', observed, 'expected:', expected)
```
Observed results (both `>` and `>=` give the range-count column below):
| Row-group pruning | Collect statistics | Page index | range count | NaN
equality count | `MAX(x)` |
|---|---|---|---:|---:|---|
| on | on | on | 1 | 0 | 1.0 |
| off | on | on | 2 | 0 | 1.0 |
| on | off | on | 1 | 0 | NaN |
| on | on | off | 1 | 0 | 1.0 |
| off | off | off | 2 | 1 | NaN |
| expected | | | 2 | 1 | NaN |
`MAX(x+0)` returns NaN in every configuration. A separate CLI control with
pruning and statistics off but page index **on** also gives 2, 1, NaN. The file
contains no page index, so disabling that feature has no effect here. Disabling
row-group pruning alone fixes the range query; equality also needs collected
statistics disabled. That distinguishes row-group pruning from an additional
use of collected bounds before scanning. The metadata aggregate shortcut is
included as supporting evidence for the same unsafe bounds, not as a claim that
all consumers share one function.
After the Python script has created `df_nan.parquet`, the equivalent CLI
control is:
```sh
datafusion-cli -q --format csv \
-c "SET datafusion.execution.parquet.pruning = false;" \
-c "SET datafusion.execution.collect_statistics = false;" \
-c "SELECT count(*) AS n FROM 'df_nan.parquet' WHERE x >= 0.5;" \
-c "SELECT count(*) AS n FROM 'df_nan.parquet' WHERE x = 'NaN'::DOUBLE;"
# n = 2, then n = 1; defaults give 1, then 0.
```
The Python API's `register_parquet(..., parquet_pruning=True)` default
overrides the session pruning option. The reproduction passes the flag
explicitly so the off control actually disables pruning. The CLI does not have
that registration override.
[DataFusion's data type
guide](https://datafusion.apache.org/user-guide/sql/data_types.html) identifies
Arrow as its execution type system. [Arrow Rust's comparison-kernel
documentation](https://github.com/apache/arrow-rs/blob/60.0.0/arrow-ord/src/cmp.rs#L63-L80)
specifies IEEE totalOrder. [DataFusion's own release
tests](https://github.com/apache/datafusion/blob/55.2.0/datafusion/sqllogictest/test_files/select.slt#L571-L575)
expect positive NaN to compare greater than zero. On the same values in
memory, NaN equality matches and the range predicate includes NaN. This example
uses only the ordinary positive NaN written by Python, not distinct signs or
payloads.
[Parquet format
2.14.0](https://github.com/apache/parquet-format/blob/apache-parquet-format-2.14.0/src/main/thrift/parquet.thrift#L312-L318)
says that when `nan_count` is absent readers “MUST assume NaNs may be
present”. Its [TYPE_ORDER reading
rules](https://github.com/apache/parquet-format/blob/apache-parquet-format-2.14.0/src/main/thrift/parquet.thrift#L1159-L1170)
say that when looking for NaN, “min and max should be ignored”. A present
count can establish whether NaNs exist; the bounds describe non-NaN values.
These reading rules also cover files written before counts were introduced. The
generated file has TYPE_ORDER, finite bounds, and no `nan_count` or page index.
Main runtime status: **not tested**. I found no packaged official DataFusion
nightly; GitHub release APIs offered no executable assets, and a source build
was not undertaken. At main `6341b5c9fc30c0215ba326a170eaa294ca1870b7`,
[row-group min/max
conversion](https://github.com/apache/datafusion/blob/6341b5c9fc30c0215ba326a170eaa294ca1870b7/datafusion/datasource-parquet/src/row_group_filter.rs#L568-L584)
still supplies the bounds without consulting `nan_count`. This suggests the
pruning path remains relevant, but is a source-based inference, not a main
reproduction. [PR #24804](https://github.com/apache/datafusion/pull/24804),
addressing #15812, remains open and unmerged as checked on 2026-10-28.
Related: #15812 is the existing report for this root cause.
[#26490](https://github.com/apache/datafusion/issues/26490) separately tracks
replacing a float column with a constant from `min == max`; this reproduction
keeps distinct finite bounds in the NaN-containing group to avoid that
confound. [#24432](https://github.com/apache/datafusion/issues/24432) is the
separate grouped-accumulator bug, already fixed on main; the `MAX` queries
above have no `GROUP BY`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
