[
https://issues.apache.org/jira/browse/SPARK-57739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-57739:
-----------------------------
Description:
h2. What
During Parquet MIN/MAX aggregate push-down,
{{ParquetUtils.getCurrentBlockMaxOrMin}} throws {{_LEGACY_ERROR_TEMP_3172}}
("No min/max found for Parquet file ... Set SQLConf ... to false and execute
again.") when a column chunk has {{!statistics.hasNonNullValue}} -- i.e. a row
group where the column is all-null. The throw is inside the per-block loop, so
it fires if *any* row group has an all-null chunk for the column (not only when
the whole column is null). There is no fallback in
{{ParquetPartitionReaderFactory}} (row and columnar paths), so the exception
propagates and *fails the query*.
h2. Why it's a bug
MIN/MAX over an all-null input is NULL by SQL semantics, and the ORC aggregate
push-down path already returns NULL for this case
({{OrcUtils.getMinMaxFromColumnStatistics}} guards {{getNumberOfValues == 0 ->
null}}). So the same query succeeds on ORC but fails on Parquet:
{code:sql}
-- column c has a row group that is entirely NULL
SELECT min(c) FROM parquet_t; -- Parquet: throws _LEGACY_ERROR_TEMP_3172;
ORC: returns NULL
{code}
This is type-agnostic (affects every push-down-eligible type), with
{{spark.sql.parquet.aggregatePushdown=true}}.
h2. Proposed fix
In {{getCurrentBlockMaxOrMin}}, treat an all-null block as "no contribution"
(skip it / return NULL) rather than throwing, so an all-null column yields NULL
and a partially-null column combines the non-null row groups -- matching ORC
and SQL semantics. The {{_LEGACY_ERROR_TEMP_3172}} condition can then be
removed (or, if kept for a genuinely-unreadable-stats case, given a proper
named error class as part of the legacy-error migration).
h2. User-facing change
Yes -- a query that currently errors will return the correct NULL. Needs a
migration/release note.
h2. Tests
Add an all-null (non-empty) column case to {{FileSourceAggregatePushDownSuite}}
asserting MIN/MAX = NULL uniformly across Parquet V1/V2 and ORC V1/V2 (the case
intentionally left unasserted in SPARK-57568 because of this divergence).
h2. Notes
Found during review of SPARK-57568 (TIME aggregate push-down), but this is
type-agnostic and unrelated to TIME, so filed as a standalone bug rather than a
sub-task.
> Parquet MIN/MAX aggregate push-down fails on an all-null row group instead of
> returning NULL
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-57739
> URL: https://issues.apache.org/jira/browse/SPARK-57739
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Max Gekk
> Priority: Major
>
> h2. What
> During Parquet MIN/MAX aggregate push-down,
> {{ParquetUtils.getCurrentBlockMaxOrMin}} throws {{_LEGACY_ERROR_TEMP_3172}}
> ("No min/max found for Parquet file ... Set SQLConf ... to false and execute
> again.") when a column chunk has {{!statistics.hasNonNullValue}} -- i.e. a
> row group where the column is all-null. The throw is inside the per-block
> loop, so it fires if *any* row group has an all-null chunk for the column
> (not only when the whole column is null). There is no fallback in
> {{ParquetPartitionReaderFactory}} (row and columnar paths), so the exception
> propagates and *fails the query*.
> h2. Why it's a bug
> MIN/MAX over an all-null input is NULL by SQL semantics, and the ORC
> aggregate push-down path already returns NULL for this case
> ({{OrcUtils.getMinMaxFromColumnStatistics}} guards {{getNumberOfValues == 0
> -> null}}). So the same query succeeds on ORC but fails on Parquet:
> {code:sql}
> -- column c has a row group that is entirely NULL
> SELECT min(c) FROM parquet_t; -- Parquet: throws _LEGACY_ERROR_TEMP_3172;
> ORC: returns NULL
> {code}
> This is type-agnostic (affects every push-down-eligible type), with
> {{spark.sql.parquet.aggregatePushdown=true}}.
> h2. Proposed fix
> In {{getCurrentBlockMaxOrMin}}, treat an all-null block as "no contribution"
> (skip it / return NULL) rather than throwing, so an all-null column yields
> NULL and a partially-null column combines the non-null row groups -- matching
> ORC and SQL semantics. The {{_LEGACY_ERROR_TEMP_3172}} condition can then be
> removed (or, if kept for a genuinely-unreadable-stats case, given a proper
> named error class as part of the legacy-error migration).
> h2. User-facing change
> Yes -- a query that currently errors will return the correct NULL. Needs a
> migration/release note.
> h2. Tests
> Add an all-null (non-empty) column case to
> {{FileSourceAggregatePushDownSuite}} asserting MIN/MAX = NULL uniformly
> across Parquet V1/V2 and ORC V1/V2 (the case intentionally left unasserted in
> SPARK-57568 because of this divergence).
> h2. Notes
> Found during review of SPARK-57568 (TIME aggregate push-down), but this is
> type-agnostic and unrelated to TIME, so filed as a standalone bug rather than
> a sub-task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
