cshuo opened a new issue, #20296:
URL: https://github.com/apache/hudi/issues/20296
### Description
Spark data skipping can silently omit matching rows after adding columns to
the Column Stats Index (CSI).
After adding `name` to the indexed column list, existing files may still
lack statistics for `name`, even though they contain non-null names. When a
query references both an originally indexed column and the newly indexed
column, the partially indexed file is incorrectly pruned.
For example, after changing the indexed column list from `age` to `age,name`:
- `name = 'alice'` returns the matching row.
- `age >= 0 AND name = 'alice'` incorrectly returns no rows with data
skipping enabled.
- The same combined predicate returns the matching row with data skipping
disabled.
This was reproduced with real Spark SQL writes and queries in a local
regression test added to `TestDataSkippingQuery`.
### Environment
- Hudi: `master` at `a41f6eaf7df0`, `1.3.0-SNAPSHOT`.
- Spark: `3.5.9`.
- Scala: `2.12.18`.
- Java: `11`.
- Table type: `COPY_ON_WRITE`.
- Metadata and column statistics enabled.
- Partition statistics disabled to isolate file-level CSI pruning.
### Steps to reproduce
Use a fresh table location and wait for each INSERT to complete.
```sql
SET hoodie.metadata.enable=true;
SET hoodie.metadata.index.column.stats.enable=true;
SET hoodie.metadata.index.partition.stats.enable=false;
SET hoodie.metadata.index.column.stats.column.list=age;
SET hoodie.enable.data.skipping=false;
CREATE TABLE csi_partial_stats (
id INT,
name STRING,
age INT,
ts BIGINT,
dt STRING
) USING hudi
TBLPROPERTIES (
primaryKey = 'id',
orderingFields = 'ts',
type = 'cow'
)
PARTITIONED BY (dt)
LOCATION '/tmp/hudi-spark-csi-partial-stats';
-- The name column exists in the data, but only age is indexed.
INSERT INTO csi_partial_stats
VALUES (1, 'alice', 25, 1, 'par1');
SET hoodie.metadata.index.column.stats.column.list=age,name;
-- Writing to another partition leaves the old file untouched.
INSERT INTO csi_partial_stats
VALUES (2, 'bob', 30, 2, 'par2');
-- Returns (1, alice, 25).
SELECT id, name, age
FROM csi_partial_stats
WHERE age >= 0 AND name = 'alice';
SET hoodie.enable.data.skipping=true;
-- Explicitly update the table property: changing only the session setting
-- can leave the creation-time false setting in effect.
ALTER TABLE csi_partial_stats
SET TBLPROPERTIES ('hoodie.enable.data.skipping' = 'true');
REFRESH TABLE csi_partial_stats;
-- Returns (1, alice, 25).
SELECT id, name, age
FROM csi_partial_stats
WHERE name = 'alice';
-- Incorrectly returns no rows.
SELECT id, name, age
FROM csi_partial_stats
WHERE age >= 0 AND name = 'alice';
```
### Expected behavior
All three SELECTs should return:
```text
1, alice, 25
```
Enabling data skipping or adding a predicate satisfied by the matching row
must not change the query result.
### Actual behavior
| Predicate | Data skipping | Actual result |
|---|---|---|
| `age >= 0 AND name = 'alice'` | Disabled | `(1, alice, 25)` |
| `name = 'alice'` | Enabled | `(1, alice, 25)` |
| `age >= 0 AND name = 'alice'` | Enabled | Empty |
The regression test confirms the required metadata state before executing
these queries:
- The CSI definition includes both `age` and `name`.
- Two files have `age` statistics.
- Only one file has `name` statistics.
- Per-file statistics coverage is `{age}` for one file and `{age, name}` for
the other.
The test fails at the final query assertion:
```text
Combined age/name predicate with data skipping enabled:
Expected Array([1,alice,25]), but got Array()
```
### Root cause
1.
[`ColumnStatsIndexSupport`](https://github.com/apache/hudi/blob/a41f6eaf7df0/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/ColumnStatsIndexSupport.scala#L289-L313)
pads missing per-file column statistics with `(null, null, null)`.
2.
[`DataSkippingUtils.genColumnValuesEqualToExpression`](https://github.com/apache/hudi/blob/a41f6eaf7df0/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/DataSkippingUtils.scala#L495-L501)
translates `name = 'alice'` into:
```sql
name_minValue <= 'alice' AND name_maxValue >= 'alice'
```
For the old file, both bounds are NULL, so this predicate evaluates to
SQL NULL rather than conservatively retaining the file.
3.
[`SparkBaseIndexSupport.getCandidateFiles`](https://github.com/apache/hudi/blob/a41f6eaf7df0/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkBaseIndexSupport.scala#L115-L143)
applies the translated predicate directly through `DataFrame.where`, which
drops rows whose predicate evaluates to NULL. Its fallback only restores files
entirely absent from the transposed index. The old file already appears because
it has `age` statistics, so it is not restored.
For the name-only query, the old file is absent from the requested
statistics projection and is retained by the unindexed-file fallback. Adding
the `age` predicate causes the file to appear in that projection and exposes
the incorrect pruning.
The data column is present from the first write; this is missing statistics,
not a missing schema field or known all-null data.
### Suggested behavior
Treat unavailable column statistics as unknown and retain a file whenever
the available statistics cannot prove that the predicate is unsatisfiable.
Spark already handles unknown null counts conservatively for `IS NULL` and
`IS NOT NULL`. Equality and range predicates also need to account for
incomplete statistics without conflating them with known all-null columns.
### Validation
A local regression test was added to `TestDataSkippingQuery`:
```text
Data skipping retains files with missing column statistics after index
expansion
```
The targeted ScalaTest run reproduced the incorrect empty result. Relevant
sources were recompiled against locally built dependencies; a full Maven build
and the complete test suite were not run. No production fix is included, and
the regression test has not yet been published in a PR.
### Related issue
The Flink reader has a separately reproduced issue involving partial
column-statistics coverage:
[apache/hudi#20289](https://github.com/apache/hudi/issues/20289).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
