cshuo opened a new issue, #20291:
URL: https://github.com/apache/hudi/issues/20291
### Bug Description
**What happened:**
Flink bucket pruning can silently omit existing rows when a compound record
key contains the literal string `__empty__` or `__null__`.
For example, with record key fields `uuid,name`, eight buckets, and
`(uuid='__empty__', name='x')`, the writer assigns the row to bucket **1**, but
the reader's equality-filter pruning selects bucket **4**. An unfiltered scan
returns the row, while filtering on both key fields returns no rows. This was
reproduced with both COPY_ON_WRITE and MERGE_ON_READ tables.
**What you expected:**
An equality query on a stored compound key should return the same matching
row as an unfiltered scan.
**Steps to reproduce:**
1. Create a Flink Hudi table with a compound primary key and bucket index:
```sql
CREATE TABLE t1 (
uuid STRING,
name STRING,
age INT,
ts TIMESTAMP(3),
`partition` STRING,
PRIMARY KEY (uuid, name) NOT ENFORCED
) PARTITIONED BY (`partition`) WITH (
'connector' = 'hudi',
'path' = '/tmp/hudi-compound-key-bucket-pruning',
'table.type' = 'COPY_ON_WRITE',
'hoodie.datasource.write.recordkey.field' = 'uuid,name',
'ordering.fields' = 'ts',
'index.type' = 'BUCKET',
'hoodie.bucket.index.num.buckets' = '8'
);
INSERT INTO t1 VALUES
('__empty__', 'x', 23, TIMESTAMP '1970-01-01 00:00:01', 'par1');
```
2. Run a batch scan and an equality query:
```sql
SELECT * FROM t1;
SELECT * FROM t1 WHERE name = 'x' AND uuid = '__empty__';
```
3. The first query returns the row, but the second returns an empty result.
The same issue occurs with the literal `__null__` and with MERGE_ON_READ tables.
**Root cause:**
The writer hashes values extracted from the encoded record key using
`KeyGenUtils.extractRecordKeysByFields`. For compound keys, that extraction
maps `__empty__` to an empty string and `__null__` to null.
`PrimaryKeyPruners.getBucketIdFunc` instead hashes the filter literals
directly, producing a different bucket ID. The mismatch also applies when only
a subset of the compound key fields is used for bucket hashing. Single-field
keys retain the literal placeholder strings and must preserve that behavior.
### Environment
- Hudi: `1.3.0-SNAPSHOT`, reproduced against commit `a41f6eaf7df0`.
- Query engine: Flink 1.20.1, batch SQL.
- Java: 11.
- Table types: COPY_ON_WRITE and MERGE_ON_READ.
- Relevant configuration: compound record key `uuid,name`, bucket index,
eight buckets.
### Logs and Stack Trace
This is a silent incorrect-result issue, not a query exception. Regression
assertions against the unpatched implementation report:
```text
Bucket ID: expected: <1> but was: <4>
SQL result: expected: <[+I[__empty__, x, 23, 1970-01-01T00:00:01, par1]]>
but was: <[]>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
