cshuo opened a new issue, #20293: URL: https://github.com/apache/hudi/issues/20293
### What happened Flink timestamp predicates can discard matching rows when a `TIMESTAMP(6)` literal is reduced to milliseconds before filtering. For example, a Parquet row containing `1970-01-01T00:00:01.000001` is stored as `1000001` microseconds. An unfiltered read returns the correct timestamp, but the equality predicate is built as `eq(ts, 1000)`, causing the matching row group to be excluded. This Parquet filtering issue occurs independently of metadata/data-skipping indexes. The same loss of precision affects other consumers of `ExpressionUtils.getValueFromLiteral()`: - Column statistics and literals are both reduced to milliseconds. Values that differ within one millisecond become indistinguishable, so strict range comparisons and inequality can incorrectly exclude matching files. - Record-index lookup reconstructs timestamp keys from milliseconds, losing the fractional component used by the writer when consistent logical timestamp keys are enabled. - Timestamp bucket-pruning keys can differ from the writer's timestamp representation. ### Expected behavior Preserve the full timestamp literal value. Convert to the required representation at the consumer boundary: milliseconds or microseconds for Parquet predicates, full-precision timestamps for statistics comparisons, and writer-compatible values for index keys. ### Reproduction 1. Write a Parquet file with an INT64 `TIMESTAMP_MICROS` column containing `1000001` (`1970-01-01T00:00:01.000001`). 2. Read it through Flink's Hudi `RecordIterators.getParquetRecordIterator()` using a `TIMESTAMP(6)` field without predicates. The row is returned correctly. 3. Repeat with an `ExpressionPredicates` equality predicate for the same timestamp. The original implementation returns no rows. Additional focused reproductions: - Set the timestamp column-statistics minimum and maximum to `1970-01-01T00:00:01.000002`, then evaluate `ts > TIMESTAMP '1970-01-01 00:00:01.000001'`. The original evaluator incorrectly rejects the file. - Derive a record-index lookup key for `1970-01-01T00:00:01.000001` with consistent logical timestamp keys enabled. The original lookup produces `1970-01-01T00:00:01`, while the writer produces `1970-01-01T00:00:01.000001`. ### Environment - Hudi: development version `1.3.0-SNAPSHOT`, reproduced against commit `a41f6eaf7df0`. - Query engine: Flink 1.20.1. - Local regression validation: Java 11. ### Observed output ```text Filtering using predicate: eq(ts, 1000) expected: <[1970-01-01T00:00:01.000001]> but was: <[]> ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
