cshuo opened a new pull request, #20294: URL: https://github.com/apache/hudi/pull/20294
### Describe the issue this Pull Request addresses Closes https://github.com/apache/hudi/issues/20293 A Flink equality predicate for `1970-01-01T00:00:01.000001` currently becomes `eq(ts, 1000)`, while the Parquet `TIMESTAMP(6)` value is stored as `1000001` microseconds. The matching row group is discarded. Millisecond truncation also affects column-statistics comparisons and timestamp index keys. ### Summary and Changelog - Preserve timestamp literals as `LocalDateTime` in `ExpressionUtils` and compare column statistics without discarding sub-millisecond precision. - Convert Parquet literals according to the referenced field: `TIMESTAMP(3)` to milliseconds and `TIMESTAMP(6)` to microseconds. Skip pushdown for other precisions, inexact conversions, or overflow; skip the entire `IN` predicate if an element cannot be converted safely. - Preserve timestamp precision in record-index lookup keys and align bucket-pruning key formatting with the Flink writer, including the existing consistent logical timestamp setting. - Add regressions for microsecond comparisons, whole-second literals against microsecond fields, negative timestamps, index-key compatibility, and actual Parquet row-group pruning. Validation: 119 focused tests passed with Flink 1.20.1 / Java 11; Checkstyle and `git diff --check` passed. The new literal, column-statistics, and record-index regressions were also run against the original implementation and failed as expected. ```bash mvn -o -pl hudi-flink-datasource/hudi-flink -am -Pflink1.20 -Dflink1.20 -Dtest=TestExpressionPredicates,TestRecordIterators,TestExpressionUtils,TestExpressionEvaluators,TestColumnStatsProbe,TestRecordLevelIndex,TestPrimaryKeyPruners,TestColumnStatsIndex -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true test ``` ### Impact Prevents timestamp precision loss during Flink filtering and index lookup. No storage-format or configuration changes. Timestamp values returned by the internal expression utility now retain their full precision, and its consumers are updated together. Parquet conversion follows the existing Flink mapping of ts3/ts6 fields to INT64 milliseconds/microseconds, without additional footer reads. This does not introduce physical-schema-aware conversion for arbitrary external file encodings or change local-timezone read behavior. ### Risk Level Medium: the shared timestamp literal representation changes across predicate and index consumers. Focused tests cover the affected paths, both consistent logical timestamp key settings, unsupported/inexact predicate fallbacks, and real Parquet reads. Other Flink versions and the full integration suite have not been run locally. ### Documentation Update None. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
