ODriscollBoys opened a new pull request, #20279:
URL: https://github.com/apache/hudi/pull/20279
### Describe the issue this Pull Request addresses
Closes #19635
`sync_validate` has two bugs and a dead guard:
1. `countNewRecords` rebuilt every catch-up instant as a completed `commit`
before reading its metadata. But the catch-up list comes from
`getCommitsTimeline`, so it also holds deltacommits and replacecommits. A
merge-on-read target, or a target with a clustering or insert overwrite in the
catch-up range, failed with `Cannot find the instant[...__commit__COMPLETED]`.
2. In mode `complete`, the connection was closed in an unguarded `finally`
while still `null` when `DriverManager.getConnection` failed, so a
`NullPointerException` hid the real `SQLException`.
3. The `sourceLatestCommit != null` check was always true.
### Summary and Changelog
`sync_validate` now reports the catch-up count for MOR targets and for
targets with replacecommits. A HiveServer2 connection failure now surfaces as
the `SQLException` it is in both record-count modes.
- `countNewRecords` takes the catch-up `HoodieInstant`s and reads each one
through `TimelineUtils.getCommitMetadata`. That resolves the instant under its
own action and reads replacecommits as replace commit metadata. Pending
instants are skipped because they have not committed any records.
- Both record-count modes now share a single `executeCountQuery`. It closes
the result set, statement and connection only after each one has been opened,
in reverse order of opening. This matches the try-with-resources shape of
`hudi-cli`'s `HiveUtil`. It also fixes the `latestPartitions` path, which
closed its connection before its statement.
- Remove the always-true `sourceLatestCommit != null` check.
- Build the `latestPartitions` `datestr` bounds with the `f` interpolator.
The `s` interpolator left `%02d` in the output verbatim (for example
`2024-1%02d-5%02d`), so that mode filtered on malformed dates.
- `TestValidateHoodieSyncProcedure`:
- The MOR pin now asserts the catch-up count.
- New replacecommit catch-up case.
- The JDBC pins for both modes assert that an `SQLException` is in the
cause chain and no `NullPointerException` is.
- New test for the `datestr` formatting.
### Impact
Fixes `sync_validate` for MOR targets, for targets with clustering or insert
overwrite in the catch-up range, and for the `latestPartitions` date bounds.
Connection failures now give an actionable error. The procedure's signature and
output format are unchanged.
### Risk Level
Low. The changes are confined to the `sync_validate` procedure. Validated
with `TestValidateHoodieSyncProcedure` on the Spark 3.5 profile (all cases
pass).
### Documentation Update
None.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
