yuqi1129 opened a new issue, #13893:
URL: https://github.com/apache/gravitino/issues/13893
### Describe the subtask
This subtask covers the three P0 items in #13828 (plan IDs 1, 2 and 4):
1. **Late-committing change-log rows.** Allocation order of change-log IDs
is not commit order. When a lower ID commits after a higher ID has already
moved the cursor, the `id > cursor` query skips it permanently.
- Fix: keep incremental polling and add bounded recent-history replay.
- The window is based on the database clock, default `replayWindowSecs`
= 60.
- Replay pages by `(created_at, id)` and deduplicates IDs locally,
default `replayMaxEntries` = 100000.
- A new sweep starts at most once per `max(pollIntervalSecs,
replayWindowSecs / 4)`.
- Add diagnostics: replay logs, plus the `replay-records-total`,
`replay-failures-total` and `dedup-evictions-total` metrics.
- Listeners opt in with `supportsReplay()`. Third-party listeners stay
incremental-only.
2. **Stale cache publication.** A point read can load a child before an
ancestor invalidation and publish it after the ancestor's index scan has
finished.
- Fix: check the invalidation epoch before publishing on point reads and
on insert, the same way batch reads already do.
- The check must not depend on the cache SPI locks being reentrant.
3. **Catalog local mutation markers.** The poller can consume a local
catalog change before its marker is registered. The leftover marker then hides
a later change to the same catalog from another server.
- Fix: remove the name-based marker skip and invalidate on every catalog
event, including local ones. Operation leases protect wrappers that are in use.
Out of scope: the replay window is bounded compensation, not lossless
delivery. Transactions older than the window and rows already pruned can still
be missed.
### Parent issue
#13828
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
