nsivabalan opened a new pull request #4800:
URL: https://github.com/apache/hudi/pull/4800
## What is the purpose of the pull request
Timeline server when serving remote requests, has a logic to refresh its
local view of the timeline based on timeline hash. Client sends a timeline hash
and timeline compares with its local timeline hash and if they differ, a
refresh of timeline happens before serving the request. But this refresh gets
triggered even if the client is behind, but the server is already caught up.
So, adding a new value to be maintained by the timeline for lastUpdatedTime.
and the same will be sent as param with remote request as well.
Fix: So, the fix ensures that timeline server triggers a refresh of local
timeline only if its lastUpdatedTime < client's lastUpdatedTime.
To discuss:
Clocks could differ in timeline server compared to that of the executor
(client) and there could be drift as well. So, not very sure if we can rely on
the exact comparison of last updated time between client and server.
Another option: I am wondering if we can rely on
lastKnownInstant(HoodieInstant) from client and compare it w/ that of timeline
in timeline server and decide whether to refresh or not instead of the
lastUpdatedTime.
## Brief change log
- Added lastUpdatedTime to HoodieTimeline which gets refreshed whenever the
timeline is updated. The same value is sent via remote requests to Timeline
server.
- Timeline server triggers a refresh of its local timeline only if its
lastUpdatedTime < client's lastUpdatedTime in addition to timeline hash
mismatch.
## Verify this pull request
- Couple of users in the community actually tested this and contributed the
patch.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
