Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23789
to look at the new patch set (#2).
Change subject: IMPALA-14535: Improve wait for HMS events sync with
hierarchical event processing
......................................................................
IMPALA-14535: Improve wait for HMS events sync with hierarchical event
processing
With hierarchical event processing, MetastoreEventsProcessor's
lastSyncedEventId_ represents the last event dispatched to the
EventExecutorService.Events are queued to appropriate
DbEventExecutors and/or TableEventExecutor for processing.
And events are removed from these queues only after they
are processed by the respective executor threads.
When a wait for HMS events sync request is issued, the
following sequence is executed:
1. The current notification event id is fetched from HMS.
2. MetastoreEventsProcessor is ensured to fetch and dispatch
all events up to and including this notification event id
to the EventExecutorService.
3. The list of databases and tables requiring synchronization
is extracted from the request.
4. To determine whether synchronization for a required
database and or table is complete, relevant DbEventExecutor
and TableEventExecutor instances are checked directly.
Synchronization is considered complete only when the
event-processing progress of the required DbProcessor and
TableProcessor instances has surpassed the captured
notification event id.
This approach guarantees that all HMS events that existed at
the time of sync request have been fully processed for the
requested databases and tables.
Testing:
- Added an FE test and ran the existing tests.
Change-Id: I55cea4cb8e04860202e56e1b1bf2596613b4946c
---
M be/src/catalog/catalog-server.cc
M fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java
M fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java
M
fe/src/test/java/org/apache/impala/catalog/events/EventExecutorServiceTest.java
7 files changed, 362 insertions(+), 40 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/23789/2
--
To view, visit http://gerrit.cloudera.org:8080/23789
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I55cea4cb8e04860202e56e1b1bf2596613b4946c
Gerrit-Change-Number: 23789
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
