Quanlong Huang has uploaded this change for review. (
http://gerrit.cloudera.org:8080/22571
Change subject: IMPALA-13684: Improve waitForHmsEvent() to only wait for
related events
......................................................................
IMPALA-13684: Improve waitForHmsEvent() to only wait for related events
waitForHmsEvent is a catalogd RPC for coordinators to send a requested
db/table names to catalogd and wait until it's safe (i.e. no stale
metadata) to start analyze the statement. Currently, catalogd waits
until it syncs to the latest HMS event regardless what the query is.
This patch improves this by only checking related events. In the ideal
case, if there are no pending events that are related, query doesn't
need to wait.
We check the pending events in the following way:
- For queries that need the db list, i.e. SHOW DATABASES, check pending
CREATE_DATABASE, DROP_DATABASE events.
- For db statements like SHOW FUNCTIONS, CREATE/ALTER/DROP DATABASE,
check pending CREATE/ALTER/DROP events on that db.
- For db statements that require the table list, i.e. SHOW TABLES,
also check CREATE_TABLE, DROP_TABLE events under that db.
- For unloaded tables, i.e. IncompleteTables, nothing need to be
checked since they don't have stale metadata.
- For loaded transactional tables, check all the pending COMMIT_TXN,
ABORT_TXN events.
- For all the table names,
- if they are all missing in the catalog, check all the pending
CREATE_TABLE, DROP_TABLE events on them.
- Otherwise, some of them are loaded, check all the table events on
them.
This patch leverages the HMS API to fetch events of several tables under
the same db in batch. MetastoreEventsProcessor.MetaDataFilter is
improved for this.
Tests:
- Ran metadata/test_event_processing.py::TestEventSyncWaiting
Change-Id: Ic033b7e197cd19505653c3ff80c4857cc474bcfc
---
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
6 files changed, 303 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/22571/1
--
To view, visit http://gerrit.cloudera.org:8080/22571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic033b7e197cd19505653c3ff80c4857cc474bcfc
Gerrit-Change-Number: 22571
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>
