Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22571
to look at the new patch set (#2).
Change subject: IMPALA-13684: Improve waitForHmsEvent() to only wait for
related events
......................................................................
IMPALA-13684: Improve waitForHmsEvent() to only wait for related events
waitForHmsEvent is a catalogd RPC for coordinators to send a requested
db/table names to catalogd and wait until it's safe (i.e. no stale
metadata) to start analyze the statement. Currently, catalogd waits
until it syncs to the latest HMS event regardless what the query is.
This patch improves this by only checking related events. In the ideal
case, if there are no pending events that are related, query doesn't
need to wait.
We check the pending events in the following way:
- For queries that need the db list, i.e. SHOW DATABASES, check pending
CREATE_DATABASE, DROP_DATABASE events on all dbs.
- For db statements like SHOW FUNCTIONS, CREATE/ALTER/DROP DATABASE,
check pending CREATE/ALTER/DROP events on that db. For db names that
are used in the query and are missing in catalog, also check their
events.
- For db statements that require the table list, i.e. SHOW TABLES,
also check CREATE_TABLE, DROP_TABLE events under that db.
- For loaded transactional tables, check all the pending COMMIT_TXN,
ABORT_TXN events.
- For all the table names,
- if they are all missing/unloaded in the catalog, check all the
pending CREATE_TABLE, DROP_TABLE events on them for their
existence.
- Otherwise, some of them are loaded, check all the table events on
them.
This patch leverages the HMS API to fetch events of several tables under
the same db in batch. MetastoreEventsProcessor.MetaDataFilter is
improved for this.
Tests:
- Ran metadata/test_event_processing.py::TestEventSyncWaiting and
custom_cluster/test_kudu.py::TestKuduHMSIntegration::test_drop_managed_kudu_table
Change-Id: Ic033b7e197cd19505653c3ff80c4857cc474bcfc
---
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
6 files changed, 248 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/22571/2
--
To view, visit http://gerrit.cloudera.org:8080/22571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic033b7e197cd19505653c3ff80c4857cc474bcfc
Gerrit-Change-Number: 22571
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
