Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22793
Change subject: IMPALA-13829: Fix waitForHmsEvent response might miss deletions ...................................................................... IMPALA-13829: Fix waitForHmsEvent response might miss deletions When a db/table is removed in the catalog cache, catalogd assigns it a new catalog version and put it into the deleteLog which is used for the catalog update thread to collect deletion updates. Once the catalog update thread collects a range of updates, it triggers GC in the deleteLog to clear items older than the last sent catalog version. The deletions will be broadcasted by statestore to all the coordinators eventually. However, waitForHmsEvent requests is also a consumer of the deleteLog and could be impacted by these GCs. waitForHmsEvent is a catalogd RPC used by coordinators when a query wants to wait until the related metadata is in synced with HMS. The response of waitForHmsEvent returns the latest metadata including the deletions on related dbs/tables. If the related deletions in deleteLog is GCed just before the waitForHmsEvent request collects the results, they will be missing in the response. Coordinator might keep using stale metadata of non-existing dbs/tables. Though the deletions are GCed and can't be collected by the response, they have been sent to statestore and will arrive in the coordinator side eventually. This patch adds one more wait step in coordinator side to wait for their arrival. Implementation In catalogd side, after collecting results for the waitForHmsEvent request, set the last sent catalog version in the response. This is a catalog version before which all the updates have been sent to statestore. In coordinator side, after applying updates of the response, waits until the local catalog version is updated till that version. So all the possibly missing deletions must have arrived. Also removed some unused imports. Tests: - Added e2e test. Without the fix, it consistently fails when runs for 2-3 times. Change-Id: I3ff7bc4d8b47d2aabec0f1fe73a3ab39e8e407cb --- M be/src/service/fe-support.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogDeltaLog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/util/DebugUtils.java M tests/metadata/test_event_processing.py 9 files changed, 80 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/22793/1 -- To view, visit http://gerrit.cloudera.org:8080/22793 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3ff7bc4d8b47d2aabec0f1fe73a3ab39e8e407cb Gerrit-Change-Number: 22793 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>
