Hello Riza Suminto, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22794
to look at the new patch set (#3).
Change subject: IMPALA-13974: Don't check catName if fetching
COMMIT_COMPACTION_EVENT or ALLOC_WRITE_ID_EVENT
......................................................................
IMPALA-13974: Don't check catName if fetching COMMIT_COMPACTION_EVENT or
ALLOC_WRITE_ID_EVENT
COMMIT_COMPACTION_EVENT is one kind of HMS notification events that are
generated when a transactional table finishes a compaction.
ALLOC_WRITE_ID_EVENT is another kind of transactional table events
generated when a data modification, e.g. INSERT, starts.
Due to the bug of HIVE-28912, the catName of them is always NULL. Note
that catName is an optional field for all kinds of HMS events:
struct NotificationEvent {
1: required i64 eventId,
2: required i32 eventTime,
3: required string eventType,
4: optional string dbName,
5: optional string tableName,
6: required string message,
7: optional string messageFormat,
8: optional string catName
}
It's OK to be NULL for some global events like OPEN_TXN, COMMIT_TXN,
ABORT_TXN, etc. But for table level events, catName should be the
catalog name of the table. Usually, it's the default catalog, 'hive'.
When checking COMMIT_COMPACTION_EVENT and ALLOC_WRITE_ID_EVENT events on
a table, we shouldn't set the filter on catName. Otherwise, they will
all be skipped due to this bug.
There are two filters we use in fetching HMS events. One is the server
side filter we set in NotificationEventRequest. The other one is the
client side filter, NotificationFilter, used in filtering events fetched
from HMS.
This patch removes the check on catName in these filters to avoid
missing COMMIT_COMPACTION_EVENT and ALLOC_WRITE_ID_EVENT.
MetastoreEventsProcessorTest.testNotificationEventRequest() has some
wrong expected numbers due to this bug. This patch fixes them and
refactors the code to be more readable.
Tests:
- Ran test_hms_event_sync_with_commit_compaction 20 times. Without the
fix, it fails in 2-3 runs.
- Ran MetastoreEventsProcessorTest.testNotificationEventRequest().
Change-Id: I2e182d32ee2bb8a69c4f71c05eb9e87a5a115f24
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
3 files changed, 45 insertions(+), 24 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/22794/3
--
To view, visit http://gerrit.cloudera.org:8080/22794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2e182d32ee2bb8a69c4f71c05eb9e87a5a115f24
Gerrit-Change-Number: 22794
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com>
