Hello Quanlong Huang, k.venureddy2...@gmail.com, Csaba Ringhofer, Impala Public
Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21175
to look at the new patch set (#13).
Change subject: IMPALA-12829: Skip processing transaction events if the table
is HMS sync disabled.
......................................................................
IMPALA-12829: Skip processing transaction events if the table is
HMS sync disabled.
For transactional tables, the event processor is not skipping abort_txn
and commit_txn_event if the database/table is HMS sync disabled. This
processing is unnecessary and helps to improve event processor lag by
skipping abort_txn, and commit_txn events if the corresponding database
or transactional tables are HMS sync disabled. The database name and
table name is present for the Alloc_write_id_event, skipping this event
is already implemented if HMS sync is disabled. Since dbname and table
name is not present for the abort_txn and commit_txn events, we need to
check if HMS sync is disabled on the HMS table property when the table
object is extracted in the CatalogServiceCatalog#addWriteIdsToTable().
Also, fixed the partitions and table refreshed metrics for CommitTxn
event.
Additional Issues discovered during testing:
1) CatalogServiceCatalog#reloadTableIfExists() didn't verify if the
current eventId is older than the table's lastSyncEventId which leads to
unecessary reloading of table for commit txns.
2) Insert queries from impala didn't update the validWriteIdList for
transactional tables in the cache, so CommitTxn events triggered by
insert events are triggering reload on unpartitioned transactional
tables again while consuming these CommitTxn events. Fixed it by
updating the validWriteIdList in the cache.
Testing:
- Added end-to-end tests to verify transaction events are skipped.
Change-Id: I5d0ecb3b756755bc04c66a538a9ae6b88011a019
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
5 files changed, 116 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/21175/13
--
To view, visit http://gerrit.cloudera.org:8080/21175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5d0ecb3b756755bc04c66a538a9ae6b88011a019
Gerrit-Change-Number: 21175
Gerrit-PatchSet: 13
Gerrit-Owner: Sai Hemanth Gantasala <saihema...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k.venureddy2...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <saihema...@cloudera.com>
