Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23382
Change subject: IMPALA-14400: Fix deadlock in CatalogServiceCatalog.getDbProperty() ...................................................................... IMPALA-14400: Fix deadlock in CatalogServiceCatalog.getDbProperty() IMPALA-13850 (part 4) modify CatalogServiceCatalog.getDb() to delay looking up catalog cache until initial reset() is complete. If EventProcessor can start processing event before reset() happen and obtain versionLock_.readLock() when calling CatalogServiceCatalog.getDbProperty(). Later on, it will hit deadlock when attempting to obtain versionLock_.writeLock() throgh getDb() / waitInitialResetCompletion(). This lock upgrade from read to write is unsafe. This patch mitigate the issue by adding CatalogServiceCatalog.getDbNoWait() that does not call waitInitialResetCompletion(). All calls to getDb() from inside CatalogServiceCatalog that begins with acquiring versionLock_.readLock() is replaced with getDbNoWait(). This is OK because nothing except EventProcessor interact with catalog cache concurrently. For EventProcessor itself, having getDbNoWait() returns null is also OK because reset() will pause EventProcessor and populate the cache soon after. Also remove acquireVersionReadLock() since there is only 1 callsite to it. Thus, lock acquisition and release and clearly shows within the method. Testing: Increase TRIGGER_RESET_METADATA_DELAY from 1s to 3s in test_metadata_after_failover_with_delayed_reset. It was easy to hit the deadlock with 3s delay before the patch. No more deadlock happen after the patch. Change-Id: I3162472ea9531add77886bf1d0d73460ff34d07a --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M tests/custom_cluster/test_catalogd_ha.py 2 files changed, 28 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/23382/1 -- To view, visit http://gerrit.cloudera.org:8080/23382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3162472ea9531add77886bf1d0d73460ff34d07a Gerrit-Change-Number: 23382 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto <[email protected]>
