Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22789 )
Change subject: IMPALA-13631: alterTableOrViewRename shouldn't hold catalog versionLock during external RPCs ...................................................................... IMPALA-13631: alterTableOrViewRename shouldn't hold catalog versionLock during external RPCs Catalog versionLock is a lock used to synchronize reads/writes of catalogVersion. It can be used to perform atomic bulk catalog operations since catalogVersion cannot change externally while the lock is being held. All other catalog operations will be blocked if the current thread holds the lock. So it shouldn't be held for a long time, especially when the current thread is invoking external RPCs for a table. CatalogOpExecutor.alterTable() is one place that could hold the lock for a long time. If the ALTER operation is a RENAME, it holds the lock until alterTableOrViewRename() finishes. HMS RPCs are invoked in this method to perform the operation, which might take an unpredictive time. The motivation of holding this lock is that RENAME is implemented as an DROP + ADD in the catalog cache. So this operation can be atomic. However, that doesn't mean we need the lock before operating the cache in CatalogServiceCatalog.renameTable(). We actually acquires the lock again in this method. So no need to keep holding the lock when invoking HMS RPCs. This patch removes holding the lock in alterTableOrViewRename(). Tests - Added e2e test for concurrent rename operations. - Also added some rename operations in test_concurrent_ddls.py Change-Id: Ie5f443b1e167d96024b717ce70ca542d7930cb4b Reviewed-on: http://gerrit.cloudera.org:8080/22789 Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Reviewed-by: Riza Suminto <riza.sumi...@cloudera.com> Reviewed-by: Joe McDonnell <joemcdonn...@cloudera.com> Reviewed-by: Michael Smith <michael.sm...@cloudera.com> --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/custom_cluster/test_concurrent_ddls.py M tests/metadata/test_ddl.py 3 files changed, 65 insertions(+), 15 deletions(-) Approvals: Impala Public Jenkins: Verified Riza Suminto: Looks good to me, approved Joe McDonnell: Looks good to me, but someone else must approve Michael Smith: Looks good to me, but someone else must approve -- To view, visit http://gerrit.cloudera.org:8080/22789 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie5f443b1e167d96024b717ce70ca542d7930cb4b Gerrit-Change-Number: 22789 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Michael Smith <michael.sm...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com>
