Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21115
Change subject: WIP IMPALA-12831: Add lock for partition map in HdfsTable ...................................................................... WIP IMPALA-12831: Add lock for partition map in HdfsTable HdfsTable.toMinimalTCatalogObject() is not always invoked with holding the table lock, e.g. in invalidating a table, we could replace an HdfsTable instance with an IncompleteTable instance. We then invoke HdfsTable.toMinimalTCatalogObject() to get the removed catalog object. However, the HdfsTable instance could be modified in the meantime by a concurrent DDL/DML that would reload it, e.g. a REFRESH statement. This causes HdfsTable.toMinimalTCatalogObject() failed by ConcurrentModificationException on the column/partition list. There are two parts in HdfsTable.toMinimalTCatalogObject() that could be failed by concurrent modifications. One is adding the columns, the other is adding the partitions. The columns are not used in the result consumers so we can simply ignore them. To protect reading the partition map, this patch adds a ReentrantReadWriteLock in HdfsTable for it. Refactor some code paths to avoid holding the lock during external RPCs, e.g. loading file metadata. TODO: HdfsTable.getPartitions() returns the map values externally. To protect its usage without exposing the partitionMapLock outside HdfsTable, we have to make sure it's only used in impalad or tests. Currently, there are several code paths in catalogd that use it, e.g. FeFsTable.Utils.getPartitionFromThriftPartitionSpec() which is used in many catalogd code paths. As a workaround, a deep copy of the value list are returned in getPartitions(). Tests - Added e2e test - TODO: run CORE tests Change-Id: I7466c291e4619b1ceac325b84f1eadc598aef29e --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M tests/custom_cluster/test_concurrent_ddls.py 7 files changed, 358 insertions(+), 173 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21115/1 -- To view, visit http://gerrit.cloudera.org:8080/21115 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7466c291e4619b1ceac325b84f1eadc598aef29e Gerrit-Change-Number: 21115 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang <[email protected]>
