[
https://issues.apache.org/jira/browse/IMPALA-7670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17949570#comment-17949570
]
Michael Smith edited comment on IMPALA-7670 at 5/5/25 8:25 PM:
---------------------------------------------------------------
The code here has evolved, but I think the underlying issue is the same. The
partition map is updated by dropPartition while also attempting to construct a
table representation from the partition map. The current stack trace (using
[55804f78|https://github.com/apache/impala/commit/55804f78740bf5e05aaee2634dec249fefb431a0]
shows up as
{code}
I0505 12:38:37.879602 207596 JniUtil.java:167]
3545f3d50f22f2cf:eaaa4cf500000000] resetMetadata request: INVALIDATE TABLE
test_alter_table_7bf47d26.i1670A_alter_1 issued by msmith2
I0505 12:38:37.881371 207596 CatalogServiceCatalog.java:3011]
3545f3d50f22f2cf:eaaa4cf500000000] Invalidating table metadata:
test_alter_table_7bf47d26.i1670A_alter_1
I0505 12:38:37.887961 206939 catalog-server.cc:735] A catalog update with 3
entries is assembled. Catalog version: 3681 Last sent catalog version: 3677
I0505 12:38:37.888170 206927 JniUtil.java:167] getCatalogDelta request: Getting
catalog delta from version 3681
I0505 12:38:37.890123 206927 CatalogServiceCatalog.java:1826] Skipped 0
partitions of table test_alter_table_7bf47d26.i1670a_alter_1 in the incremental
update
I0505 12:38:37.890344 206927 CatalogServiceCatalog.java:1839] Collected 3
partition update(s):
1:HDFS_PARTITION:test_alter_table_7bf47d26.i1670a_alter_1:(x=1,x=2,x=3),
version=3686, original size=(avg=555, min=555, max=556, sum=1667), compressed
size=(avg=464, min=461, max=470, sum=1394)
I0505 12:38:37.890429 206927 catalog-server.cc:1419] Collected update:
1:TABLE:test_alter_table_7bf47d26.i1670a_alter_1, version=3687, original
size=1999, compressed size=1087
E0505 12:38:37.890707 207596 JniUtil.java:184]
3545f3d50f22f2cf:eaaa4cf500000000] Error in INVALIDATE TABLE
test_alter_table_7bf47d26.i1670A_alter_1 issued by msmith2. Time spent: 11ms
I0505 12:38:37.890842 207596 jni-util.cc:321]
3545f3d50f22f2cf:eaaa4cf500000000] java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1469)
at java.util.HashMap$KeyIterator.next(HashMap.java:1493)
at
org.apache.impala.catalog.HdfsTable.toMinimalTCatalogObject(HdfsTable.java:2218)
at
org.apache.impala.catalog.CatalogServiceCatalog.addIncompleteTable(CatalogServiceCatalog.java:2457)
at
org.apache.impala.catalog.CatalogServiceCatalog.invalidateTable(CatalogServiceCatalog.java:3075)
at
org.apache.impala.service.CatalogOpExecutor.execResetMetadataImpl(CatalogOpExecutor.java:7099)
at
org.apache.impala.service.CatalogOpExecutor.execResetMetadata(CatalogOpExecutor.java:7028)
at
org.apache.impala.service.JniCatalog.lambda$resetMetadata$4(JniCatalog.java:326)
at
org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
at
org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
at
org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:100)
at
org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:242)
at
org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:256)
at
org.apache.impala.service.JniCatalog.resetMetadata(JniCatalog.java:325)
{code}
IMPALA-12831 added a table lock on HdfsTable.toMinimalTCatalogObject, but other
methods are updating {{droppedPartitions_}} without the table lock.
I encountered this issue in TestDdlStatements.test_alter_table while running
tests/metadata/test_ddl.py in a loop (with {{-n 12}} for concurrency).
was (Author: JIRAUSER288956):
The code here has evolved, but I think the underlying issue is the same. The
partition map is updated by dropPartition while also attempting to construct a
table representation from the partition map. The current stack trace (using
[55804f78|https://github.com/apache/impala/commit/55804f78740bf5e05aaee2634dec249fefb431a0]
shows up as
{code}
I0505 12:38:37.879602 207596 JniUtil.java:167]
3545f3d50f22f2cf:eaaa4cf500000000] resetMetadata request: INVALIDATE TABLE
test_alter_table_7bf47d26.i1670A_alter_1 issued by msmith2
I0505 12:38:37.881371 207596 CatalogServiceCatalog.java:3011]
3545f3d50f22f2cf:eaaa4cf500000000] Invalidating table metadata:
test_alter_table_7bf47d26.i1670A_alter_1
I0505 12:38:37.887961 206939 catalog-server.cc:735] A catalog update with 3
entries is assembled. Catalog version: 3681 Last sent catalog version: 3677
I0505 12:38:37.888170 206927 JniUtil.java:167] getCatalogDelta request: Getting
catalog delta from version 3681
I0505 12:38:37.890123 206927 CatalogServiceCatalog.java:1826] Skipped 0
partitions of table test_alter_table_7bf47d26.i1670a_alter_1 in the incremental
update
I0505 12:38:37.890344 206927 CatalogServiceCatalog.java:1839] Collected 3
partition update(s):
1:HDFS_PARTITION:test_alter_table_7bf47d26.i1670a_alter_1:(x=1,x=2,x=3),
version=3686, original size=(avg=555, min=555, max=556, sum=1667), compressed
size=(avg=464, min=461, max=470, sum=1394)
I0505 12:38:37.890429 206927 catalog-server.cc:1419] Collected update:
1:TABLE:test_alter_table_7bf47d26.i1670a_alter_1, version=3687, original
size=1999, compressed size=1087
E0505 12:38:37.890707 207596 JniUtil.java:184]
3545f3d50f22f2cf:eaaa4cf500000000] Error in INVALIDATE TABLE
test_alter_table_7bf47d26.i1670A_alter_1 issued by msmith2. Time spent: 11ms
I0505 12:38:37.890842 207596 jni-util.cc:321]
3545f3d50f22f2cf:eaaa4cf500000000] java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1469)
at java.util.HashMap$KeyIterator.next(HashMap.java:1493)
at
org.apache.impala.catalog.HdfsTable.toMinimalTCatalogObject(HdfsTable.java:2218)
at
org.apache.impala.catalog.CatalogServiceCatalog.addIncompleteTable(CatalogServiceCatalog.java:2457)
at
org.apache.impala.catalog.CatalogServiceCatalog.invalidateTable(CatalogServiceCatalog.java:3075)
at
org.apache.impala.service.CatalogOpExecutor.execResetMetadataImpl(CatalogOpExecutor.java:7099)
at
org.apache.impala.service.CatalogOpExecutor.execResetMetadata(CatalogOpExecutor.java:7028)
at
org.apache.impala.service.JniCatalog.lambda$resetMetadata$4(JniCatalog.java:326)
at
org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
at
org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
at
org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:100)
at
org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:242)
at
org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:256)
at
org.apache.impala.service.JniCatalog.resetMetadata(JniCatalog.java:325)
{code}
IMPALA-12831 added a table lock on HdfsTable.toMinimalTCatalogObject, but other
methods are updating {{droppedPartitions_}} without the table lock.
> Drop table with a concurrent refresh throws ConcurrentModificationException
> ---------------------------------------------------------------------------
>
> Key: IMPALA-7670
> URL: https://issues.apache.org/jira/browse/IMPALA-7670
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 3.1.0
> Reporter: Bharath Vissapragada
> Assignee: Michael Smith
> Priority: Critical
> Fix For: Not Applicable
>
>
> * This bug was found on a V2 Catalog and probably also applies to V1.
> Saw this in the Catalog server.
> {noformat}
> I1004 16:38:55.236702 85380 jni-util.cc:308]
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
> at java.util.HashMap$ValueIterator.next(HashMap.java:1471)
> at
> org.apache.impala.catalog.FeFsTable$Utils.getPartitionFromThriftPartitionSpec(FeFsTable.java:407)
> at
> org.apache.impala.catalog.HdfsTable.getPartitionFromThriftPartitionSpec(HdfsTable.java:694)
> at
> org.apache.impala.catalog.Catalog.getHdfsPartition(Catalog.java:407)
> at
> org.apache.impala.catalog.Catalog.getHdfsPartition(Catalog.java:386)
> at
> org.apache.impala.service.CatalogOpExecutor.bulkAlterPartitions(CatalogOpExecutor.java:3193)
> at
> org.apache.impala.service.CatalogOpExecutor.dropTableStats(CatalogOpExecutor.java:1255)
> at
> org.apache.impala.service.CatalogOpExecutor.dropStats(CatalogOpExecutor.java:1148)
> at
> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:301)
> at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:157)
> {noformat}
> Still need to dig into it, but seems like something is off with locking
> somewhere.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
