[
https://issues.apache.org/jira/browse/IMPALA-14068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-14068:
------------------------------------
Attachment: catalogd_jstack_recover_partitions_blocks_ep.txt
> AlterTableRecoverPartitions blocks HMS event processor
> ------------------------------------------------------
>
> Key: IMPALA-14068
> URL: https://issues.apache.org/jira/browse/IMPALA-14068
> Project: IMPALA
> Issue Type: Bug
> Reporter: Quanlong Huang
> Priority: Major
> Attachments: catalogd_jstack_recover_partitions_blocks_ep.txt
>
>
> When running ALTER TABLE RECOVER PARTITIONS on a large table, HMS event
> processor is blocked in processing the AddPartition events of the table until
> the operation finishes. This introduces a significant lag in HMS event
> processing.
> Stacktrace of the EventProcessor:
> {noformat}
> "MetastoreEventsProcessor-ProcessEvents [Processing ADD_PARTITION on
> scale_4k_500cols_db.tbl1, eventId=85020] [Attempting to write lock table
> scale_4k_500cols_db.tbl1 with a timeout of 7200000 ms]" #72 daemon prio=5
> os_prio=0 tid=0x000000001255f000 nid=0x324a waiting on condition
> [0x00007fe393372000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.tryLock(CatalogServiceCatalog.java:617)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.tryWriteLock(CatalogServiceCatalog.java:513)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.tryWriteLock(CatalogServiceCatalog.java:520)
> at
> org.apache.impala.service.CatalogOpExecutor.tryWriteLock(CatalogOpExecutor.java:7602)
> at
> org.apache.impala.service.CatalogOpExecutor.addPartitionsIfNotRemovedLater(CatalogOpExecutor.java:4469)
> at
> org.apache.impala.catalog.events.MetastoreEvents$AddPartitionEvent.processTableEvent(MetastoreEvents.java:2479)
> at
> org.apache.impala.catalog.events.MetastoreEvents$MetastoreTableEvent.process(MetastoreEvents.java:1304)
> at
> org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:696)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1302)
> - locked <0x00007fe3ecb5a458> (a
> org.apache.impala.catalog.events.MetastoreEventsProcessor)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1045)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor$$Lambda$60/45320991.run(Unknown
> Source)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){noformat}
> Table lock is held by the AlterTableRecoverPartitions operation
> {noformat}
> "Thread-7783 [ALTER_TABLE scale_4k_500cols_db.tbl1 issued by
> [email protected]] [Recovering 4000 partitions for
> scale_4k_500cols_db.tbl1] [Loading file and block metadata for 4000 paths for
> table scale_4k_500cols_db.tbl1]" #44181 prio=5 os_prio=0
> tid=0x0000000010e22000 nid=0x5a6e waiting on condition [0x00007fe389713000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007feb087e57a8> (a
> java.util.concurrent.FutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at
> org.apache.impala.catalog.ParallelFileMetadataLoader.loadInternal(ParallelFileMetadataLoader.java:192)
> at
> org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:140)
> at
> org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:817)
> at
> org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:770)
> at
> org.apache.impala.catalog.HdfsTable.createAndLoadPartitions(HdfsTable.java:966)
> at
> org.apache.impala.service.CatalogOpExecutor.addHdfsPartitions(CatalogOpExecutor.java:1663)
> at
> org.apache.impala.service.CatalogOpExecutor.alterTableRecoverPartitions(CatalogOpExecutor.java:6026)
> at
> org.apache.impala.service.CatalogOpExecutor.alterTable(CatalogOpExecutor.java:1298)
> at
> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:454)
> at
> org.apache.impala.service.JniCatalog.lambda$execDdl$3(JniCatalog.java:302)
> at
> org.apache.impala.service.JniCatalog$$Lambda$105/494935334.call(Unknown
> Source)
> at
> org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
> at
> org.apache.impala.service.JniCatalogOp$$Lambda$67/903378472.call(Unknown
> Source)
> at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:100)
> at
> org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:231)
> at
> org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:245)
> at
> org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:301){noformat}
> We need to improve the self-event detection for such AddPartition events.
> CC [~VenuReddy], [~hemanth619]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
