[
https://issues.apache.org/jira/browse/IMPALA-13842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17935692#comment-17935692
]
ASF subversion and git services commented on IMPALA-13842:
----------------------------------------------------------
Commit 5e4f882203ba07c467ac1ae2feb5420c0abc9936 in impala's branch
refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=5e4f88220 ]
IMPALA-13842: Enable graceful shutdown for test_redaction
test_redation in both test_query_live.py and test_query_log.py enables
workload management feature, but not followed by enabling graceful
shutdown. As consequence, if test cluster is shutting down, HMS lock
might still held and not unlocked properly. Other tests that want to
write to sys.impala_query_log will not able to obtain the HMS lock
needed. This patch fix the issue by enabling graceful shutdown for
test_redaction.
Testing:
- Run and pass the tests.
Change-Id: Iefabe07b9d32fbc74957bf3fa1cb8445ad7e2b71
Reviewed-on: http://gerrit.cloudera.org:8080/22621
Reviewed-by: Jason Fehr <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Test against sys.impala_query_log should wait until coordinator quiesce.
> ------------------------------------------------------------------------
>
> Key: IMPALA-13842
> URL: https://issues.apache.org/jira/browse/IMPALA-13842
> Project: IMPALA
> Issue Type: Bug
> Components: Test
> Reporter: Riza Suminto
> Assignee: Jason Fehr
> Priority: Major
> Labels: broken-build
> Fix For: Impala 5.0.0
>
>
> We are seeing evidence of HMS deadlock for table sys.impala_query_log in
> downstream build.
> {noformat}
> I0307 02:24:06.992715 18405 SnapshotProducer.java:422]
> 5f4ecf63310cb7f5:e2db615800000000] Committed snapshot 4325155866837015464
> (MergeAppend)
> I0307 02:24:07.002012 18405 LoggingMetricsReporter.java:38]
> 5f4ecf63310cb7f5:e2db615800000000] Received metrics report:
> CommitReport{tableName=ImpalaHiveCatalog.sys.impala_query_log,
> snapshotId=4325155866837015464, sequenceNumber=9, operation=append,
> commitMetrics=CommitMetricsResult{totalDuration=TimerResult{timeUnit=NANOSECONDS,
> totalDuration=PT0.241321564S, count=1}, attempts=CounterResult{unit=COUNT,
> value=1}, addedDataFiles=CounterResult{unit=COUNT, value=1},
> removedDataFiles=null, totalDataFiles=CounterResult{unit=COUNT, value=9},
> addedDeleteFiles=null, addedEqualityDeleteFiles=null,
> addedPositionalDeleteFiles=null, removedDeleteFiles=null,
> removedEqualityDeleteFiles=null, removedPositionalDeleteFiles=null,
> totalDeleteFiles=CounterResult{unit=COUNT, value=0},
> addedRecords=CounterResult{unit=COUNT, value=1}, removedRecords=null,
> totalRecords=CounterResult{unit=COUNT, value=41},
> addedFilesSizeInBytes=CounterResult{unit=BYTES, value=26244},
> removedFilesSizeInBytes=null, totalFilesSizeInBytes=CounterResult{unit=BYTES,
> value=273450}, addedPositionalDeletes=null, removedPositionalDeletes=null,
> totalPositionalDeletes=CounterResult{unit=COUNT, value=0},
> addedEqualityDeletes=null, removedEqualityDeletes=null,
> totalEqualityDeletes=CounterResult{unit=COUNT, value=0}},
> metadata={iceberg-version=Apache Iceberg 1.5.2.2025.0.20.0-45 (commit
> f2db4a27a43aaec790af8b9e2ac51f4a50b26260)}}
> I0307 02:24:07.002182 18405 CatalogServiceCatalog.java:1327]
> 5f4ecf63310cb7f5:e2db615800000000] Added catalog version 2126 in table's
> sys.impala_query_log in-flight events
> W0307 02:24:07.053566 18405 Tasks.java:456]
> 5f4ecf63310cb7f5:e2db615800000000] Retrying task after failure: Waiting for
> lock on table sys.impala_query_log
> Java exception follows:
> org.apache.iceberg.hive.MetastoreLock$WaitingForLockException: Waiting for
> lock on table sys.impala_query_log
> at
> org.apache.iceberg.hive.MetastoreLock.lambda$acquireLock$1(MetastoreLock.java:217)
> at
> org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
> at
> org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
> at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
> at
> org.apache.iceberg.hive.MetastoreLock.acquireLock(MetastoreLock.java:209)
> at org.apache.iceberg.hive.MetastoreLock.lock(MetastoreLock.java:146)
> at
> org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:182)
> at
> org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:135)
> at
> org.apache.iceberg.BaseTransaction.lambda$commitSimpleTransaction$3(BaseTransaction.java:427)
> at
> org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
> at
> org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
> at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
> at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
> at
> org.apache.iceberg.BaseTransaction.commitSimpleTransaction(BaseTransaction.java:423)
> at
> org.apache.iceberg.BaseTransaction.commitTransaction(BaseTransaction.java:318)
> at
> org.apache.impala.service.CatalogOpExecutor.insertIntoIcebergTable(CatalogOpExecutor.java:7593)
> at
> org.apache.impala.service.CatalogOpExecutor.updateCatalogImpl(CatalogOpExecutor.java:7386)
> at
> org.apache.impala.service.CatalogOpExecutor.updateCatalog(CatalogOpExecutor.java:7227)
> at
> org.apache.impala.service.JniCatalog.lambda$updateCatalog$15(JniCatalog.java:510)
> at
> org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
> at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:100)
> at
> org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:251)
> at
> org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:265)
> at
> org.apache.impala.service.JniCatalog.updateCatalog(JniCatalog.java:509)
> ...
> W0307 02:25:49.672634 18405 Tasks.java:456]
> 5f4ecf63310cb7f5:e2db615800000000] Retrying task after failure: Waiting for
> lock on table sys.impala_query_log{noformat}
> This might be related to IMPALA-13772 that disable FETCH_ROWS_TIMEOUT_MS
> option.
> During custom cluster test run, impala minicluster is restart between tests.
> If test does not wait until coordinator quiesce (all pending INSERT INTO
> sys.impala_query_log is done), it might be the case that HMS lock is being
> hold when CatalogD terminates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
