[
https://issues.apache.org/jira/browse/HIVE-28659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17905565#comment-17905565
]
Simhadri Govindappa commented on HIVE-28659:
--------------------------------------------
This seems to be a genuine Test failure .
I was trying to start Metastore with MYSQL using the latest master branch.
But the metastore is failing to start due to deadlock . (PS: I don't see this
issue in hive 4.0.1. ) .
Please find the stack trace below.
{noformat}
2024-12-13T14:05:31,949 INFO [Metastore Scheduled Worker 6]
metastore.HMSHandler: Created RawStore:
org.apache.hadoop.hive.metastore.ObjectStore@5bd5b219
2024-12-13T14:05:32,935 INFO [Metastore Scheduled Worker 8]
service.AcidOpenTxnsCounterService: Open txn counter ran for 0 seconds.
isAliveCounter: 61
2024-12-13T14:05:41,934 INFO [Metastore Scheduled Worker 8]
txn.CompactionTxnHandler: Start to clean empty aborted or committed TXNS
2024-12-13T14:05:51,933 INFO [Metastore Scheduled Worker 9]
txn.CompactionTxnHandler: Start to clean empty aborted or committed TXNS
2024-12-13T14:06:01,932 INFO [Metastore Scheduled Worker 6]
txn.CompactionTxnHandler: Start to clean empty aborted or committed TXNS
2024-12-13T14:06:11,930 INFO [Metastore Scheduled Worker 12]
txn.CompactionTxnHandler: Start to clean empty aborted or committed TXNS
2024-12-13T14:06:21,933 INFO [Metastore Scheduled Worker 11]
txn.CompactionTxnHandler: Start to clean empty aborted or committed TXNS
2024-12-13T14:06:31,931 INFO [Metastore Scheduled Worker 2]
service.AcidHouseKeeperService: Starting to run AcidHouseKeeperService
2024-12-13T14:06:31,934 INFO [Metastore Scheduled Worker 9]
txn.CompactionTxnHandler: Start to clean empty aborted or committed TXNS
2024-12-13T14:06:31,938 INFO [Metastore Scheduled Worker 2]
functions.CleanTxnToWriteIdTableFunction: Removed 0 rows from TXN_TO_WRITE_ID
with Txn Low-Water-Mark: 1
2024-12-13T14:06:31,947 INFO [Metastore Scheduled Worker 12]
metastore.HMSHandler: Opening raw store with implementation class:
org.apache.hadoop.hive.metastore.ObjectStore
2024-12-13T14:06:31,947 INFO [Metastore Scheduled Worker 12]
metastore.ObjectStore: RawStore:
org.apache.hadoop.hive.metastore.ObjectStore@565c37e, with PersistenceManager:
null will be shutdown
2024-12-13T14:06:31,947 INFO [Metastore Scheduled Worker 12]
metastore.ObjectStore: RawStore:
org.apache.hadoop.hive.metastore.ObjectStore@565c37e, with PersistenceManager:
org.datanucleus.api.jdo.JDOPersistenceManager@684f4e2 created in the thread
with id: 51
2024-12-13T14:06:31,949 INFO [Metastore Scheduled Worker 12]
metastore.HMSHandler: Created RawStore:
org.apache.hadoop.hive.metastore.ObjectStore@565c37e
2024-12-13T14:06:33,931 INFO [Metastore Scheduled Worker 3]
service.AcidOpenTxnsCounterService: Open txn counter ran for 0 seconds.
isAliveCounter: 122
2024-12-13T14:06:41,933 INFO [Metastore Scheduled Worker 3]
txn.CompactionTxnHandler: Start to clean empty aborted or committed TXNS
2024-12-13T14:06:51,933 INFO [Metastore Scheduled Worker 1]
txn.CompactionTxnHandler: Start to clean empty aborted or committed TXNS
2024-12-13T14:07:01,926 INFO [Metastore Scheduled Worker 9]
txn.CompactionTxnHandler: Start to clean empty aborted or committed TXNS
2024-12-13T14:07:02,069 ERROR [Metastore Scheduled Worker 2]
metastore.MaterializationsRebuildLockCleanerTask: Unexpected error in thread:
Metastore Scheduled Worker 2, message: Unable to lock MaterializationRebuild
due to: PreparedStatementCallback; SQL [INSERT INTO "AUX_TABLE" ("MT_KEY1",
"MT_KEY2") VALUES(?, 0)]; Deadlock found when trying to get lock; try
restarting transaction; nested exception is
com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock found
when trying to get lock; try restarting transaction
java.lang.RuntimeException: Unable to lock MaterializationRebuild due to:
PreparedStatementCallback; SQL [INSERT INTO "AUX_TABLE" ("MT_KEY1", "MT_KEY2")
VALUES(?, 0)]; Deadlock found when trying to get lock; try restarting
transaction; nested exception is
com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock found
when trying to get lock; try restarting transaction
at
org.apache.hadoop.hive.metastore.txn.TxnStoreMutex.acquireLock(TxnStoreMutex.java:85)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.MaterializationsRebuildLockCleanerTask.run(MaterializationsRebuildLockCleanerTask.java:65)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_362]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
~[?:1.8.0_362]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
~[?:1.8.0_362]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
~[?:1.8.0_362]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_362]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: org.springframework.dao.DeadlockLoserDataAccessException:
PreparedStatementCallback; SQL [INSERT INTO "AUX_TABLE" ("MT_KEY1", "MT_KEY2")
VALUES(?, 0)]; Deadlock found when trying to get lock; try restarting
transaction; nested exception is
com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock found
when trying to get lock; try restarting transaction
at
org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:268)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1575)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:667)
~[spring-jdbc-5.3.39.jar:5.3.39]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960)
~[spring-jdbc-5.3.39.jar:5.3.39]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:981)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.update(NamedParameterJdbcTemplate.java:331)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.apache.hadoop.hive.metastore.txn.TxnStoreMutex.acquireLock(TxnStoreMutex.java:81)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
... 8 more
Caused by: com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException:
Deadlock found when trying to get lock; try restarting transaction
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:123)
~[mysql-connector-java-8.0.28.jar:8.0.28]
at
com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
~[mysql-connector-java-8.0.28.jar:8.0.28]
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953)
~[mysql-connector-java-8.0.28.jar:8.0.28]
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1098)
~[mysql-connector-java-8.0.28.jar:8.0.28]
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1046)
~[mysql-connector-java-8.0.28.jar:8.0.28]
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1371)
~[mysql-connector-java-8.0.28.jar:8.0.28]
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1031)
~[mysql-connector-java-8.0.28.jar:8.0.28]
at
org.apache.hive.com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at
org.apache.hive.com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at
org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:965)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:648)
~[spring-jdbc-5.3.39.jar:5.3.39]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960)
~[spring-jdbc-5.3.39.jar:5.3.39]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:981)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.update(NamedParameterJdbcTemplate.java:331)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.apache.hadoop.hive.metastore.txn.TxnStoreMutex.acquireLock(TxnStoreMutex.java:81)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
... 8 more{noformat}
> Fix the flaky test TestMetaStoreDeadlock
> ----------------------------------------
>
> Key: HIVE-28659
> URL: https://issues.apache.org/jira/browse/HIVE-28659
> Project: Hive
> Issue Type: Test
> Security Level: Public(Viewable by anyone)
> Reporter: Zhihua Deng
> Assignee: Zhihua Deng
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
