[
https://issues.apache.org/jira/browse/HIVE-28669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng resolved HIVE-28669.
--------------------------------
Resolution: Fixed
Fix has been pushed to master. Thank you [~Aggarwal_Raghav] [~Indhumathi27] and
[~dkuzmenko] for the review!
> Deadlock found when TxnStoreMutex trying to acquireLock
> -------------------------------------------------------
>
> Key: HIVE-28669
> URL: https://issues.apache.org/jira/browse/HIVE-28669
> Project: Hive
> Issue Type: Bug
> Reporter: Zhihua Deng
> Assignee: Zhihua Deng
> Priority: Major
> Labels: hive-4.1.0-must, pull-request-available
> Fix For: 4.1.0
>
>
> Build from the latest master, use the MySQL as back db to run the HMS. After
> a while, in HMS logs, we can see the deadlock exception:
> {noformat}
> 2024-12-16T09:21:39,677 ERROR [Metastore Scheduled Worker 13]
> service.AcidHouseKeeperService: Unexpected exception in thread: Metastore
> Scheduled Worker 13, message: Unable to lock HouseKeeper due to:
> PreparedStatementCallback; SQL [INSERT INTO "AUX_TABLE" ("MT_KEY1",
> "MT_KEY2") VALUES(?, 0)]; (conn=1315) Deadlock found when trying to get lock;
> try restarting transaction; nested exception is
> java.sql.SQLTransactionRollbackException: (conn=1315) Deadlock found when
> trying to get lock; try restarting transaction
> java.lang.RuntimeException: Unable to lock HouseKeeper due to:
> PreparedStatementCallback; SQL [INSERT INTO "AUX_TABLE" ("MT_KEY1",
> "MT_KEY2") VALUES(?, 0)]; (conn=1315) Deadlock found when trying to get lock;
> try restarting transaction; nested exception is
> java.sql.SQLTransactionRollbackException: (conn=1315) Deadlock found when
> trying to get lock; try restarting transaction
> at
> org.apache.hadoop.hive.metastore.txn.TxnStoreMutex.acquireLock(TxnStoreMutex.java:85)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.txn.service.AcidHouseKeeperService.run(AcidHouseKeeperService.java:83)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_292]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> ~[?:1.8.0_292]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> ~[?:1.8.0_292]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> ~[?:1.8.0_292]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_292]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_292]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292]
> Caused by: org.springframework.dao.DeadlockLoserDataAccessException:
> PreparedStatementCallback; SQL [INSERT INTO "AUX_TABLE" ("MT_KEY1",
> "MT_KEY2") VALUES(?, 0)]; (conn=1315) Deadlock found when trying to get lock;
> try restarting transaction; nested exception is
> java.sql.SQLTransactionRollbackException: (conn=1315) Deadlock found when
> trying to get lock; try restarting transaction
> at
> org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:268)
> ~[spring-jdbc-5.3.39.jar:5.3.39]
> at
> org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73)
> ~[spring-jdbc-5.3.39.jar:5.3.39]
> at
> org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1575)
> ~[spring-jdbc-5.3.39.jar:5.3.39]
> at
> org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:667)
> ~[spring-jdbc-5.3.39.jar:5.3.39]
> at
> org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960)
> ~[spring-jdbc-5.3.39.jar:5.3.39]
> at
> org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:981)
> ~[spring-jdbc-5.3.39.jar:5.3.39]
> at
> org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.update(NamedParameterJdbcTemplate.java:331)
> ~[spring-jdbc-5.3.39.jar:5.3.39]
> at
> org.apache.hadoop.hive.metastore.txn.TxnStoreMutex.acquireLock(TxnStoreMutex.java:81)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> ... 8 more
> Caused by: java.sql.SQLTransactionRollbackException: (conn=1315) Deadlock
> found when trying to get lock; try restarting transaction
> at
> org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:245)
> ~[mariadb-java-client-2.5.0.jar:?]
> at
> org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.getException(ExceptionMapper.java:164)
> ~[mariadb-java-client-2.5.0.jar:?]
> at
> org.mariadb.jdbc.MariaDbStatement.executeExceptionEpilogue(MariaDbStatement.java:244)
> ~[mariadb-java-client-2.5.0.jar:?]
> at
> org.mariadb.jdbc.ClientSidePreparedStatement.executeInternal(ClientSidePreparedStatement.java:225)
> ~[mariadb-java-client-2.5.0.jar:?]
> at
> org.mariadb.jdbc.ClientSidePreparedStatement.execute(ClientSidePreparedStatement.java:145)
> ~[mariadb-java-client-2.5.0.jar:?]
> at
> org.mariadb.jdbc.ClientSidePreparedStatement.executeUpdate(ClientSidePreparedStatement.java:176)
> ~[mariadb-java-client-2.5.0.jar:?]
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:965)
> ~[spring-jdbc-5.3.39.jar:5.3.39]2024-12-16T09:21:39,677 ERROR [Metastore
> Scheduled Worker 13] service.AcidHouseKeeperService: Unexpected exception in
> thread: Metastore Scheduled Worker 13, message: Unable to lock HouseKeeper
> due to: PreparedStatementCallback; SQL [INSERT INTO "AUX_TABLE" ("MT_KEY1",
> "MT_KEY2") VALUES(?, 0)]; (conn=1315) Deadlock found when trying to get lock;
> try restarting transaction; nested exception is
> java.sql.SQLTransactionRollbackException: (conn=1315) Deadlock found when
> trying to get lock; try restarting transaction
> java.lang.RuntimeException: Unable to lock HouseKeeper due to:
> PreparedStatementCallback; SQL [INSERT INTO "AUX_TABLE" ("MT_KEY1",
> "MT_KEY2") VALUES(?, 0)]; (conn=1315) Deadlock found when trying to get lock;
> try restarting transaction; nested exception is
> java.sql.SQLTransactionRollbackException: (conn=1315) Deadlock found when
> trying to get lock; try restarting transaction
> at
> org.apache.hadoop.hive.metastore.txn.TxnStoreMutex.acquireLock(TxnStoreMutex.java:85)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.txn.service.AcidHouseKeeperService.run(AcidHouseKeeperService.java:83)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_292]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> ~[?:1.8.0_292]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> ~[?:1.8.0_292]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> ~[?:1.8.0_292]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_292]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_292]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292]
> Caused by: org.springframework.dao.DeadlockLoserDataAccessException:
> PreparedStatementCallback; SQL [INSERT INTO "AUX_TABLE" ("MT_KEY1",
> "MT_KEY2") VALUES(?, 0)]; (conn=1315) Deadlock found when trying to get lock;
> try restarting transaction; nested exception is
> java.sql.SQLTransactionRollbackException: (conn=1315) Deadlock found when
> trying to get lock; try restarting transaction
> at
> org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:268)
> ~[spring-jdbc-5.3.39.jar:5.3.39]
> at
> org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73)
> ~[spring-jdbc-5.3.39.jar:5.3.39]
> at
> org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1575)
> ~[spring-jdbc-5.3.39.jar:5.3.39]
> at
> org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:667)
> ~[spring-jdbc-5.3.39.jar:5.3.39]
> at
> org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960)
> ~[spring-jdbc-5.3.39.jar:5.3.39]
> at
> org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:981)
> ~[spring-jdbc-5.3.39.jar:5.3.39]
> at
> org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.update(NamedParameterJdbcTemplate.java:331)
> ~[spring-jdbc-5.3.39.jar:5.3.39]
> at
> org.apache.hadoop.hive.metastore.txn.TxnStoreMutex.acquireLock(TxnStoreMutex.java:81)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> ... 8 more
> Caused by: java.sql.SQLTransactionRollbackException: (conn=1315) Deadlock
> found when trying to get lock; try restarting transaction
> at
> org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:245)
> ~[mariadb-java-client-2.5.0.jar:?]
> at
> org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.getException(ExceptionMapper.java:164)
> ~[mariadb-java-client-2.5.0.jar:?]
> at
> org.mariadb.jdbc.MariaDbStatement.executeExceptionEpilogue(MariaDbStatement.java:244)
> ~[mariadb-java-client-2.5.0.jar:?]
> at
> org.mariadb.jdbc.ClientSidePreparedStatement.executeInternal(ClientSidePreparedStatement.java:225)
> ~[mariadb-java-client-2.5.0.jar:?]
> at
> org.mariadb.jdbc.ClientSidePreparedStatement.execute(ClientSidePreparedStatement.java:145)
> ~[mariadb-java-client-2.5.0.jar:?]
> at
> org.mariadb.jdbc.ClientSidePreparedStatement.executeUpdate(ClientSidePreparedStatement.java:176)
> ~[mariadb-java-client-2.5.0.jar:?]
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:965)
> ~[spring-jdbc-5.3.39.jar:5.3.39]
> ...{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
