I see (in HikariCP-2.5.1.jar) so perhaps upgrading the library is an option.
public HikariConfig() {
this.dataSourceProperties = new Properties();
this.healthCheckProperties = new Properties();
this.minIdle = -1;
this.maxPoolSize = -1;
this.maxLifetime = MAX_LIFETIME;
this.connectionTimeout = CONNECTION_TIMEOUT;
this.validationTimeout = VALIDATION_TIMEOUT;
this.idleTimeout = IDLE_TIMEOUT;
this.isAutoCommit = true;
this.isInitializationFailFast = true;
String systemProp = System.getProperty("hikaricp.configurationFile");
if(systemProp != null) {
this.loadProperties(systemProp);
}
}
From: Igor Kuzmenko <f1she...@gmail.com>
Reply-To: "user@hive.apache.org" <user@hive.apache.org>
Date: Saturday, March 25, 2017 at 5:05 PM
To: "user@hive.apache.org" <user@hive.apache.org>
Subject: Re: Hive TxnHandler::lock method run into dead lock.
Hi, Eugene.
I've tried hicaricp, it didn't work either. Hikari pool has same 10
maxConnections limit. When creating pool there's no explicit set of max pool
size and in HikariConfig constructor it's hardcoded to value of 10:
public HikariConfig()
{
dataSourceProperties = new Properties();
connectionTimeout = CONNECTION_TIMEOUT;
idleTimeout = IDLE_TIMEOUT;
isAutoCommit = true;
isJdbc4connectionTest = true;
minIdle = -1;
maxPoolSize = 10;
maxLifetime = MAX_LIFETIME;
isRecordMetrics = false;
transactionIsolation = -1;
metricsTrackerClassName = "com.zaxxer.hikari.metrics.CodaHaleMetricsTracker";
customizer = new IConnectionCustomizer() {
@Override
public void customize(Connection connection) throws SQLException
{
}
};
}
On Thu, Mar 23, 2017 at 8:29 PM, Eugene Koifman
<ekoif...@hortonworks.com<mailto:ekoif...@hortonworks.com>> wrote:
Can you try use “hikaricp” connection pool manager? It seems to be using
default which is no limit.
Eugene
From: Igor Kuzmenko <f1she...@gmail.com<mailto:f1she...@gmail.com>>
Reply-To: "user@hive.apache.org<mailto:user@hive.apache.org>"
<user@hive.apache.org<mailto:user@hive.apache.org>>
Date: Monday, March 20, 2017 at 2:17 PM
To: "user@hive.apache.org<mailto:user@hive.apache.org>"
<user@hive.apache.org<mailto:user@hive.apache.org>>
Subject: Re: Hive TxnHandler::lock method run into dead lock.
Sorry miss clicked.
2) TxnHandler::lock method request new connection when executing this line of
code:
ConnectionLockIdPair
connAndLockId =
enqueueLockWithRetry(rqst);
3) After that
folowing this
stacktrace:
- TxnHandler::lock
-
TxnHandler::checkLockWithRetry
-
TxnHandler::checkLock
In
checkLock method we reach this line:
handle
=
getMutexAPI().acquireLock(MUTEX_KEY.CheckLock.name());
4)
acquireLock method requests another connection to DB:
dbConn
=
getDbConn(Connection.TRANSACTION_READ_COMMITTED);
So all in all if i call
TxnHandler::lock
method in 10 threads same time at first I get all connections to db, that
stored in pool, and at
acquireLock I will stuck because there's no free connection.
Does anyone run into this problem? How can I avoid this problem?
Code was taken from here:
https://github.com/hortonworks/hive-release/blob/HDP-2.5.0.0-tag/metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
I guess the closest branch in
apach
repo is:
https://github.com/apache/hive/blob/branch-2.1/metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
On Tue, Mar 21, 2017 at 12:07 AM, Igor Kuzmenko
<f1she...@gmail.com<mailto:f1she...@gmail.com>> wrote:
Hello I'm running Hortonworks data platform 2.5.0.0 with included hive.
I'm using storm hive bolt to load data into my hive. But launching many hive
bolt always leads me to TimeoutException on calling hive metastore. Metastore
logs full of Exception like this:
2017-03-15 18:46:12,436 ERROR [pool-5-thread-11]: txn.TxnHandler
(TxnHandler.java:getDbConn(1834)) - There is a problem with a connection from
the pool, retrying(rc=7): Timed out waiting for a free available connection.
(SQLState=08001, ErrorCode=0)
java.sql.SQLException: Timed out waiting for a free available connection.
at
com.jolbox.bonecp.DefaultConnectionStrategy.getConnectionInternal(DefaultConnectionStrategy.java:88)
at
com.jolbox.bonecp.AbstractConnectionStrategy.getConnection(AbstractConnectionStrategy.java:90)
at com.jolbox.bonecp.BoneCP.getConnection(BoneCP.java:553)
at com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:131)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.getDbConn(TxnHandler.java:1827)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:873)
at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:814)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:5751)
After looking through code I found out
1) TxnHandler class uses connection pool to get db connections and it's size
is 10.
2) TxnHandler::lock method requset new connection whe executing this line of
code:
