Damon Cortesi created HIVE-20600:
------------------------------------
Summary: Metastore connection leak
Key: HIVE-20600
URL: https://issues.apache.org/jira/browse/HIVE-20600
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 2.3.3
Reporter: Damon Cortesi
Attachments: consume_threads.py
Within the execute method of HiveServer2, there appears to be a connection
leak. With fairly straightforward series of INSERT statements, the connection
count in the logs continues to increase over time. Under certain loads, this
can also consume all underlying threads of the Hive metastore and result in HS2
becoming unresponsive to new connections.
The log below is the result of some python code executing a single insert
statement, and then looping through a series of 10 more insert statements. We
can see there's one dangling connection left open after each execution leaving
us with 12 open connections (11 from the execute statements + 1 from HS2
startup).
{code}
2018-09-19T17:14:32,108 INFO [main([])]: hive.metastore
(HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore,
current connections: 1
2018-09-19T17:14:48,175 INFO [29049f74-73c4-4f48-9cf7-b4bfe524a85b
HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore
(HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore,
current connections: 2
2018-09-19T17:15:05,543 INFO [HiveServer2-Background-Pool: Thread-36([])]:
hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
metastore, current connections: 1
2018-09-19T17:15:05,548 INFO [HiveServer2-Background-Pool: Thread-36([])]:
hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
metastore, current connections: 2
2018-09-19T17:15:05,932 INFO [HiveServer2-Background-Pool: Thread-36([])]:
hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
metastore, current connections: 1
2018-09-19T17:15:05,935 INFO [HiveServer2-Background-Pool: Thread-36([])]:
hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
metastore, current connections: 2
2018-09-19T17:15:06,123 INFO [HiveServer2-Background-Pool: Thread-36([])]:
hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
metastore, current connections: 1
2018-09-19T17:15:06,126 INFO [HiveServer2-Background-Pool: Thread-36([])]:
hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
metastore, current connections: 2
...
2018-09-19T17:15:20,626 INFO [29049f74-73c4-4f48-9cf7-b4bfe524a85b
HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore
(HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore,
current connections: 12
2018-09-19T17:15:21,153 INFO [HiveServer2-Background-Pool: Thread-162([])]:
hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
metastore, current connections: 11
2018-09-19T17:15:21,155 INFO [HiveServer2-Background-Pool: Thread-162([])]:
hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
metastore, current connections: 12
2018-09-19T17:15:21,306 INFO [HiveServer2-Background-Pool: Thread-162([])]:
hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
metastore, current connections: 11
2018-09-19T17:15:21,308 INFO [HiveServer2-Background-Pool: Thread-162([])]:
hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
metastore, current connections: 12
2018-09-19T17:15:21,385 INFO [HiveServer2-Background-Pool: Thread-162([])]:
hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
metastore, current connections: 11
2018-09-19T17:15:21,387 INFO [HiveServer2-Background-Pool: Thread-162([])]:
hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
metastore, current connections: 12
2018-09-19T17:15:21,541 INFO [HiveServer2-Handler-Pool: Thread-31([])]:
hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
metastore, current connections: 13
2018-09-19T17:15:21,542 INFO [HiveServer2-Handler-Pool: Thread-31([])]:
hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
metastore, current connections: 12
{code}
Attached is a simple [impyla|https://github.com/cloudera/impyla] script that
triggers the condition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
