[JIRA] (JENKINS-60536) Jenkins job hangs after git rev-list command

[d.fra...@crossflight.com (JIRA)]

Title: Message Title

David France commented on JENKINS-60536   Re: Jenkins job hangs after git rev-list command We've being seeing much the same problem stack that Kieran reported, trying to progress issues by a workflow action in Jira.  We first became aware of the problem on 20/12/2019: "Executor #1 for master : executing deployToUAT #1619" #647 daemon prio=5 os_prio=0 cpu=436.80ms elapsed=236.46s tid=0x000000006 2b92000 nid=0x2590 waiting on condition [0x000000006644d000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.5/Native Method) - parking to wait for <0x0000000083d46f58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.5/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.5/AbstractQueuedSynchronizer.j ava:885) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.5/AbstractQueuedSynchronizer.java:917) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.5/AbstractQueuedSynchronizer.java:1240) at java.util.concurrent.locks.ReentrantLock.lock(java.base@11.0.5/ReentrantLock.java:267) at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:278) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.j ava:295) at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129) at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75) at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108) at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseH ttpPromiseAsyncClient.java:36) at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417) at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364) at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297) at com.atlassian.jira.rest.client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianH ttpClientDecorator.java:83) at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253) at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.j ava:69) at com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJqlImplGet(AsynchronousSearchRestClient. java:109) at com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJql(AsynchronousSearchRestClient.java:94 ) at hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:196) at hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:136) at hudson.plugins.jira.JiraSite.progressMatchingIssues(JiraSite.java:1043) at hudson.plugins.jira.JiraIssueUpdateBuilder.perform(JiraIssueUpdateBuilder.java:105) at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741) at hudson.model.Build$BuildExecution.build(Build.java:206) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504) at hudson.model.Run.execute(Run.java:1853) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:427)    The problem looks to be a problem with an earlier connection in the connection pool in the HttpAsyncClient. The earlier open connection is failing to be closed before processing the new request. We did a couple of experiments in the jira-plugin source, and managed to find a way get the connection to complete. (This is hardly a submission quality change, but perhaps it offers a clue or two). The first hope was that changing the HttpClientOptions.maxConnectionsPerHost value would stop connection pool holding the connections, but forcing this (in the code) to either 1 or 2000 had no effect. Instead, as a rough workaround we tried bypassing the connection pooling, by creating a new HttpAsyncClientBuilder against a new PoolingNHttpClientConnectionManager for each request. This is will be leaky - there is nothing to close the new connection pools. We don't know how soon to expect a problem though, Jenkins has stayed up so far. The change is here. (Essentially it extracts a method in the constructor, to create the HttpAsyncClientBuilder, then also calls it from doExecute().) For reference this is working against JIRA Cloud. Not sure why the connections started failing to close, but this was enough to force our builds through for the time being. Add Comment   This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.203639.1576745651000.3705.1578403621745%40Atlassian.JIRA.