[JIRA] [ircbot-plugin] (JENKINS-28175) config change deadlock Jenkins when pircx.shutdown() is invoked

[has...@free.fr (JIRA)]

Antoine Musso created JENKINS-28175 config change deadlock Jenkins when pircx.shutdown() is invoked Issue Type: Bug Assignee: kutzi Components: ircbot-plugin Created: 30/Apr/15 2:10 PM Description: We have recently upgraded the IRC plugin from 2.25 to 2.26. On configuration change, the ircbot plugin invokes PircBotX.shutdown(). For some reason it never finish and the conf change is stalled. A side effect is that jobs sending notifications ends up being blocked waiting for an instance of the irc connection provider. The only fix is to restart Jenkins entirely. Our bug has a few more explanations https://phabricator.wikimedia.org/T96183 and a full thread dump attached https://phabricator.wikimedia.org/P584 Here are the blocked threads: Two jobs are blocked: "Executor #2 for integration-slave-trusty-1016 : executing browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce #234" prio=5 BLOCKED "Executor #1 for integration-slave-trusty-1012 : executing browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce #494" prio=5 BLOCKED hudson.plugins.ircbot.v2.IRCConnectionProvider.getInstance(IRCConnectionProvider.java:14) hudson.plugins.ircbot.IrcPublisher.getIMConnection(IrcPublisher.java:102) hudson.plugins.im.IMPublisher.sendNotification(IMPublisher.java:374) hudson.plugins.im.IMPublisher.notifyChatsOnBuildEnd(IMPublisher.java:585) hudson.plugins.im.IMPublisher.notifyOnBuildEnd(IMPublisher.java:304) hudson.plugins.im.IMPublisher.perform(IMPublisher.java:291) ... A configuration submit change is blocked as well: "Handling POST /ci/configSubmit from X.X.X.X : RequestHandlerThread[#1683]" daemon prio=5 WAITING sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236) org.pircbotx.Channel.getMode(Channel.java:127) org.pircbotx.Channel.getModeArgument(Channel.java:182) org.pircbotx.Channel.getChannelKey(Channel.java:239) org.pircbotx.PircBotX.shutdown(PircBotX.java:2872) hudson.plugins.ircbot.v2.IRCConnection.close(IRCConnection.java:102) hudson.plugins.im.IMConnectionProvider.releaseConnection(IMConnectionProvider.java:92) hudson.plugins.ircbot.v2.IRCConnectionProvider.setDesc(IRCConnectionProvider.java:19) hudson.plugins.ircbot.IrcPublisher$DescriptorImpl.configure(IrcPublisher.java:336) jenkins.model.Jenkins.configureDescriptor(Jenkins.java:2915) jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:2878) ... Some other related threads: "JenkinsIsBusyListener-thread" daemon prio=5 BLOCKED hudson.plugins.im.IMConnectionProvider.currentConnection(IMConnectionProvider.java:83) hudson.plugins.im.JenkinsIsBusyListener.setStatus(JenkinsIsBusyListener.java:118) hudson.plugins.im.JenkinsIsBusyListener.updateIMStatus(JenkinsIsBusyListener.java:109) hudson.plugins.im.JenkinsIsBusyListener.access$000(JenkinsIsBusyListener.java:20) hudson.plugins.im.JenkinsIsBusyListener$3.run(JenkinsIsBusyListener.java:98) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask.run(FutureTask.java:262) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) "IM-Reconnector-Thread" daemon prio=5 BLOCKED hudson.plugins.im.IMConnectionProvider$ConnectorRunnable.run(IMConnectionProvider.java:175) java.lang.Thread.run(Thread.java:745) Looking at the git changelog I noticed https://github.com/jenkinsci/ircbot-plugin/commit/98b0105a743d062abf957c285cdded06fedd3fa3 which changes in IRCConnection.close: - this.pircConnection.disconnect(); + this.pircConnection.shutdown(true); Which was done to fix a leak (JENKINS-25349). Environment: jenkins 1.596.2 ircbot-plugin 2.26 Project: Jenkins Priority: Critical Reporter: Antoine Musso

This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira

--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.