[JIRA] [ec2] (JENKINS-23787) EC2-plugin not spooling up stopped nodes - "still in the queue ... all nodes of label ... are offline"

[cr...@2ndquadrant.com (JIRA)]

Craig Ringer edited a comment on JENKINS-23787 EC2-plugin not spooling up stopped nodes - "still in the queue ... all nodes of label ... are offline" Here's an example slave log from such a case. Starting existing instance: i-185cce5a result:{StartingInstances: [{InstanceId: i-185cce5a,CurrentState: {Code: 0,Name: pending},PreviousState: {Code: 80,Name: stopped}}]} Starting existing instance: i-185cce5a result:{StartingInstances: [{InstanceId: i-185cce5a,CurrentState: {Code: 0,Name: pending},PreviousState: {Code: 80,Name: stopped}}]} Connecting to 10.0.1.17 on port 22, with timeout 10000. Connected via SSH. bootstrap() Getting keypair... Using key: jenkins_ec2_key 3c:05:cc:a3:75:e9:2b:01:01:a4:b0:a2:0b:bd:18:32:d3:59:ae:5d -----BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEAl5L6kI5Dzmvo6gYt7FgWnt/Z//ybp+l2YB0F8CkoZMM0wj16wCXwJvDlZOeC QAt3FoKQn8s/OTqcaUMKKhBefGgQCmI+ZaKE3/Xu5AGMxu66bqI Authenticating as admin take over connection Executing init script sudo: unable to resolve host ip-10-0-1-17 sudo: unable to resolve host ip-10-0-1-17 Hit http://ftp.ie.debian.org squeeze Release.gpg Ign http://ftp.ie.debian.org/debian/ squeeze/main Translation-en Hit http://ftp.ie.debian.org squeeze Release Hit http://security.debian.org squeeze/updates Release.gpg Ign http://security.debian.org/ squeeze/updates/main Translation-en Hit http://ftp.ie.debian.org squeeze/main Sources Hit http://ftp.ie.debian.org squeeze/main amd64 Packages Hit http://security.debian.org squeeze/updates Release Hit http://security.debian.org squeeze/updates/main Sources Hit http://security.debian.org squeeze/updates/main amd64 Packages Reading package lists... Reading package lists... Building dependency tree... Broadcast message from root@ip-10-0-1-17 (Fri Jul 18 04:44:23 2014): The system is going down for system halt NOW! Jenkins thinks the state of the node is "Offline" with message "This node is being launched" but the log clearly shows that it's actually shut down. In this case the node was launched by a previous job. It looks like it must've been idling out when the job began, though there aren't any timestamps in the job console output to confirm: Started by an SCM change [EnvInject] - Loading node environment variables. Building on master in workspace /var/lib/jenkins/workspace/bdr_linux .... git spam here .... Triggering bdr_linux ? x64,debian7 Triggering bdr_linux ? x64,amazonlinux201209 Triggering bdr_linux ? x86,amazonlinux201209 Triggering bdr_linux ? x86,debian6 Configuration bdr_linux ? x64,amazonlinux201209 is still in the queue: Amazon Linux 2012.09 EBS 64-bit (i-205dcf62) is offline Configuration bdr_linux ? x64,amazonlinux201209 is still in the queue: All nodes of label ?amazonlinux201209&&x64? are offline Configuration bdr_linux ? x64,amazonlinux201209 is still in the queue: Waiting for next available executor on amazonlinux201209&&x64 bdr_linux ? x64,amazonlinux201209 completed with result SUCCESS bdr_linux ? x86,amazonlinux201209 completed with result SUCCESS Configuration bdr_linux ? x86,debian6 is still in the queue: Debian6_x86 EBS (i-185cce5a) is offline The build start timestamp for the matrix build is Jul 18, 2014 4:34:14 . The EC2 plugin logs show it being idled down: Jul 18, 2014 10:43:11 AM INFO hudson.plugins.ec2.EC2RetentionStrategy _check Idle timeout: Debian6_x86 EBS (i-185cce5a) Jul 18, 2014 10:43:11 AM INFO hudson.plugins.ec2.EC2AbstractSlave idleTimeout EC2 instance idle time expired: i-185cce5a Jul 18, 2014 10:43:11 AM INFO hudson.plugins.ec2.EC2AbstractSlave stop EC2 instance stopped: i-185cce5a The worker node is in a different timezone to Jenkins, hence the mismatched times. You can see the node went down at :44, moments after idle stop was requested. I'm beginning to suspect that this issue is related to matrix builds, and specifically to matrix builds with a touchstone build that runs first.

This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira

--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.