[JIRA] [ec2] (JENKINS-23787) EC2-plugin not spooling up stopped nodes - "still in the queue ... all nodes of label ... are offline"

Fri, 18 Jul 2014 03:48:07 -0700

 
Craig Ringer edited a comment on Bug JENKINS-23787

Here's an example slave log from such a case.

Starting existing instance: i-185cce5a result:{StartingInstances: [{InstanceId: i-185cce5a,CurrentState: {Code: 0,Name: pending},PreviousState: {Code: 80,Name: stopped}}]}
Starting existing instance: i-185cce5a result:{StartingInstances: [{InstanceId: i-185cce5a,CurrentState: {Code: 0,Name: pending},PreviousState: {Code: 80,Name: stopped}}]}
Connecting to 10.0.1.17 on port 22, with timeout 10000.
Connected via SSH.
bootstrap()
Getting keypair...
Using key: jenkins_ec2_key
3c:05:cc:a3:75:e9:2b:01:01:a4:b0:a2:0b:bd:18:32:d3:59:ae:5d
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAl5L6kI5Dzmvo6gYt7FgWnt/Z//ybp+l2YB0F8CkoZMM0wj16wCXwJvDlZOeC
QAt3FoKQn8s/OTqcaUMKKhBefGgQCmI+ZaKE3/Xu5AGMxu66bqI
Authenticating as admin
take over connection
Executing init script
sudo: unable to resolve host ip-10-0-1-17
sudo: unable to resolve host ip-10-0-1-17
Hit http://ftp.ie.debian.org squeeze Release.gpg
Ign http://ftp.ie.debian.org/debian/ squeeze/main Translation-en
Hit http://ftp.ie.debian.org squeeze Release
Hit http://security.debian.org squeeze/updates Release.gpg
Ign http://security.debian.org/ squeeze/updates/main Translation-en
Hit http://ftp.ie.debian.org squeeze/main Sources
Hit http://ftp.ie.debian.org squeeze/main amd64 Packages
Hit http://security.debian.org squeeze/updates Release
Hit http://security.debian.org squeeze/updates/main Sources
Hit http://security.debian.org squeeze/updates/main amd64 Packages
Reading package lists...
Reading package lists...
Building dependency tree... 

Broadcast message from root@ip-10-0-1-17 (Fri Jul 18 04:44:23 2014):




The system is going down for system halt NOW!

Jenkins thinks the state of the node is "Offline" with message "This node is being launched" but the log clearly shows that it's actually shut down.

In this case the node was launched by a previous job. It looks like it must've been idling out when the job began, though there aren't any timestamps in the job console output to confirm:

Started by an SCM change
[EnvInject] - Loading node environment variables.
Building on master in workspace /var/lib/jenkins/workspace/bdr_linux
.... git spam here ....
Triggering bdr_linux ? x64,debian7
Triggering bdr_linux ? x64,amazonlinux201209
Triggering bdr_linux ? x86,amazonlinux201209
Triggering bdr_linux ? x86,debian6
Configuration bdr_linux ? x64,amazonlinux201209 is still in the queue: Amazon Linux 2012.09 EBS 64-bit  (i-205dcf62) is offline
Configuration bdr_linux ? x64,amazonlinux201209 is still in the queue: All nodes of label ?amazonlinux201209&&x64? are offline
Configuration bdr_linux ? x64,amazonlinux201209 is still in the queue: Waiting for next available executor on amazonlinux201209&&x64
bdr_linux ? x64,amazonlinux201209 completed with result SUCCESS
bdr_linux ? x86,amazonlinux201209 completed with result SUCCESS
Configuration bdr_linux ? x86,debian6 is still in the queue: Debian6_x86 EBS (i-185cce5a) is offline

and the logs show it being idled down:

Jul 18, 2014 10:43:11 AM INFO hudson.plugins.ec2.EC2RetentionStrategy _check
Idle timeout: Debian6_x86 EBS (i-185cce5a)
Jul 18, 2014 10:43:11 AM INFO hudson.plugins.ec2.EC2AbstractSlave idleTimeout
EC2 instance idle time expired: i-185cce5a
Jul 18, 2014 10:43:11 AM INFO hudson.plugins.ec2.EC2AbstractSlave stop
EC2 instance stopped: i-185cce5a
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to