I'm seeing a bug that appears to be a bad interaction between Puppet
and SystemD on Debian Jessie that causes my Docker daemon to go
unavailable. I'm running both Docker and Puppet from upstream:
root@docker0:~# dpkg --list | grep 'lxc-docker\|puppet'
ii facter 2.4.4-1puppetlabs1 all
Ruby module for collecting simple facts about a host operating system
ii hiera 1.3.4-1puppetlabs1 all A
simple pluggable Hierarchical Database.
ii lxc-docker 1.6.2 amd64
Linux container runtime
ii lxc-docker-1.6.2 1.6.2 amd64
Linux container runtime
ii puppet 3.8.1-1puppetlabs1 all
Centralized configuration management - agent startup and compatibility scripts
ii puppet-common 3.8.1-1puppetlabs1 all
Centralized configuration management
ii puppetlabs-release 1.0-11 all
"Package to install Puppet Labs gpg key and apt repo"
For this test I've stripped my manifest down to the minimum: only a
service declaration for Docker on the node:
node "docker0.tjnii.com" {
#Debug
service { 'docker':
ensure => running,
enable => true,
}
}
Docker is started by Puppet via SystemD without issue:
root@docker0:~# puppet agent --test
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for docker0.tjnii.com
Info: Applying configuration version '1434419985'
Notice: /Stage[main]/Main/Node[docker0.tjnii.com]/Service[docker]/ensure:
ensure changed 'stopped' to 'running'Info:
/Stage[main]/Main/Node[docker0.tjnii.com]/Service[docker]: Unscheduling refresh
on Service[docker]
Notice: Finished catalog run in 0.88 seconds
root@docker0:~# systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; disabled)
Drop-In: /etc/systemd/system/docker.service.d
service-overrides.conf
Active: active (running) since Mon 2015-06-15 20:01:45 MDT; 32s ago
Docs: http://docs.docker.com
Main PID: 25160 (docker)
CGroup: /system.slice/docker.service
25160 /usr/bin/docker -d -H fd://
<Snip Logs>
root@docker0:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED
STATUS PORTS NAMES
root@docker0:~# ps awwfux | grep [d]ocker
root 25160 0.6 0.9 273044 14984 ? Ssl 20:01 0:00
/usr/bin/docker -d -H fd://
root@docker0:~# ls /var/run/docker.sock -l
srw-rw---- 1 root docker 0 Jun 15 20:01 /var/run/docker.sock
After a few Puppet daemon runs (I haven't nailed it down to a fixed number,
perhaps
a race?) Puppet will think Docker is down and try and restart it:
Jun 15 20:32:35 docker0 puppet-agent[1271]:
(/Stage[main]/Main/Node[docker0.tjnii.com]/Service[docker]/ensure) ensure
changed 'stopped' to 'running'
Jun 15 20:32:35 docker0 puppet-agent[1271]: Finished catalog run in 1.10 seconds
This will replace the socket with a new file:
root@docker0:~# ls /var/run/docker.sock -l
srw-rw---- 1 root docker 0 Jun 15 20:32 /var/run/docker.sock
However the Docker daemon is not restarted. Note that the pid hasn't
changed from when it was initially started:
root@docker0:~# ps awwfux | grep [d]ocker
root 25160 0.4 1.1 557024 17008 ? Ssl 20:01 0:11 /usr/bin/docker
-d -H fd://
So after Puppet tries to start the service it looks like the Docker daemon
becomes
disconnected from its socket. Puppet is exacerbating this, if I stop Puppet
Docker
is stable. I'm wondering if this has something to do with the docker.socket
SystemD service, but I don't know enough about SystemD to troubleshoot that.
This is the SystemD config from Docker:
root@docker0:~# systemctl cat docker.service
# /lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target docker.socket
Requires=docker.socket
[Service]
ExecStart=/usr/bin/docker -d -H fd://
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
[Install]
WantedBy=multi-user.target
# /etc/systemd/system/docker.service.d/service-overrides.conf
[Service]
EnvironmentFile=-/etc/default/docker
root@docker0:~# systemctl cat docker.socket
# /lib/systemd/system/docker.socket
[Unit]
Description=Docker Socket for the API
PartOf=docker.service
[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
[Install]
WantedBy=sockets.target
I'm not sure if this is a Puppet bug, SystemD bug, or Docker bug. has anyone
else seen
behaviour like this with Puppet and SystemD?
--Tom Noonan II
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/20150615205357.6a6f913d%40vncBox.tjnii.com.
For more options, visit https://groups.google.com/d/optout.
