Hi Jason,
On Wed, Jun 11, 2014 at 06:02:34AM -0400, Jason Ziemba wrote:
> Hi Lukas,
>
> Responses in-line.
> Ultimately, it appears as though haproxy doesn't like certs residing on an
> NFS mount so I'll need to explore other options at this point.
>
>
>
> > > In further tests, I adjusted the bind line to the following (explicitly
> > > defining a second key)
> > > bind <ip>:443 ssl crt <nfs>/default.pem crt <nfs>/site2.pem crt
> > > <local> crt <nfs> strict-sni
> > >
> > > Accessing site2 with the above bind line worked correctly.
> >
> > I'm not sure if its a good idea to load certificates from remote NFS
> > servers. What if haproxy starts when the NFS path is not yet available
> > or mounted? HAProxy will certainly not recover.
> >
> > While I haven't (yet) run in to this, as system reboots are fairly few and
> far between, since this is a front-end system it's usually rebooted by
> hand, and monitored on it's way up.
> Waiting for a pending mount and then manually (re)starting haproxy would be
> a fairly trivial task, albeit, an extra task.
>
>
> >
> >
> > > Removing the 'site2.pem' from the bind line, leaving the <nfs> and
> > > <local> folders defined to be processed resulted in site2 dropping
> > > offline (due to the strict-sni).
> > > Copying the site2.pem back to the local folder, leaving it out of
> > > haproxy.cfg (and restarting the daemon) restored site2 to service.
> >
> > So it all comes down to the combination of a generic path to a folder
> > and NFS.
> >
> >
> >
> > > I'm not sure how to get enough debug data out of haproxy to determine
> > > what it's doing during init to determine if it's finding the key files
> > > on the NFS to pull in to it's running config, or anything else that
> > > could prove useful in further troubleshooting this.
> > >
> > > Any suggestions is greatly appreciated.
> >
> > Run haproxy through strace during startup.
> >
> > Something like
> > $ sudo strace /sbin/haproxy -db -f /etc/haproxy.cfg
> >
> > During the strace runs, haproxy locates the local path (from it's config)
> and then traverses it to locate and load all pem files within.
> When it finds the NFS mount, it acknowledges it's there, but never loads
> any files from it, that aren't specifically assigned in the configuration.
>
> It 'registers' both directory paths with the equivalent line in the logs:
> open("<local>", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 4
> open("<nfs>", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 4
>
> The next line after the local entry is the first file it encounters, which
> generates a stat64 and a number of open entries for each file.
> The next line after the nfs entry is stat64("<nfs>/..",
> {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0, and then nothing else
> referring to this path occurs again in the strace output.
Could you please test if removing the two following lines from function
ssl_sock_load_cert() in src/ssl_sock.c fixes the issue for you :
if (!S_ISREG(buf.st_mode))
continue;
If so it would be nice to see what mode appears here.
Thanks,
Willy
