Ok I've tracked this problem down specifically to the usage of check tracking.
That is to say, the backend "example-api" is set to track the backend
"example-http". When that tracking is enabled and one of the servers in the
backend goes down then all of haproxy goes down and never recovers.
So this works:
server myhost myhost.example.com:8445 ssl ca-file
/usr/local/ssl/certs/cacerts.cert
But this does not:
server myhost myhost.example.com:8445 track example-http/myhost ssl ca-file
/usr/local/ssl/certs/cacerts.cert
This is definitely a regression from 1.7 because I used this feature in 1.7
without issue.
> On Jan 16, 2018, at 10:36 PM, Paul Lockaby <[email protected]> wrote:
>
> I'm experiencing a problem that I can't diagnose but I can recreate pretty
> consistently. I have a single server that responds for example.com and
> api.example.com and it runs haproxy. All the names run through an SSL front
> door but an ACL makes it such that requests for example.com get sent to 8443
> where Apache runs and requests for api.example.com get sent to 8445 where the
> same instance of haproxy runs and does further examination of the request and
> sends it to an application server running on localhost.
>
> This configuration works great except when I take a server out of the
> rotation by disabling it with disable-on-404. As soon as I take any server
> out of the rotation, haproxy completely stops responding to ANY requests for
> ANY backend even things that aren't part of the group such as the stats
> backend and frontend. If I put the server back in to service haproxy does not
> recover. I must restart haproxy on all hosts to recover. Nothing shows up in
> the logs and I can't figure out how to debug it such that I can provide more
> information but it's very consistently reproducible using the configuration
> below. I am running 1.8.3 and I have not tried this on 1.7 or earlier
> versions of 1.8.
>
> Thanks for your help.
> -Paul
>
>
>
> global
> log /dev/log local0
> user nobody
> group nobody
> tune.ssl.default-dh-param 2048
> stats socket /var/run/haproxy.sock user nobody group nobody
> daemon
>
> defaults
> timeout connect 5000ms
> timeout client 600000ms
> timeout server 600000ms
>
> option httplog
> option forwardfor
> option http-server-close
> option contstats
>
> frontend stats-frontend
> bind *:2999
> mode http
> log global
> stats enable
> stats uri /haproxy
>
> backend stats-backend
> mode http
> log global
> server stats /var/run/haproxy.sock check
>
> frontend secured
> # get the list of certificate options from a list in a file
> bind *:443 ssl crt-list /srv/haproxy/certificates.lst
> mode http
> log global
>
> # tell backend connections what our ssl client cn is
> http-request set-header X-SSL-Client-Verify %[ssl_c_verify]
> http-request set-header X-SSL-Client-DN %{+Q}[ssl_c_s_dn]
> http-request set-header X-SSL-Client-CN %{+Q}[ssl_c_s_dn(cn)]
> http-request set-header X-SSL-Issuer-DN %{+Q}[ssl_c_i_dn]
> http-request set-header X-SSL-Issuer-CN %{+Q}[ssl_c_i_dn(cn)]
>
> acl server-status path_beg /server-
> use_backend bogus-http if server-status
>
> # connection requests for apis go to the api backends
> acl request_api hdr_beg(Host) -i api.
> use_backend example-api if request_api
>
> default_backend example-http
>
> backend example-http
> mode http
> log global
> balance source
> hash-type consistent
> option httpchk GET /haproxy/alive.txt
> http-check disable-on-404
> server myhost myhost.example.com:8443 check ssl ca-file
> /usr/local/ssl/certs/cacerts.cert
>
> backend bogus-http
> mode http
> errorfile 503 /netops/www/haproxy/403.http
>
> backend example-api
> mode http
> log global
> balance roundrobin
> option httpchk GET /haproxy/alive.txt
> http-check disable-on-404
> server myhost myhost.example.com:8445 track example-http/myhost ssl
> ca-file /usr/local/ssl/certs/cacerts.cert
>
> frontend localhost-api-frontend
> bind *:8445 ssl crt /usr/local/ssl/certs/example.com.pem
> mode http
> log global
> option forwardfor if-none
> option dontlog-normal
>
> # the alerts api backend
> acl alerts-api_host hdr_beg(Host) -i api.alerts
> use_backend localhost-api-backend-alerts if alerts-api_host
>
> default_backend bogus-http
>
> backend localhost-api-backend-alerts
> mode http
> log global
> option forwardfor if-none
> option dontlog-normal
> server localhost localhost:4002
>
>
>
>
>
>
>
> And the certificates.lst file referenced above looks like this:
>
>
> # this order is because we need to work with older clients that don't
> # speak sni and this works for them in our setup.
> /usr/local/ssl/certs/example.com.pem *
> /usr/local/ssl/certs/example.com.pem [ca-file
> /usr/local/ssl/certs/example-ca.cert verify optional] api.example.com
>
>
>
