Hi, we had two crashes yesterday within about 2 hours.
HA-Proxy version 1.8.4-de425f6 2018/02/26 Copyright 2000-2018 Willy Tarreau <[email protected]> Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-null-dereference -Wno-unused-label OPTIONS = USE_LINUX_SPLICE=1 USE_LIBCRYPT=1 USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with OpenSSL version : OpenSSL 1.1.0f 25 May 2017 Running on OpenSSL version : OpenSSL 1.1.0f 25 May 2017 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Encrypted password support via crypt(3): yes Built with multi-threading support. Built with PCRE version : 8.39 2016-06-14 Running on PCRE version : 8.39 2016-06-14 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with zlib version : 1.2.8 Running on zlib version : 1.2.8 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with network namespace support. Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available filters : [SPOE] spoe [COMP] compression [TRACE] trace root@66b9ab4204d8:/code# gdb /usr/local/sbin/haproxy core GNU gdb (Debian 7.12-6) 7.12.0.20161007-git Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/local/sbin/haproxy...done. [New LWP 10] warning: .dynamic section for "/lib64/ld-linux-x86-64.so.2" is not at the expected address (wrong library or version mismatch?) [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/local/sbin/haproxy -f /etc/haproxy.cfg'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __eb_delete (node=0x55dae9d8db30, node@entry=0x55dae8bdd230) at ebtree/ebtree.h:720 720 ebtree/ebtree.h: No such file or directory. (gdb) bt #0 __eb_delete (node=0x55dae9d8db30, node@entry=0x55dae8bdd230) at ebtree/ebtree.h:720 #1 eb_delete (node=node@entry=0x55dae9d8db30) at ebtree/ebtree.c:25 #2 0x000055dae7bc36f5 in eb32_delete (eb32=0x55dae9d8db30) at ebtree/eb32tree.h:106 #3 __task_unlink_wq (t=0x55dae9d8dad0) at include/proto/task.h:145 #4 task_unlink_wq (t=<optimized out>) at include/proto/task.h:153 #5 task_delete (t=<optimized out>) at include/proto/task.h:192 #6 process_stream (t=t@entry=0x55dae9d8dad0) at src/stream.c:2514 #7 0x000055dae7c3f792 in process_runnable_tasks () at src/task.c:229 #8 0x000055dae7bf2674 in run_poll_loop () at src/haproxy.c:2399 #9 run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2461 #10 0x000055dae7b6cfea in main (argc=<optimized out>, argv=0x7ffcff36a218) at src/haproxy.c:3050 global log /dev/log local0 warning maxconn 50000 tune.ssl.default-dh-param 2048 ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS ssl-default-bind-options no-sslv3 no-tls-tickets ssl-default-server-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS ssl-default-server-options no-sslv3 no-tls-tickets defaults log global mode http timeout connect 3s timeout client 30s timeout server 120s timeout tunnel 3600s timeout http-keep-alive 1s timeout http-request 15s option http-server-close option httplog option forwardfor errorfile 503 /config/503.html errorfile 408 /dev/null userlist httpauth user foo bar resolvers docker nameserver docker 127.0.0.11:53 hold valid 2s frontend http bind 0.0.0.0:80 reqadd X-Forwarded-Proto:\ http acl is_assets hdr_dom(host) -i ${ASSET_HOST} use_backend varnish-backend if is_assets default_backend phoenix-backend frontend https bind 0.0.0.0:443 ssl crt "/letsencrypt/certificates/${CERTIFICATE_NAME}.pem" alpn h2,http/1.1 no-sslv3 rspadd Strict-Transport-Security:\ max-age=31536000 # cowboy crashes when invalid headers are sent # see https://github.com/ninenines/cowboy/issues/943 acl invalid_keepalive_header hdr(Connection) -i keep-alive\ Te reqirep ^Connection:\ keep-alive\ Te Connection:\ keep-alive,\ Te if invalid_keepalive_header acl invalid_keepalive_header_1 hdr(Connection) -i Te\ keep-alive reqirep ^Connection:\ Te\ keep-alive Connection:\ keep-alive,\ Te if invalid_keepalive_header_1 reqadd X-Forwarded-Proto:\ https acl is_assets hdr_dom(host) -i ${ASSET_HOST} acl is_metrics hdr_dom(host) -i m.foo.com acl is_graphs hdr_dom(host) -i g.foo.com acl is_ci hdr_dom(host) -i c.foo.com use_backend varnish-backend if is_assets use_backend prometheus-backend if is_metrics use_backend grafana-backend if is_graphs use_backend ci-backend if is_ci default_backend phoenix-backend backend varnish-backend server varnish varnish:80 resolvers docker init-addr libc,last,none check port 80 inter 200 backend phoenix-backend option httpchk GET /status server phoenix phoenix:4000 resolvers docker init-addr libc,last,none check inter 200 backend prometheus-backend acl auth_ok http_auth(httpauth) http-request auth realm httpauth unless auth_ok server prometheus prometheus:9090 resolvers docker init-addr last,none check port 9090 backend grafana-backend server grafana grafana:3000 resolvers docker init-addr last,none check port 3000 backend ci-backend server drone-server drone-server:8000 resolvers docker init-addr last,none check port 8000
