Hi,
we are running haproxy 1.5.18 on Ubuntu 14.
Recently we had some issues with misuse of our api by botnets coming from many
different IPs
and implemented a mitigation based on haproxys stick-table feature which works
pretty good for us.
The relevant part of our config looks like this:
acl document_request path_beg -i /api/v1/sessions
acl too_many_tries sc0_gpc0_rate() gt 5
acl mark_seen sc0_inc_gpc0 gt 0
stick-table type string size 100k expire 5m store gpc0_rate(5s)
tcp-request content track-sc0 hdr(True-Client-IP) if
METH_POST document_request
use_backend l2b_slow_down if mark_seen too_many_tries
backend l2b_slow_down
mode http
timeout tarpit 20s
http-request tarpit
Now, when some IP (IPv4 or IPv6 doesn't matter since we use string here) sends
more than
5 POST requests to our api resource in 5 seconds, it will end up in the table
and all
subsequent requests are delayed and answered with an error 500.
After 5 minutes the table entry should expire. This usually works but sometimes
it
doesn't. We see entries in the stick table whose expiry is down to 0 but they
don't
get removed since the "use" counter is something greater than zero
It then looks like this:
echo "show table l1f_loadbalancer" | sudo socat stdio /var/run/haproxy.stat
# table: l1f_loadbalancer, type: string, size:102400, used:7
0x429af6c: key=37.191.169.199 use=1 exp=0 gpc0_rate(5000)=0
0x2f1d54c: key=37.191.204.19 use=1 exp=0 gpc0_rate(5000)=0
0x3c105bc: key=50.115.248.18 use=1 exp=0 gpc0_rate(5000)=0
0x3293ebc: key=65.214.65.27 use=1 exp=0 gpc0_rate(5000)=0
0x30b0acc: key=68.180.24.112 use=2 exp=0 gpc0_rate(5000)=0
0x53571bc: key=68.180.57.63 use=1 exp=0 gpc0_rate(5000)=0
0x300f93c: key=68.187.196.250 use=1 exp=0 gpc0_rate(5000)=0
I also cannot delete them while they are "in use":
echo "clear table l1f_loadbalancer key 50.115.248.18" | sudo socat stdio
/var/run/haproxy.stat
Entry currently in use, cannot remove
clear table without a key only removes entries which are at use=0
The only way to get rid of them is a complete restart of haproxy
Can anybody tell me why this is happening? The documentation, while being great
in general
is a bit short on the stick-table output so I am not really sure what "use=x"
exactly stands
for if it is > 0.
Thanks
Gernot
