On Wed, Dec 14, 2022 at 12:50:10PM +0100, Iago Alonso wrote: > Hi, > > We are not sure what element produces the errors, in the haproxy logs we > don't see them.
Then they don't pass through haproxy nor do they come from haproxy. > What does it happen with the new connections when we hit the > limit? If you're speaking about the maxconnrate/maxsessrate, the connection acceptance is simply delayed until the rate becomes lower again. > What server resource should be affected by it, if any? None at all, actually the connection remains in the kernel's accept queue. However this can have as a side effect that once too many are queued, new ones are just rejected once the kernel queue is full, and maybe the client translates this into a 5xx (this wouldn't shock me as long as there is a way for the user to figure it). > We have our logs in `warning` level so we do not see the response time on > the haproxy logs. Ah, that's an important point. So in fact you almost don't have logs, and maybe you''re really having 5xx that you don't log. You could perhaps just leave them at "info" level and add "option dontlog-normal" that will eliminate logs for successful requests and only log the failed ones. > In the `haproxy_backend_response_time_average_seconds` we > see "normal" values (20ms average and some spikes at less than 80ms). OK, that's good. > We see a spike in the `haproxy_backend_connect_time_average_seconds`, as > well as in the `haproxy_server_connect_time_average_seconds` metrics when > the errors start. Interesting, then this could be related to either a conntrack filling on one element in the chain (including both extremities), or a shortage of source ports when dealing with too many connections. The cost of finding a source port can sometimes become extreme in the system, I've personally seen connect() taking up to 50 ms to succeed! In this case, just increasing the source port range can be enough, just like explicitly setting source port ranges in haproxy (one per "server" line). In this last case, it saves the system from having to find one port, but you must be certain not to collide with listening ports. > We have more haproxies (that we were not stressing in the > test) in front of the same backends, serving without issues. We are using > https between to communicate to the backends too, and we see that the task > consuming the majority of the resources is `ssl_sock_io_cb`. Ah that's pretty useful :-) It's very likely dealing with the handshake. Could you please run "perf top" on this machine and list the 10 top-most lines ? I'm interested in seeing if you're saturating on crypto functions or locking functions (e.g. "native_queued_spin_lock_slowpath"), that would indicate an abuse of mutexes. By the way, are you running with OpenSSL 3.0 ? That one is absolutely terrible and makes extreme abuse of mutexes and locks, to the point that certain workloads were divided by 2-digit numbers between 1.1.1 and 3.0. It took me one day to figure that my load generator which was caping at 400 conn/s was in fact suffering from an accidental build using 3.0 while in 1.1.1 the perf went back to 75000/s! > Our traffic is not request rate intensive, but we do have a high amount of > concurrent connections. For example, at 200k connections, we have 10k rps, > and at 300k connections, we have 14k rps. Are your connections kept alive long or regularly closed ? In the stats page when you hover on the "Sessions/Total" line of a backend or server, there is this (for example): Cum. sessions: 1253716 New connections: 84396 Reused connections: 1169320 (93%) It says that for 1.25M requests, 84k connections were needed and that the rest (93%) could reuse an existing connection. If your reuse rate is too low, you're indeed dealing with new connections that can take some time over SSL. > In our tests, we never saturate our network bandwidth, we hit as much as 50% > our available bandwidth for the server. OK fine. It's perfectly normal not to saturate the network bandwidth using small requests, as the cost of parsing & handling a request is orders of magnitude higher than the cost of processing the equivalent packet. My concern was that I had no idea about the response size, so it could have been possible that a link got filled. Regards, Willy
