Hi Neal, On Thu, Mar 16, 2017 at 11:40:52AM -0400, Neal Cardwell wrote: > On Thu, Mar 16, 2017 at 7:31 AM, Lutz Vieweg <l...@5t9.de> wrote: > > > > On 03/15/2017 11:55 PM, Willy Tarreau wrote: > >> > >> At least I can say I've seen many people enable it without understanding > >> its impact, confusing it > >> with tcp_tw_reuse, and copy-pasting it from random blogs and complaining > >> about issues in > >> production. > > > > > > I currently wonder: What it the correct advise to an operator who needs > > to run one server instance that is meant to accept thousands of new, > > short-lived TCP connections per minute? > > Note that for this to be a problem there would have to be thousands of > new, short-lived TCP connections per minute from a single source IP > address to a single destination IP address. Normal client software > should not be doing this. AFAIK this is pretty rare, unless someone is > running a load test or has an overly-aggressive monitoring system. NAT > boxes or proxies with that kind of traffic should be running with > multiple public source IPs.
In fact it's the regular stuff with reverse-proxies. You can scan the whole source port range every second. But when enabling timestamps, you benefit from PAWS and you don't have any problem anymore, everything works pretty well. > But if/when the problem occurs, then the feasible solutions I'm aware > of, in approximate descending order of preference, are: > > (1) use longer connections from the client side (browsers and RPC libraries > are > usually pretty good about keeping connections open for a long time, so > this > is usually sufficient) > > (2) have the client do the close(), so the client is the side to carry the > TIME_WAIT state That's impossible for proxies, as you can't connect again from the same source port, causing the performances to be divided by more than 100. What proxies have to do when they're forced to close first an outgoing connection is to set SO_LINGER to (0,0) so that an RST is used and the source port can be reused. But as you guess, if that RST gets lost, then next opening is not that beautiful : either [SYN, ACK, RST, pause, SYN, SYN-ACK, ACK] or [SYN, RST, pause SYN, SYN-ACK, ACK] depending on whether the SYN appears in the previous window or not. > (3) have the server use SO_LINGER with a timeout of 0, so that > the connection is closed with a RST and the server carries no > TIME_WAIT state The problem is that it also kills the tail data. Quite frankly, the only issues I'm used to see are with clients closing first and with reusing source connections. As soon as timestamps are enabled on both sides and people don't blindly play with tcp_tw_recycle, I really never face any connection issue. Willy
