There have been a few reports by users of front end web proxies and other systems under FreeBSD that port randomization causes them problems under load. This seems to be due to a combination of port randomization and rapid connections to the same host causing ports to be recycled before the ISN has advanced past the end of the previous connection, thereby causing the TIME_WAIT socket on the receiving end to ignore the new SYN.
I haven't proven it, but I imagine that part of the problem is due to the interaction of established and finished connections. Say that ports 1500 -> 1600 have established connections, and that arc4random picks port 1504 as the next one to try. Rather than picking another random port, we simply look sequentially through the rest of the ports, and will pick 1601. I had considered continuing to do random port guesses until an unused one was found, but that has the nasty property of looping an undetermined amount of time, and it would most likely not help anything anyway.
The solution I've come up with that should provide a decent tradeoff between the increased security of port randomization and the realities of the problem of quick port recycling is attached. What it does is only randomize the first net.inet.ip.portrange.randomized ports assigned each second. Beyond that, it uses sequentual allocation. The default I picked, 20, is arbitrary. Initial testing by a front-end web proxy admin shows that 20 is too high to avoid problems in his case, but 1 works decently.
Although this isn't a perfect fix, I think that it should be acceptable for the vast majority of systems, and I'd like to get it in before 4.11-release ships. To be conservative, I'll probably choose a value like 5, which should be fine for most systems out there. Super specialized users will always be able to lower it to 0.
Since this is an important issue, I was wondering if anyone had any suggestions on how to improve this method, or how to solve the problem in another way.
Remember that 4.11 ships in the next few weeks, so any solution which gets added to it must be relatively simple, we can't be doing anything like changing the TCP state machine. (Yes, I've considered removing the ISN > window requirement from the TIME_WAIT state, but that's not something to be changing on a whim.)
Also note that 5.x and 6.x handle TIME_WAIT recycling in the exact same way 4.x does; the sequence number check is just now in the function tcp_timewait rather than in tcp_input. So, the same problems plagues all versions. (Despite my other modifications to sequence number generation which were attempts to fix it.)
Comments appreciated,
Mike "Silby" Silbersack
--- /usr/src/sys.old/netinet/in_pcb.c Thu Dec 16 03:26:11 2004
+++ in_pcb.c Thu Dec 16 03:27:29 2004
@@ -95,8 +95,10 @@
int ipport_hifirstauto = IPPORT_HIFIRSTAUTO; /* 49152 */
int ipport_hilastauto = IPPORT_HILASTAUTO; /* 65535 */
-/* Shall we allocate ephemeral ports in random order? */
-int ipport_randomized = 0;
+/* How many ephemeral port randomizations should occur each second? */
+int ipport_maxrps = 20;
+struct timeval ipport_lastrandom;
+int ipport_currps;
#define RANGECHK(var, min, max) \
if ((var) < (min)) { (var) = (min); } \
@@ -135,7 +137,7 @@
SYSCTL_PROC(_net_inet_ip_portrange, OID_AUTO, hilast, CTLTYPE_INT|CTLFLAG_RW,
&ipport_hilastauto, 0, &sysctl_net_ipport_check, "I", "");
SYSCTL_INT(_net_inet_ip_portrange, OID_AUTO, randomized, CTLFLAG_RW,
- &ipport_randomized, 0, "");
+ &ipport_maxrps, 0, "");
/*
* in_pcb.c: manage the Protocol Control Blocks.
@@ -323,7 +325,8 @@
/*
* counting down
*/
- if (ipport_randomized)
+ if (ipport_maxrps &&
+ ppsratecheck(&ipport_lastrandom, &ipport_currps, ipport_maxrps))
*lastport = first -
(arc4random() % (first - last));
count = first - last;
@@ -343,7 +346,8 @@
/*
* counting up
*/
- if (ipport_randomized)
+ if (ipport_maxrps &&
+ ppsratecheck(&ipport_lastrandom, &ipport_currps, ipport_maxrps))
*lastport = first +
(arc4random() % (last - first));
count = last - first;
--- /usr/src/sys.old/netinet/in_pcb.c Thu Nov 25 00:43:21 2004
+++ /usr/src/sys/netinet/in_pcb.c Thu Dec 16 02:55:46 2004
@@ -97,8 +97,10 @@
int ipport_reservedhigh = IPPORT_RESERVED - 1; /* 1023 */
int ipport_reservedlow = 0;
-/* Shall we allocate ephemeral ports in random order? */
-int ipport_randomized = 1;
+/* How many ephemeral port randomizations should occur each second? */
+int ipport_maxrps = 20;
+struct timeval ipport_lastrandom;
+int ipport_currps;
#define RANGECHK(var, min, max) \
if ((var) < (min)) { (var) = (min); } \
@@ -142,7 +144,7 @@
SYSCTL_INT(_net_inet_ip_portrange, OID_AUTO, reservedlow,
CTLFLAG_RW|CTLFLAG_SECURE, &ipport_reservedlow, 0, "");
SYSCTL_INT(_net_inet_ip_portrange, OID_AUTO, randomized,
- CTLFLAG_RW, &ipport_randomized, 0, "");
+ CTLFLAG_RW, &ipport_maxrps, 0, "");
/*
* in_pcb.c: manage the Protocol Control Blocks.
@@ -404,7 +406,8 @@
/*
* counting down
*/
- if (ipport_randomized)
+ if (ipport_maxrps &&
+ ppsratecheck(&ipport_lastrandom, &ipport_currps, ipport_maxrps))
*lastport = first -
(arc4random() % (first - last));
count = first - last;
@@ -422,7 +425,8 @@
/*
* counting up
*/
- if (ipport_randomized)
+ if (ipport_maxrps &&
+ ppsratecheck(&ipport_lastrandom, &ipport_currps, ipport_maxrps))
*lastport = first +
(arc4random() % (last - first));
count = last - first;
_______________________________________________ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
