On 03Jan16, Adrian Chadd allegedly wrote: > It doesn't help at low connection rates. It helps at high connection / > concurrency rates as the time going in/out of the kernel and getting > back to steady state execution changes.
This is what I found when I did a comparison at $dayjob. You obviously need more than one message per *mmsg() call to benefit from the syscall amortization which can only occur at very high message rates. It also depends on whether the *mmsg() calls are implemented as mere wrappers around recvmsg() and sendmsg() or whether the implementation goes deeper into the driver to get better batching benefits. (I vaguely thought that one or two drivers in Linux had real recvmmsg() support.) But there are a number of other factors that come into play beyond message rates. The first is that a typical UDP application does very little work per message thus the syscall costs are proportionally more significant than for TCP applications. Eg. DNS or NTP vs HTTP. The second is that a typical syscall sequence of a high performance program likely involves 4 syscalls per packet: select/kqueue/epoll, recvmsg, gettimeofday and sendmsg. The third is that a high performance program probably is multi-threaded so add in a couple of locking syscalls around select/kqueue and you're up to 6 syscalls per request packet. That's a lot of syscalls for an application that might only need a small number of memory references to process a request. The final thing is that on both FBSD and Linux I found a surprising amount of serialization within the UDP socket stack so having multiple threads sit on a blocking recvmsg() at high packet rates is not as productive as you might expect. This level of serialization surprized me given that UDP processing within the kernel should be pretty minimal. Combining all these factors I found it impossible to construct any sort of UDP application that would run any where near line rates on modern beefy machines. It it's not apparent, these are all arguments for implementing *mmsg() calls and for looking at UDP stack serialization. Fortunately for FBSD there is netmap which is moderately well suited to UDP applications... though you still have to do a fair amount of packet decode and encode and it's only applicable in dedicated or specialized deployments. A final observation is that the semantics of *mmsg() may benefit from a bit of scrutiny as they cannot accurately represent some conditions possible with multiple *msg() calls. Eg, if a signal arrives after more than zero packets have been processed by recvmmsg() what is the correct return value? -1 or the count of messages returned? Mark. _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
