On Mon, 2021-02-01 at 20:07 -0800, Jakub Kicinski wrote: > On Mon, 1 Feb 2021 20:41:45 +0800 Menglong Dong wrote: > > I am thinking about making sendmsg/recvmsg process multiple messages > > at once, which is possible to reduce the number of system calls. > > > > Take the receiving of udp as an example, we can copy multiple skbs to > > msg_iov and make sure that every iovec contains a udp package. > > > > Is this a good idea? This idea seems clumsy compared to the incoming > > 'io-uring' based zerocopy, but maybe it can help...
Indeed since the introduction of some security vulnerability mitigation, syscall overhead is relevant and amortizing it with bulk operations gives very measurable performances gain. Potentially bulk operation also reduce RETPOLINE overhead, but AFAICS all the indirect calls in the relevant code path has been already mitigated with the indirect call wrappers. Note that you can already process several packets with a single syscall using sendmmsg/recvmmsg. Both have issues with error reporting and timeout and IIRC still don't amortize the overhead introduced e.g. by CONFIG_HARDENED_USERCOPY. Additionally, recvmmsg/sendmmsg are not cache-friendly. As noted by Eric long time ago: https://marc.info/?l=linux-netdev&m=148010858826712&w=2 perf tests in lab with recvmmsg/sendmmsg could be great, but performance with real workload much less. You could try fine-tuning the bulk size (mmsg nr) for your workload and H/W. Likely a burst size above 8 is a no go. For the TX path there is already a better option - for some specific workload - using UDP_SEGMENT. In the RX path, for bulk transfer, you could try enabling UDP_GRO. As far as I can see, the idea you are proposing will be quite alike recvmmsg(), with the possible additional benefit of bulk dequeue from the UDP receive queue. Note that this latter optimization, since commmit 2276f58ac5890, will give very little perfomance gain. In the TX path there is no lock at all for the uncorking case, so the performance gain should come only from the bulk syscall. You will probably also need to cope with cmsg and msgname, so overall I don't see much differences from recvmmsg()/sendmmsg(), did I misread something? Thanks! Paolo
