I want to share my research about my milter tcp speed issue:
I have turned on smtpd -v in master.cf to see the mss in the logfile:
On my ubuntu 18.04 the loopback device has a mtu of 65536 which results
in a "vstream_tweak_tcp: TCP_MAXSEG 21845 "
This is slow and has the nagle timeout problem.
When I set the mtu to 65535 (or 65534) it still gets TCP_MAXSEG 21845
(and is still slow).
Once i do "ip link set dev lo mtu 21845" i will get vstream_tweak_tcp:
TCP_MAXSEG 21793 and the performance is great (only 1 second for 100mb
body).
Installing the Milter on another machine (which is available throught
the eno1 (1500mtu) interface i will get TCP_MAXSEG 1448 and still bad
performance (43secs instead of 1.30m).
Since this is strange i prefer to switch to unixsocket.
I also researched where the delay happened in my milter: its while
waiting for the 64k packet
https://github.com/mschneider82/milter/blob/b4a6d7c78ac39b8a1d71b53b74bc08d95652d310/session.go#L101
Best regards
Matthias Schneider
Am 19.08.19 um 20:05 schrieb Wietse Venema:
Matthias Schneider:
Hi Wietse,
I suspect that the bottleneck is on the receiving side.
- Maybe the loopback stack does not like the 65535 block size. Try
using an ethernet interface address instead.
I tried switching from interface lo to eno1 - same issue.
What's the en01 MTU size? Postfix has code (in vstream_tweak.c
which is called by the Milter client and other Postfix code) to
automatically increase the VSTREAM buffer from 4096 to the maximal
segment size, to avoid Nagle delays, though that would not help if
the MTU is > 65535 (the Milter's 'packet' size).
I would therefore not expect Nagle delays on en01.
I compared some tcpdumps with postfix as client and my milterclient
(which is fast) as client.
It seems that we run into nagle timeout (the ACK is always delayed).
Setting TCP_QUICKACK on my milter socket doesnt help, is there a way to
set TCP_NODELAY on the milter connection in postfix?
You've got the source.
Wietse
