--- Begin Message ---
On 21 Oct 2021, at 17:51, Bob McMahon via Make-wifi-fast
<make-wifi-f...@lists.bufferbloat.net> wrote:
> Hi All,
>
> Sorry for the spam. I'm trying to support a meaningful TCP message latency
> w/iperf 2 from the sender side w/o requiring e2e clock synchronization. I
> thought I'd try to use the TCP_NOTSENT_LOWAT event to help with this. It
> seems that this event goes off when the bytes are in flight vs have reached
> the destination network stack. If that's the case, then iperf 2 client
> (sender) may be able to produce the message latency by adding the drain time
> (write start to TCP_NOTSENT_LOWAT) and the sampled RTT.
>
> Does this seem reasonable?
I’m not 100% sure what you’re asking, but I will try to help.
When you set TCP_NOTSENT_LOWAT, the TCP implementation won’t report your
endpoint as writable (e.g., via kqueue or epoll) until less than that threshold
of data remains unsent. It won’t stop you writing more bytes if you want to, up
to the socket send buffer size, but it won’t *ask* you for more data until the
TCP_NOTSENT_LOWAT threshold is reached. In other words, the TCP implementation
attempts to keep BDP bytes in flight + TCP_NOTSENT_LOWAT bytes buffered and
ready to go. The BDP of bytes in flight is necessary to fill the network pipe
and get good throughput. The TCP_NOTSENT_LOWAT of bytes buffered and ready to
go is provided to give the source software some advance notice that the TCP
implementation will soon be looking for more bytes to send, so that the buffer
doesn’t run dry, thereby lowering throughput. (The old SO_SNDBUF option
conflates both “bytes in flight” and “bytes buffered and ready to go” into the
same number.)
If you wait for the TCP_NOTSENT_LOWAT notification, write a chunk of n bytes of
data, and then wait for the next TCP_NOTSENT_LOWAT notification, that will tell
you roughly how long it took n bytes to depart the machine. You won’t know why,
though. The bytes could depart the machine in response for acks indicating that
the same number of bytes have been accepted at the receiver. But the bytes can
also depart the machine because CWND is growing. Of course, both of those
things are usually happening at the same time.
How to use TCP_NOTSENT_LOWAT is explained in this video:
<https://developer.apple.com/videos/play/wwdc2015/719/?time=2199>
Later in the same video is a two-minute demo (time offset 42:00 to time offset
44:00) showing a “before and after” demo illustrating the dramatic difference
this makes for screen sharing responsiveness.
<https://developer.apple.com/videos/play/wwdc2015/719/?time=2520>
Stuart Cheshire
--- End Message ---
_______________________________________________
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel
