While investigating this further, I read on
http://www.masterraghu.com/subjects/np/introduction/unix_network_programming_v1.3/ch07lev1sec5.html
under "SO_RCVBUF and SO_SNDBUF Socket Options":
When setting the size of the TCP socket receive buffer, the
ordering of the function calls is important. This is because of
TCP's window scale option (Section 2.6), which is exchanged with
the peer on the SYN segments when the connection is established.
For a client, this means the SO_RCVBUF socket option must be set
before calling connect. For a server, this means the socket option
must be set for the listening socket before calling listen. Setting
this option for the connected socket will have no effect whatsoever
on the possible window scale option because accept does not return
with the connected socket until TCP's three-way handshake is
complete. That is why this option must be set for the listening
socket. (The sizes of the socket buffers are always inherited from
the listening socket by the newly created connected socket: pp.
462–463 of TCPv2.)
As mentioned in a previous post, I had already discovered about
needing to set the socket buffers before connect, but I didn't know
about setting them before the call to listen() in order to get the
buffer sizes inherited by the accepted sockets.
After fixing this in my test program, all problems disappeared when
keeping the send and receive buffers the same on both sides.
However, when only setting the send and receive buffers on the client
socket (not on the (accepted or) listen socket), epoll_wait() still
stalls 43ms. When the SO_SNDBUF is smaller than 33182 bytes.
Here is the latest version of my test program:
https://github.com/CarloWood/ai-evio-testsuite/blob/master/src/epoll_bug.c
I have to retract most of my "bug" report, it might even not really be
a bug then... but nevertheless, what remains strange is the fact
that setting the socket buffer sizes on the accepted sockets can lead
to so much crippling effect, while the quoted website states:
Setting this option for the connected socket will have no effect
whatsoever on the possible window scale option because accept does
not return with the connected socket until TCP's three-way
handshake is complete.
And when only setting the socket buffer sizes for the client socket
(that I use to send back received data; so this is the sending
side now) then why does epoll_wait() stall 43 ms per call when the
receiving side is using the default (much larger) socket buffer sizes?
That 43 ms is STILL crippling-- slowing down the transmission of the
data to a trickling speed compared to what it should be.
--
Carlo Wood <ca...@alinoe.com>
