----- Original Message -----
From: "Andre Oppermann" <an...@freebsd.org>
To: "Steven Hartland" <kill...@multiplay.co.uk>
Cc: <freebsd-net@freebsd.org>; <lstew...@freebsd.org>
Sent: Tuesday, August 02, 2011 10:25 PM
Subject: Re: tcp failing to recover from a packet loss under 8.2-RELEASE?
On 02.08.2011 14:32, Steven Hartland wrote:
----- Original Message ----- From: "Steven Hartland"
Setting net.inet.tcp.reass.maxsegments=8148 and rerunning the
tests appears to result in a solid 14MB/s, its still running a
full soak test but looking very promising :)
Ok so full test has completed with over 134GB of data transferred
without a hitch.
I haven't been able to complete this test without error on this
target machine before the change to maxsegments; so packets being
dropped in the tcp reassembly code due to tcp_reass global zone
exhaustion is almost certainly the cause of the stalls we're
seeing, which is good news :)
The zone exhaustion is part of the problem but not the real cause
of the stall you're seeing. When the reassembly zone is exhausted
the retransmit that would fill the hole between the socket buffer
and the out-of-order segments in the reassembly queue can't be
processed anymore. This brings all TCP sessions with data in the
reassembly queue to a standstill.
I suspect there are a few contributing factors at play causing
this.
1. There is known to be a dodgy fibre on the test routing, which
is scheduled for cleaning, but is currently causing a small amount
of packet loss between the sender and receiver.
Packet loss and the use of the reassembly queue.
2. The target machine has 24 cores and is running 8 queues on
the igb interface which could result in the requirement to reorder
packets even if they arrived in order on the wire. I say this as
disabling msix on igb0, which results in just one queue, did reduce
the occurrence rate of the problem.
Maybe this is a compounding factor.
3. Combining a low latency (<1ms) high throughput connection
~64MB/s with a lower throughput but still relatively high bandwidth
~14MB/s with a 10ms latency.
What matters here is the socket buffer and receive window size.
On the reciever the only none default sysclt's are:-
vfs.read_max=32
net.inet.tcp.inflight.enable=0
net.inet.tcp.sendspace=65536
kern.ipc.maxsockbuf=524288
kern.maxfiles=50000
I look forward to hearing peoples thoughts on what the actual fix
should be: increased default nmbclusters, decreased nmbclusters =>
maxsegments divisor, or something else?
These items may or may not need some adjustments. Though it will
eventually wedge again. The reassembly queue must be able to process
the one missing segment despite having exhausted the global zone limit.
I had fixed that at one point in time. It seems that it got lost with
the recent changes.
Please try this patch:
http://people.freebsd.org/~andre/tcp_reass.c-logdebug+missingsegment-20110802.diff
Would it not make sense to always use stack for the missing segment, as
this would avoid the allocation e.g.
http://www.multiplaygameservers.com/dropzone/tcp_reass.c-logdebug+missingsegment-20110803.diff
Also added a two additional cases to protect against using uma_zfree on
the stack based tqs, not 100% clear if these could be triggered or
not, but thought I'd check?
Regards
Steve
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please
telephone +44 845 868 1337
or return the E.mail to postmas...@multiplay.co.uk.
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
