On 11.03.2013 08:52, Ihsan Junaidi Ibrahim wrote:
Hi,
I'm on 9.0-RELEASE-p3 and have had a number of instances where my igb0 network
connectivity locked up under heavy load.
This problem is also known on CURRENT and we are under active investigation
on how to solve it properly.
I've had another em1 device which I used for side-band access which does not
have as much load as my primary igb0 device.
Jumbo mbufs are only used if you increase the MTU on your interface.
As long as you stay on the standard ethernet MTU of 1500 no jumbo
mbufs will be used.
I've read previous cases of this but can't seem to find any fitting solutions.
I've tried increasing mbufs and jumbo mbufs:
nsv01-kul:ihsan $ sysctl -a | grep nmb
kern.ipc.nmbjumbo16: 3200
kern.ipc.nmbjumbo9: 25600
kern.ipc.nmbjumbop: 12800
kern.ipc.nmbclusters: 262144
but I'm still receiving a lot of 'requests for jumbo clusters denied' stats on
netstat -m output.
nsv01-kul:ihsan $ netstat -m
7347/3063/10410 mbufs in use (current/cache/total)
334/2658/2992/262144 mbuf clusters in use (current/cache/total/max)
334/1330 mbuf+clusters out of packet secondary zone in use (current/cache)
1749/1310/3059/12800 4k (page size) jumbo clusters in use
(current/cache/total/max)
5116/563/5679/25600 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
55544K/16388K/71933K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/25154247/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
The server has not locked up yet after increasing the mbufs (been less than 8
hours since increasing this) but can someone enlighten me what could be the
causes for jumbo clusters requests being denied although I've increased the 9k
jumbo clusters to 25600?
The problem is not the limits but memory fragmentation. Jumbo
mbufs larger than 4k (PAGE_SIZE) have to be created with adjacent
and contiguous VM pages in physical and virtual memory. As soon
as your server becomes busy the physical memory becomes 'fragmented'
in the sense that such jumbo mbuf requests can't be fulfilled anymore
at allocation time.
Most modern high speed network cards can do scatter DMA and could
simply split up a received jumbo frame across as many normal mbuf
clusters as necessary.
For a temporary fix you could forgo using a MTU larger than 4096.
--
Andre
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
