On Mon, 15 Dec 2008, Robert Watson wrote:
On Mon, 15 Dec 2008, Yony Yossef wrote:
I'm testing an Ethernet driver on FreeBSD 6.3.
Running netstat -m during an ethernt stress test I see that the
"mbuf+clusters out of packet secondary zone in use" number is growing
gradualy. Problem is it never goes down after I stop the test, so it's
pushing the "mbufs in use" up until the following stress test iterations
reach the OS limit. What does this number mean?
I realized that I didn't quite answer your question here, so to be more
specific: the FreeBSD mbuf allocator has a number of slab allocator zones it
can draw from, depending on the type of request. These are enumerated in the
vmstat -z output:
rob...@fledge:~> vmstat -z | head -1 ; vmstat -z | grep -i mbuf
ITEM SIZE LIMIT USED FREE REQUESTS FAILURES
mbuf_packet: 256, 0, 262, 222, 202448978, 0
mbuf: 256, 0, 9, 527, 686757049, 0
mbuf_cluster: 2048, 25600, 484, 304, 4017957, 0
mbuf_jumbo_pagesize: 4096, 12800, 7, 298, 18924376, 0
mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 406, 8873247, 0
The 'mbuf' zone allocates simple mbufs from a cache. The various cluster
zones allocate cluster storage of various sizes, from the 2k default cluster
size up to various jumbo sizes used when sending large amounts of data or when
jumbograms are configured for a network interface that supports them.
The mbuf_packet (mbuf+cluster) zone is a special zone allocating mbufs with
pre-attached clusters, which was an optimization that seemed to help a lot a
few years ago. In particular, you don't have to enter the memory allocator
twice in order to find an mbuf with a cluster. There are some downsides, not
least more complicated book-keeping, and some issues with how to account for
and manage memory across the various caches. Notice that all mbuf_packet FREE
(cached) clusters are billed as used clusters for the mbuf_cluster zone, as
while UMA knows that mbuf_packet counts as a regular mbuf allocation, it
doesn't know about the cluster hooked off it; netstat -m adjusts the reported
cluster allocation for this before printing stats.
Recently, we've been discussing moving to a variable-size mbuf scheme
supporting large mbuf sizes with embedded storage, which seems to improve
memory efficient quite a bit. There are some downsides -- one issue is that
it somewhat complicates reference-counted cluster use (although not much);
another is that data is no longer page-aligned which will make page-flipping
less convenient on the receiver. Currently, if the hardware supports
header-spitting, you can imagine the data going fairly easily into
appropriately sized clusters (especially if the hardware supports LRO
directly), and then flipping the pages into user memory on receive. With
mbufs stuck on the front end of the page, not so much.
Robert N M Watson
Computer Laboratory
University of Cambridge
It seems likely that one of two things is happening:
(1) a leak of mbuf + clusters
(2) a bug in stats on mbuf + clusters
Could you paste the output of "vmstat -z | grep -i mbuf" into an e-mail?
That's the underlying vm stats from the zone used to generate netstat -m
output, and might shed more light on things.
Robert N M Watson
Computer Laboratory
University of Cambridge
506391/126009/632400 mbufs in use (current/cache/total)
141035/121109/262144/262144 mbuf clusters in use (current/cache/total/max)
131054/610 mbuf+clusters out of packet secondary zone in use
(current/cache)
Thanks
Yony
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
