On Wed, Feb 22, 2012 at 11:56:29AM -0800, Jack Vogel wrote:
> Using igb and/or ixgbe on a reasonably powered server requires 1K mbuf
> clusters per MSIX vector,
> that's how many are in a ring. Either driver will configure 8 queues on a
> system with that many or more
> cores, so 8K clusters per port...
>
> My test engineer has a system with 2 igb ports, and 2 10G ixgbe, this is
> hardly heavy duty, and yet this
> exceeds the default mbuf pool on the installed kernel (1024 + maxusers *
> 64).
>
> Now, this can be immediately fixed by a sysadmin after that first boot, but
> it does result in the second
> driver that gets started to complain about inadequate buffers.
>
> I think the default calculation is dated and should be changed, but am not
> sure the best way, so are
> there suggestions/opinions about this, and might we get it fixed before 8.3
> is baked?
I have hit this problem recently, too.
Maybe the issue mostly/only exists on 32-bit systems.
Here is a possible approach:
1. nmbclusters consume the kernel virtual address space so there
must be some upper limit, say
VM_LIMIT = 256000 (translates to 512MB of address space)
2. also you don't want the clusters to take up too much of the available
memory. This one would only trigger for minimal-memory systems,
or virtual machines, but still...
MEM_LIMIT = (physical_ram / 2) / 2048
3. one may try to set a suitably large, desirable number of buffers
TARGET_CLUSTERS = 128000
4. and finally we could use the current default as the absolute minimum
MIN_CLUSTERS = 1024 + maxusers*64
Then at boot the system could say
nmbclusters = min(TARGET_CLUSTERS, VM_LIMIT, MEM_LIMIT)
nmbclusters = max(nmbclusters, MIN_CLUSTERS)
In turn, i believe interfaces should do their part and by default
never try to allocate more than a fraction of the total number
of buffers, if necessary reducing the number of active queues.
what do people think ?
cheers
luigi
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
