Hi,
I am running FreeBSD 8.1p3 amd64 with 6Gb of RAM. This
is a router using Quagga for BGP. It has two 10Gb/s network connections
using the intel ixgbe 2.4.4 driver. It is handling routing for about
200 remote sites and about 300k simultaneous sessions through the box.
After a few hours at high traffic volumes (just over a 1Gb/s) it will
stop routing any traffic. It look like I am running out of mbuf's. I
have increased them but I am worried that there is a limit as to how
much I can increase them without starving the kernel of space. Console
is still responsive but no network.
I have looked over Tuning(7) and some other great
articles on the net on performance tuning FreeBSD but it has left me a
little confused. Can someone help to explain the best way to
troubleshoot this issue and any formulas to tune it.
Here is some information:
# netstat -m
66042/18063/84105 mbufs in use (current/cache/total)
66039/15931/81970/524288 mbuf clusters in use (current/cache/total/max)
66039/15881 mbuf+clusters out of packet secondary zone in use
(current/cache)
0/5/5/524288 4k (page size) jumbo clusters in use
(current/cache/total/max)
0/0/0/131072 9k jumbo clusters in use (current/cache/total/max)
0/0/0/65536 16k jumbo clusters in use (current/cache/total/max)
148588K/36397K/184986K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
# more /boot/loader.conf
geom_mirror_load="YES"
loader_logo="beastie"
vesa_load="YES"
ixgbe_load="YES"
hw.ixgbe.txd=4096
hw.ixgbe.rxd=4096
kern.ipc.nmbjumbop=524288
kern.ipc.nmbclusters=524288
kern.ipc.maxsockbuf=4194304
# more /etc/sysctl.conf
#security.bsd.see_other_uids=0
kern.ipc.nmbjumbop=524288
kern.ipc.nmbclusters=524288
kern.ipc.maxsockbuf=4194304
hw.intr_storm_threshold=9000
net.inet.icmp.icmplim=1000
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights
reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-RELEASE-p1 #3: Thu Mar 29 08:53:27 MDT 2012
r...@schoolgw2.epsb.ca:/usr/obj/usr/src/sys/DELLR710x64 amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU X5560 @ 2.80GHz (2793.02-MHz
K8-class CPU)
Origin = "GenuineIntel" Id = 0x106a5 Family = 6 Model = 1a
Stepping = 5
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE
,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x9ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PD
CM,DCA,SSE4.1,SSE4.2,POPCNT>
AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 6442450944 (6144 MB)
avail memory = 6170185728 (5884 MB)
ACPI APIC Table: <DELL PE_SC3 >
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads
cpu0 (BSP): APIC ID: 16
cpu1 (AP): APIC ID: 17
cpu2 (AP): APIC ID: 18
cpu3 (AP): APIC ID: 19
cpu4 (AP): APIC ID: 20
cpu5 (AP): APIC ID: 21
cpu6 (AP): APIC ID: 22
cpu7 (AP): APIC ID: 23
cpu8 (AP): APIC ID: 0
cpu9 (AP): APIC ID: 1
cpu10 (AP): APIC ID: 2
cpu11 (AP): APIC ID: 3
cpu12 (AP): APIC ID: 4
cpu13 (AP): APIC ID: 5
cpu14 (AP): APIC ID: 6
cpu15 (AP): APIC ID: 7
....
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.4>
port 0xbcc0-0xbcdf mem 0xdd800000-0xdd9fffff,0xdd7f8000-0xdd7fbfff irq
50 at device 0.0 on pci8
ix0: Using MSIX interrupts with 9 vectors
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: Ethernet address: 00:1b:21:d8:d8:be
ix0: PCI Express Bus: Speed 5.0Gb/s Width x8
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.4>
port 0xbce0-0xbcff mem 0xdda00000-0xddbfffff,0xdd7fc000-0xdd7fffff irq
40 at device 0.1 on pci8
ix1: Using MSIX interrupts with 9 vectors
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: Ethernet address: 00:1b:21:d8:d8:bf
ix1: PCI Express Bus: Speed 5.0Gb/s Width x8
# ifconfig -a
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
options=405b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLA
N_HWTSO>
ether 00:1b:21:d8:d8:be
inet 192.168.248.228 netmask 0xffffffe0 broadcast
192.168.248.255
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
options=405b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLA
N_HWTSO>
ether 00:1b:21:d8:d8:bf
inet 192.168.250.40 netmask 0xffffff00 broadcast 192.168.250.255
inet 192.168.250.23 netmask 0xffffff00 broadcast 192.168.250.255
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
---- Kirk
Kirk Davis
Supervisor - Network Operations
ITS, Edmonton Public Schools
One Kingsway Ave. NW
Edmonton, Alberta
T5H 4G9
(780) 429-8308
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
