On Thu, Jan 27, 2011 at 08:39:40PM +0100, Damien Fleuriot wrote: > > > On 1/27/11 7:46 PM, Sergey Lobanov wrote: > > В сообщении от Пятница 28 января 2011 00:55:35 автор Damien Fleuriot > > написал: > >> On 1/27/11 6:41 PM, Vogel, Jack wrote: > >>> Jeremy is right, if you have a problem the first step is to try the > >>> latest code. > >>> > >>> However, when I look at the interrupts below I don't see what the problem > >>> is? The Broadcom seems to have about the same rate, it just doesn't have > >>> MSIX (multiple vectors). > >>> > >>> Jack > >> > >> My main concern is that the CPU %interrupt is quite high, also, we seem > >> to be experiencing input errors on the interfaces. > > Would you show igb tuning which is done in loader.conf and output of sysctl > > dev.igb.0? > > Did you rise number of igb descriptors such as: > > hw.igb.rxd=4096 > > hw.igb.txd=4096 ? > > There is no tuning at all on our part in the loader's conf. > > Find below the sysctls: > > # sysctl -a |grep igb > dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 1.7.3 > dev.igb.0.%driver: igb > dev.igb.0.%location: slot=0 function=0 > dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10d6 subvendor=0x8086 > subdevice=0x145a class=0x020000 > dev.igb.0.%parent: pci14 > dev.igb.0.debug: -1 > dev.igb.0.stats: -1 > dev.igb.0.flow_control: 3 > dev.igb.0.enable_aim: 1 > dev.igb.0.low_latency: 128 > dev.igb.0.ave_latency: 450 > dev.igb.0.bulk_latency: 1200 > dev.igb.0.rx_processing_limit: 100 > dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 1.7.3 > dev.igb.1.%driver: igb > dev.igb.1.%location: slot=0 function=1 > dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10d6 subvendor=0x8086 > subdevice=0x145a class=0x020000 > dev.igb.1.%parent: pci14 > dev.igb.1.debug: -1 > dev.igb.1.stats: -1 > dev.igb.1.flow_control: 3 > dev.igb.1.enable_aim: 1 > dev.igb.1.low_latency: 128 > dev.igb.1.ave_latency: 450 > dev.igb.1.bulk_latency: 1200 > dev.igb.1.rx_processing_limit: 100
I'm not aware of how to tune igb(4), so the advice Sergey gave you may be applicable. You'll need to schedule downtime to adjust those tunables however (since a reboot will be requried). I also reviewed the munin graphs. I don't see anything necessarily wrong. However, you omitted yearly graphs for the network interfaces. Why I care about that: The pf state table (yearly) graph basically correlates with the CPU usage (yearly) graph, and I expect that the yearly network graphs would show a similar trend: an increase in your overall traffic over the course of a year. What I'm trying to figure out is what you're concerned about. You are in fact pushing anywhere between 60-120MBytes/sec across these interfaces. Given those numbers, I'm not surprised by the ""high"" interrupt usage. Graphs of this nature usually indicate that you're hitting a "bottleneck" (for lack of better word) where you're simply doing "too much" with a single machine (given its network throughput). The machine is spending a tremendous amount of CPU time handling network traffic, and equally as much with regards to the pf usage. If you want my opinion based on the information I have so far, it's this: you need to scale your infrastructure. You can no longer rely on a single machine to handle this amount of traffic. As for the network errors you see -- to get low-level NIC and driver statistics, you'll need to run "sysctl dev.igb.X.stats=1" then run "dmesg" and look at the numbers shown (the sysctl command won't output anything itself). This may help indicate where the packets are being lost. You should also check the interface counters on the switch which these interfaces are connected to. I sure hope it's a managed switch which can give you those statistics. Hope this helps, or at least acts as food for thought. -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | _______________________________________________ freebsd-pf@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-pf To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"
