Re: Too much interrupts on ixgbe
25.10.2011 11:21, Sergey Saley ???: Jack Vogel wrote: On Tue, Oct 25, 2011 at 12:22 AM, Sergey Saley<sergeysaley@>wrote: Ryan Stone-2 wrote: On Mon, Oct 24, 2011 at 3:51 PM, Sergey Saley<sergeysaley@> wrote: MPD5, netgraph, pppoe.Types of traffic - any (customer traffic). Bying this card I counted on a 3-4G traffic at 3-4K pppoe sessions. It turned to 600-700Mbit/s, about 50K pps at 700-800 pppoe sessions. PPPoE is your problem. The Intel cards can't load-balance PPPoE traffic, so everything goes to one queue. It may be possible to write a netgraph module to load-balance the traffic across your CPUs. OK, thank You for explanation. And what about the large number of interrupts? As for me, it's too much... irq256: ix0:que 0 240536944 6132 irq257: ix0:que 1 89090444 2271 irq258: ix0:que 2 93222085 2376 irq259: ix0:que 3 89435179 2280 irq260: ix0:link 1 0 irq261: ix1:que 0 269468769 6870 irq262: ix1:que 1 110974 2 irq263: ix1:que 2 434214 11 irq264: ix1:que 3 112281 2 irq265: ix1:link 1 0 How do you decide its 'too much' ? It may be that with your traffic you end up not being able to use offloads, just thinking. Its not like the hardware just "makes it up", it interrupts on the last descriptor of a packet which has the RS bit set. With TSO you will get larger chunks of data and thus less interrupts but your traffic probably doesn't qualify for it. It's easy. I have several servers with a similar task and load. About 30K pps, about 500-600M traffic, about 600-700 pppoe connections. One difference - em Here is a typical vmstat -i point06# vmstat -i interrupt total rate irq17: atapci0 6173367 0 cpu0: timer 3904389748465 irq256: em0 3754877950447 irq257: em1 2962728160352 cpu2: timer 3904389720465 cpu1: timer 3904389720465 cpu3: timer 3904389721465 Total22341338386 2661 point05# vmstat -i interrupt total rate irq14: ata0 35 0 irq19: atapci1 8323568 0 cpu0: timer 3905440143465 irq256: em0 3870403571461 irq257: em1 1541695487183 cpu1: timer 3905439895465 cpu3: timer 3905439895465 cpu2: timer 3905439895465 Total21042182489 2506 point04# vmstat -i interrupt total rate irq19: atapci0 6047874 0 cpu0: timer 3901683760464 irq256: em0823774953 98 irq257: em1 1340659093159 cpu1: timer 3901683730464 cpu2: timer 3901683730464 cpu3: timer 3901683730464 Total1216870 2117 BTW, maybe there is a possibility to make a traffic separation per several queues by vlan tag? That would be a partial solution... ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Too much interrupts on ixgbe
There is my FreeBSD box: kernel --- # # GENERIC -- Generic kernel configuration file for FreeBSD/i386 # # For more information on this file, please read the config(5) manual page, # and/or the handbook section on Kernel Configuration Files: # # http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html # # The handbook is also available locally in /usr/share/doc/handbook # if you've installed the doc distribution, otherwise always see the # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the # latest information. # # An exhaustive list of options and more detailed explanations of the # device lines is also present in the ../../conf/NOTES and NOTES files. # If you are in doubt as to the purpose or necessity of a line, check first # in NOTES. # # $FreeBSD: head/sys/i386/conf/GENERIC 221743 2011-05-10 16:44:16Z jkim $ cpu I686_CPU ident POINT07 options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption options INET# InterNETworking options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_DIRHASH # Improve performance on big directories options MD_ROOT # MD is a potential root device options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS# Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_LABEL # Provides labelization options KTRACE # ktrace(1) support options STACK # stack(9) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options PRINTF_BUFR_SIZE=128# Prevent printf output being interspersed. options KBD_INSTALL_CDEV# install a CDEV entry in /dev options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) #optionsKDTRACE_HOOKS # Kernel DTrace hooks options INCLUDE_CONFIG_FILE # Include this file in kernel # To make an SMP kernel, the next two lines are needed options SMP # Symmetric MultiProcessor Kernel device apic# I/O APIC # CPU frequency control device cpufreq # Bus support. device acpi device pci # Floppy drives device fdc # ATA controllers device ahci# AHCI-compatible SATA controllers device ata # Legacy ATA/SATA controllers options ATA_CAM # Handle legacy controllers with CAM options ATA_STATIC_ID # Static device numbering device mvs # Marvell 88SX50XX/88SX60XX/88SX70XX/SoC SATA device siis# SiliconImage SiI3124/SiI3132/SiI3531 SATA # ATA/SCSI peripherals device scbus # SCSI bus (required for ATA/SCSI) device ch # SCSI media changers device da # Direct Access (disks) device sa # Sequential Access (tape etc) device cd # CD device pass# Passthrough device (direct ATA/SCSI access) device ses # SCSI Environmental Services (and SAF-TE) # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device kbdmux # keyboard multiplexer device vga # VGA video card driver # syscons is the default console driver, resembling an SCO console device sc options SC_PIXEL_MODE # add support for the raster text mode device agp # support several AGP chipsets # Power management support (see NOTES for more options) #device apm # Add suspend/resume support for the i8254. device pmtimer # Serial (COM) ports device uart# Generic UART driver # Parallel port device ppc device ppbus # Parallel port bus (required) device lpt # Printer #device plip# TCP/IP over parallel device ppi # Parallel port interface device #device vpo # Requires scbus and da # If you've got a "dumb" serial or parallel PCI card that is # supported by the puc(4) glue driver, uncomment the following # li
Re: Too much interrupts on ixgbe
Jack Vogel wrote: > > You need to increase your mbuf pool, note below in your messages where it > has > insufficient to configure for ix1, I don't know that this will change the > interrupt > difference but it should be addressed anyway. > Thank You for answer! Nobody's fault but mine :-( Being fixing... Jack Vogel wrote: > > I also noticed that you have the adapter in a PCIE 1.0 slot (only 2.5 > Gb/s), > that > will limit you, you should move to a 2.0 slot if possible. > PCI Express Bus: Speed 2.5Gb/s Width x8 Does it not enough? -- View this message in context: http://freebsd.1045724.n5.nabble.com/Too-much-interrupts-on-ixgbe-tp4931883p4933498.html Sent from the freebsd-net mailing list archive at Nabble.com. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Too much interrupts on ixgbe
Jack Vogel wrote: > > On Mon, Oct 24, 2011 at 10:46 AM, Sergey Saley <sergeysaley@>wrote: > >> >> Jack Vogel wrote: >> > >> > You need to increase your mbuf pool, note below in your messages where >> it >> > has >> > insufficient to configure for ix1, I don't know that this will change >> the >> > interrupt >> > difference but it should be addressed anyway. >> > >> >> > Nothing has changed after increasing mbufs count increasing: irq256: ix0:que 03013004 2431 irq257: ix0:que 1 970295783 irq258: ix0:que 2 574782463 irq259: ix0:que 3 520764420 irq260: ix0:link 1 0 irq261: ix1:que 03185946 2571 irq262: ix1:que 1 20425 16 irq263: ix1:que 2 10098 8 irq264: ix1:que 3 6999 5 irq265: ix1:link 3 0 Still only one irq being really used for ix1... -- View this message in context: http://freebsd.1045724.n5.nabble.com/Too-much-interrupts-on-ixgbe-tp4931883p4933765.html Sent from the freebsd-net mailing list archive at Nabble.com. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Too much interrupts on ixgbe
Ryan Stone-2 wrote: > > On Mon, Oct 24, 2011 at 3:08 PM, Sergey Saley <sergeysaley@> wrote: >> Nothing has changed after increasing mbufs count increasing: >> >> irq256: ix0:que 0 3013004 2431 >> irq257: ix0:que 1 970295 783 >> irq258: ix0:que 2 574782 463 >> irq259: ix0:que 3 520764 420 >> irq260: ix0:link 1 0 >> irq261: ix1:que 0 3185946 2571 >> irq262: ix1:que 1 20425 16 >> irq263: ix1:que 2 10098 8 >> irq264: ix1:que 3 6999 5 >> irq265: ix1:link 3 0 > > What kind of traffic are you sending? How many flows? Is it straight > Ethernet/IP/TCP(/UDP), or is there any kind of tunneling in use? > > MPD5, netgraph, pppoe.Types of traffic - any (customer traffic). Bying this card I counted on a 3-4G traffic at 3-4K pppoe sessions. It turned to 600-700Mbit/s, about 50K pps at 700-800 pppoe sessions. -- View this message in context: http://freebsd.1045724.n5.nabble.com/Too-much-interrupts-on-ixgbe-tp4931883p4933934.html Sent from the freebsd-net mailing list archive at Nabble.com. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Too much interrupts on ixgbe
Arnaud Lacombe-6 wrote: > > Hi, > > On Mon, Oct 24, 2011 at 4:32 PM, Ryan Stone <rysto32@> wrote: >> On Mon, Oct 24, 2011 at 3:51 PM, Sergey Saley <sergeysaley@> wrote: >>> MPD5, netgraph, pppoe.Types of traffic - any (customer traffic). >>> Bying this card I counted on a 3-4G traffic at 3-4K pppoe sessions. >>> It turned to 600-700Mbit/s, about 50K pps at 700-800 pppoe sessions. >> >> PPPoE is your problem. The Intel cards can't load-balance PPPoE >> traffic, so everything goes to one queue. It may be possible to write >> a netgraph module to load-balance the traffic across your CPUs. >> > NetGraph already runs a thread per CPU, as you can see in top's > output. There is also still have plenty of CPU usable, so I'd assume > some hard-limit are hit, maybe any of `net.graph.maxdata' or > `net.graph.maxalloc'. > > Sergey, what is the output of: > > # vmstat -z | grep NetGraph > > # vmstat -z | grep NetGraph ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP NetGraph items: 72, 4118, 2, 404,40806244, 0, 0 NetGraph data items: 72,522, 0, 406,658514523, 0, 0 -- View this message in context: http://freebsd.1045724.n5.nabble.com/Too-much-interrupts-on-ixgbe-tp4931883p4935167.html Sent from the freebsd-net mailing list archive at Nabble.com. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Too much interrupts on ixgbe
Ryan Stone-2 wrote: > > On Mon, Oct 24, 2011 at 3:51 PM, Sergey Saley <sergeysaley@> wrote: >> MPD5, netgraph, pppoe.Types of traffic - any (customer traffic). >> Bying this card I counted on a 3-4G traffic at 3-4K pppoe sessions. >> It turned to 600-700Mbit/s, about 50K pps at 700-800 pppoe sessions. > > PPPoE is your problem. The Intel cards can't load-balance PPPoE > traffic, so everything goes to one queue. It may be possible to write > a netgraph module to load-balance the traffic across your CPUs. > OK, thank You for explanation. And what about the large number of interrupts? As for me, it's too much... irq256: ix0:que 0 240536944 6132 irq257: ix0:que 1 89090444 2271 irq258: ix0:que 2 93222085 2376 irq259: ix0:que 3 89435179 2280 irq260: ix0:link 1 0 irq261: ix1:que 0 269468769 6870 irq262: ix1:que 1 110974 2 irq263: ix1:que 2 434214 11 irq264: ix1:que 3 112281 2 irq265: ix1:link 1 0 -- View this message in context: http://freebsd.1045724.n5.nabble.com/Too-much-interrupts-on-ixgbe-tp4931883p4935272.html Sent from the freebsd-net mailing list archive at Nabble.com. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Too much interrupts on ixgbe
Jack Vogel wrote: > > On Tue, Oct 25, 2011 at 12:22 AM, Sergey Saley <sergeysaley@>wrote: > >> >> Ryan Stone-2 wrote: >> > >> > On Mon, Oct 24, 2011 at 3:51 PM, Sergey Saley <sergeysaley@> >> wrote: >> >> MPD5, netgraph, pppoe.Types of traffic - any (customer traffic). >> >> Bying this card I counted on a 3-4G traffic at 3-4K pppoe sessions. >> >> It turned to 600-700Mbit/s, about 50K pps at 700-800 pppoe sessions. >> > >> > PPPoE is your problem. The Intel cards can't load-balance PPPoE >> > traffic, so everything goes to one queue. It may be possible to write >> > a netgraph module to load-balance the traffic across your CPUs. >> > >> >> OK, thank You for explanation. >> And what about the large number of interrupts? >> As for me, it's too much... >> irq256: ix0:que 0 240536944 6132 >> irq257: ix0:que 1 89090444 2271 >> irq258: ix0:que 2 93222085 2376 >> irq259: ix0:que 3 89435179 2280 >> irq260: ix0:link 1 0 >> irq261: ix1:que 0 269468769 6870 >> irq262: ix1:que 1 110974 2 >> irq263: ix1:que 2 434214 11 >> irq264: ix1:que 3 112281 2 >> irq265: ix1:link 1 0 >> >> > How do you decide its 'too much' ? It may be that with your traffic you > end > up > not being able to use offloads, just thinking. Its not like the hardware > just "makes > it up", it interrupts on the last descriptor of a packet which has the RS > bit set. > With TSO you will get larger chunks of data and thus less interrupts but > your > traffic probably doesn't qualify for it. > It's easy. I have several servers with a similar task and load. About 30K pps, about 500-600M traffic, about 600-700 pppoe connections. One difference - em Here is a typical vmstat -i point06# vmstat -i interrupt total rate irq17: atapci0 6173367 0 cpu0: timer 3904389748465 irq256: em0 3754877950447 irq257: em1 2962728160352 cpu2: timer 3904389720465 cpu1: timer 3904389720465 cpu3: timer 3904389721465 Total22341338386 2661 point05# vmstat -i interrupt total rate irq14: ata0 35 0 irq19: atapci1 8323568 0 cpu0: timer 3905440143465 irq256: em0 3870403571461 irq257: em1 1541695487183 cpu1: timer 3905439895465 cpu3: timer 3905439895465 cpu2: timer 3905439895465 Total21042182489 2506 point04# vmstat -i interrupt total rate irq19: atapci0 6047874 0 cpu0: timer 3901683760464 irq256: em0823774953 98 irq257: em1 1340659093159 cpu1: timer 3901683730464 cpu2: timer 3901683730464 cpu3: timer 3901683730464 Total1216870 2117 -- View this message in context: http://freebsd.1045724.n5.nabble.com/Too-much-interrupts-on-ixgbe-tp4931883p4935427.html Sent from the freebsd-net mailing list archive at Nabble.com. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Too much interrupts on ixgbe
Hi! To be continued... I replaced 82598 (2 cx4 ports) with 82599 (2 sfp+ ports) without any configuration changes. The same host, same system, same tuning, same load and traffic. Was (82598): # vmstat -i interrupt total rate irq19: atapci0 1667110 0 cpu0: timer 3386721810328 irq256: ix0:que 0 3395843376329 irq257: ix0:que 1 2642665824256 irq258: ix0:que 2 2838302235275 irq259: ix0:que 3 2176207954211 irq260: ix0:link 18 0 irq261: ix1:que 0 282359321 27 irq262: ix1:que 1 3989170496387 irq263: ix1:que 2 375956573 36 irq264: ix1:que 3 3966352151384 irq265: ix1:link 1 0 irq266: em0:rx 010283114 0 irq269: em1:rx 010283114 0 cpu3: timer 3386697130328 cpu1: timer 3386709028328 cpu2: timer 3386700908328 Total33235920163 3225 Now (82599): # vmstat -i interrupt total rate irq1: atkbd0 353 1 irq19: atapci0 1368 3 cpu0: timer 682557 1978 irq256: ix0:que 03555342 10305 irq257: ix0:que 13120223 9044 irq258: ix0:que 23408333 9879 irq259: ix0:que 33279717 9506 irq260: ix0:link 4 0 irq261: ix1:que 02905248 8421 irq262: ix1:que 12647858 7674 irq263: ix1:que 22411908 6991 irq264: ix1:que 32319469 6723 irq265: ix1:link 3 0 irq266: em0:rx 0 338 0 irq269: em1:rx 0 338 0 cpu1: timer 682529 1978 cpu2: timer 682527 1978 cpu3: timer 682527 1978 Total 26380642 76465 Why interrupt count is so huge? What's wrong with this card (or with driver)? -- View this message in context: http://freebsd.1045724.n5.nabble.com/Too-much-interrupts-on-ixgbe-tp4931883p4988068.html Sent from the freebsd-net mailing list archive at Nabble.com. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
