[Bug 202680] Silent data corruption on em(4) interfaces
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=202680 --- Comment #14 from Dmitry Afanasiev --- I tried to use freebsd-current from nightly snapshot: FreeBSD sunrise0 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r289044: Thu Oct 8 21:21:40 UTC 2015 r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 But nothing changed - after 2 days uptime I again got incorrect MD5 checksum and messages from ssh: Corrupted MAC on input. Disconnecting: Packet corrupt -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: nice stuff from cloudflare (and, we need something like ethtool!)
On Fri, Oct 16, 2015 at 1:25 PM, Jim Thompson wrote: > > > >> On Oct 16, 2015, at 12:06 AM, Ian Smith wrote: >> >>> On Thu, 15 Oct 2015 17:03:55 +0800, Julian Elischer wrote: On 10/10/15 10:59 PM, Luigi Rizzo wrote: the nice folks at cloudflare implemented a nice feature in netmap that puts some queues of the NIC in netmap mode leaving others attached to the host stack https://blog.cloudflare.com/single-rx-queue-kernel-bypass-with-netmap/ and use ethtool (and native NIC filters) to steer traffic around. [FWIW, the chelsio native netmap driver is similar except that the netmap queue has a different MAC address] While their code was developed on linux, it should run almost unmodified on FreeBSD (and we plan to import it soon), except for the fact that we don't have ethtool hence no device-independent mechanism to configure traffic steering. We really need to address the latter. >>> >>> I suspect the answer may be a device dependent sysctl >> >> Interesting; care to flesh out your ideas a bit on how that might work? >> >> I've done nothing more than skim ethtool(8) on linuxcommand.org, and >> wondered why its functionality wasn't incorporated into ifconfig, but >> then ifconfig (on FreeBSD anyway) is tending towards obesity already > > Luigi already did netlink sockets for FreeBSD. > > https://github.com/luigirizzo/netlink-freebsd ha, the netlink for BSD, interesting :) -- Tomorrow Will Never Die ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
[Bug 203630] [Hyper-V] [nat] [tcp] 10.2 NAT bug in TCP stack or hyperv netsvc driver
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203630 --- Comment #13 from Eddy --- (In reply to Wei Hu from comment #12) After some tests, "disable_csum_20151016.patch" doesn't solve the issue for me. The last r285236 patch worked. Do I have to first apply the r285236 patch and then the disable_csum_20151016.patch? I only applied the new patch on clean sources. -- You are receiving this mail because: You are on the CC list for the bug. ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
ixl 40G bad performance?
Hi, I'm running a few simple tests on -CURRENT with a pair of dual-port Intel XL710 boards, which are seen by the kernel as: ixl0: mem 0xdc80-0xdcff,0xdd808000-0xdd80 irq 32 at device 0.0 on pci3 ixl0: Using MSIX interrupts with 33 vectors ixl0: f4.40 a1.4 n04.53 e80001dca ixl0: Using defaults for TSO: 65518/35/2048 ixl0: Ethernet address: 68:05:ca:32:0b:98 ixl0: PCI Express Bus: Speed 8.0GT/s Width x8 ixl0: netmap queues/slots: TX 32/1024, RX 32/1024 ixl1: mem 0xdc00-0xdc7f,0xdd80-0xdd807fff irq 32 at device 0.1 on pci3 ixl1: Using MSIX interrupts with 33 vectors ixl1: f4.40 a1.4 n04.53 e80001dca ixl1: Using defaults for TSO: 65518/35/2048 ixl1: Ethernet address: 68:05:ca:32:0b:99 ixl1: PCI Express Bus: Speed 8.0GT/s Width x8 ixl1: netmap queues/slots: TX 32/1024, RX 32/1024 ixl0: link state changed to UP ixl1: link state changed to UP I have two identical machines connected with patch cables (no switch). iperf performance is bad: # iperf -c 10.0.1.2 Client connecting to 10.0.1.2, TCP port 5001 TCP window size: 32.5 KByte (default) [ 3] local 10.0.1.1 port 19238 connected with 10.0.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 3.91 GBytes 3.36 Gbits/sec As is flood ping latency: # sudo ping -f 10.0.1.2 PING 10.0.1.2 (10.0.1.2): 56 data bytes .^C --- 10.0.1.2 ping statistics --- 41927 packets transmitted, 41926 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.084/0.116/0.145/0.002 ms Any ideas on what's going on here? Testing 10G ix interfaces between the same two machines results in 9.39 Gbits/sec and flood ping latencies of 17 usec. Thanks, Lars PS: Full dmesg attached. Copyright (c) 1992-2015 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-CURRENT #2 483de3c(muclab)-dirty: Mon Oct 19 11:01:16 CEST 2015 el...@laurel.muccbc.hq.netapp.com:/usr/home/elars/obj/usr/home/elars/src/sys/MUCLAB amd64 FreeBSD clang version 3.7.0 (tags/RELEASE_370/final 246257) 20150906 VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz (2000.05-MHz K8-class CPU) Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7 Features=0xbfebfbff Features2=0x1fbee3ff AMD Features=0x2c100800 AMD Features2=0x1 XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 137438953472 (131072 MB) avail memory = 133484290048 (127300 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: < > FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs FreeBSD/SMP: 2 package(s) x 8 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 cpu8 (AP): APIC ID: 8 cpu9 (AP): APIC ID: 9 cpu10 (AP): APIC ID: 10 cpu11 (AP): APIC ID: 11 cpu12 (AP): APIC ID: 12 cpu13 (AP): APIC ID: 13 cpu14 (AP): APIC ID: 14 cpu15 (AP): APIC ID: 15 cpu16 (AP): APIC ID: 32 cpu17 (AP): APIC ID: 33 cpu18 (AP): APIC ID: 34 cpu19 (AP): APIC ID: 35 cpu20 (AP): APIC ID: 36 cpu21 (AP): APIC ID: 37 cpu22 (AP): APIC ID: 38 cpu23 (AP): APIC ID: 39 cpu24 (AP): APIC ID: 40 cpu25 (AP): APIC ID: 41 cpu26 (AP): APIC ID: 42 cpu27 (AP): APIC ID: 43 cpu28 (AP): APIC ID: 44 cpu29 (AP): APIC ID: 45 cpu30 (AP): APIC ID: 46 cpu31 (AP): APIC ID: 47 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard random: entropy device external interface module_register_init: MOD_LOAD (vesa, 0x8094fb90, 0) error 19 netmap: loaded module vtvga0: on motherboard smbios0: at iomem 0xf04d0-0xf04ee on motherboard smbios0: Version: 2.7, BCD Revision: 2.7 cryptosoft0: on motherboard aesni0: on motherboard acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 cpu12: on acpi0 cpu13: on acpi0 cpu14: on acpi0 cpu15: on acpi0 cpu16: on acpi0 cpu17: on acpi0 cpu18: on acpi0 cpu19: on acpi0 cpu20: on acpi0 cpu21: on acpi0 cpu22: on acpi0 cpu23: on acpi0 cpu24: on acpi0 cpu25: on acpi0 cpu26: on acpi0 cpu27: on acpi0 cpu28: on acpi0 cpu29: on acpi0 cpu30: on acpi0 cpu31: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 hpet0: iomem 0xfed0-0xfed003ff on acpi0 Timecounter "HP
Re: ixl 40G bad performance?
i would look at the following: - c states and clock speed - make sure you never go below C1, and fix the clock speed to max. Sure these parameters also affect the 10G card, but there may be strange interaction that trigger the power saving modes in different ways - interrupt moderation (may affect ping latency, do not remember how it is set in ixl but probably a sysctl - number of queues (32 is a lot i wouldn't use more than 4-8), may affect cpu-socket affinity - tso and flow director - i have seen bad effects of accelerations so i would run the iperf test with of these features disabled on both sides, and then enable them one at a time - queue sizes - the driver seems to use 1024 slots which is about 1.5 MB queued, which in turn means you have 300us (and possibly half of that) to drain the queue at 40Gbit/s. 150-300us may seem an eternity, but if a couple of cores fall into c7 your budget is gone and the loss will trigger a retransmission and window halving etc. cheers luigi On Mon, Oct 19, 2015 at 6:52 AM, Eggert, Lars wrote: > Hi, > > I'm running a few simple tests on -CURRENT with a pair of dual-port Intel > XL710 boards, which are seen by the kernel as: > > ixl0: mem > 0xdc80-0xdcff,0xdd808000-0xdd80 irq 32 at device 0.0 on pci3 > ixl0: Using MSIX interrupts with 33 vectors > ixl0: f4.40 a1.4 n04.53 e80001dca > ixl0: Using defaults for TSO: 65518/35/2048 > ixl0: Ethernet address: 68:05:ca:32:0b:98 > ixl0: PCI Express Bus: Speed 8.0GT/s Width x8 > ixl0: netmap queues/slots: TX 32/1024, RX 32/1024 > ixl1: mem > 0xdc00-0xdc7f,0xdd80-0xdd807fff irq 32 at device 0.1 on pci3 > ixl1: Using MSIX interrupts with 33 vectors > ixl1: f4.40 a1.4 n04.53 e80001dca > ixl1: Using defaults for TSO: 65518/35/2048 > ixl1: Ethernet address: 68:05:ca:32:0b:99 > ixl1: PCI Express Bus: Speed 8.0GT/s Width x8 > ixl1: netmap queues/slots: TX 32/1024, RX 32/1024 > ixl0: link state changed to UP > ixl1: link state changed to UP > > I have two identical machines connected with patch cables (no switch). iperf > performance is bad: > > # iperf -c 10.0.1.2 > > Client connecting to 10.0.1.2, TCP port 5001 > TCP window size: 32.5 KByte (default) > > [ 3] local 10.0.1.1 port 19238 connected with 10.0.1.2 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 3.91 GBytes 3.36 Gbits/sec > > As is flood ping latency: > > # sudo ping -f 10.0.1.2 > PING 10.0.1.2 (10.0.1.2): 56 data bytes > .^C > --- 10.0.1.2 ping statistics --- > 41927 packets transmitted, 41926 packets received, 0.0% packet loss > round-trip min/avg/max/stddev = 0.084/0.116/0.145/0.002 ms > > Any ideas on what's going on here? Testing 10G ix interfaces between the same > two machines results in 9.39 Gbits/sec and flood ping latencies of 17 usec. > > Thanks, > Lars > > PS: Full dmesg attached. > > Copyright (c) 1992-2015 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 11.0-CURRENT #2 483de3c(muclab)-dirty: Mon Oct 19 11:01:16 CEST 2015 > > el...@laurel.muccbc.hq.netapp.com:/usr/home/elars/obj/usr/home/elars/src/sys/MUCLAB > amd64 > FreeBSD clang version 3.7.0 (tags/RELEASE_370/final 246257) 20150906 > VT(vga): resolution 640x480 > CPU: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz (2000.05-MHz K8-class CPU) > Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7 > > Features=0xbfebfbff > > Features2=0x1fbee3ff > AMD Features=0x2c100800 > AMD Features2=0x1 > XSAVE Features=0x1 > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID > TSC: P-state invariant, performance statistics > real memory = 137438953472 (131072 MB) > avail memory = 133484290048 (127300 MB) > Event timer "LAPIC" quality 600 > ACPI APIC Table: < > > FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs > FreeBSD/SMP: 2 package(s) x 8 core(s) x 2 SMT threads > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > cpu2 (AP): APIC ID: 2 > cpu3 (AP): APIC ID: 3 > cpu4 (AP): APIC ID: 4 > cpu5 (AP): APIC ID: 5 > cpu6 (AP): APIC ID: 6 > cpu7 (AP): APIC ID: 7 > cpu8 (AP): APIC ID: 8 > cpu9 (AP): APIC ID: 9 > cpu10 (AP): APIC ID: 10 > cpu11 (AP): APIC ID: 11 > cpu12 (AP): APIC ID: 12 > cpu13 (AP): APIC ID: 13 > cpu14 (AP): APIC ID: 14 > cpu15 (AP): APIC ID: 15 > cpu16 (AP): APIC ID: 32 > cpu17 (AP): APIC ID: 33 > cpu18 (AP): APIC ID: 34 > cpu19 (AP): APIC ID: 35 > cpu20 (AP): APIC ID: 36 > cpu21 (AP): APIC ID: 37 > cpu22 (AP): APIC ID: 38 > cpu23 (AP): APIC ID: 39 > cpu24 (AP): APIC ID: 40 > cpu25 (AP): APIC ID: 41 > cpu26 (AP): APIC ID: 42 > cpu27 (AP): APIC ID: 43 > cpu28 (AP): APIC ID: 44 > cpu29 (AP): APIC ID: 45 > cpu30 (AP): APIC ID: 46 > cpu31 (AP): A
Re: ixl 40G bad performance?
Hi, On 2015-10-19, at 16:20, Luigi Rizzo wrote: > > i would look at the following: > - c states and clock speed - make sure you never go below C1, > and fix the clock speed to max. > Sure these parameters also affect the 10G card, but there > may be strange interaction that trigger the power saving > modes in different ways I already have powerd_flags="-a max -b max -n max" in rc.conf, which I hope should be enough. > - interrupt moderation (may affect ping latency, > do not remember how it is set in ixl but probably a sysctl ixl(4) describes two sysctls that sound like they control AIM, and they default to off: hw.ixl.dynamic_tx_itr: 0 hw.ixl.dynamic_rx_itr: 0 > - number of queues (32 is a lot i wouldn't use more than 4-8), > may affect cpu-socket affinity With hw.ixl.max_queues=4 in loader.conf, performance is still unchanged. > - tso and flow director - i have seen bad effects of > accelerations so i would run the iperf test with > of these features disabled on both sides, and then enable > them one at a time No change with "ifconfig -tso4 -tso6 -rxcsum -txcsum -lro". How do I turn off flow director? > - queue sizes - the driver seems to use 1024 slots which is > about 1.5 MB queued, which in turn means you have 300us > (and possibly half of that) to drain the queue at 40Gbit/s. > 150-300us may seem an eternity, but if a couple of cores fall > into c7 your budget is gone and the loss will trigger a > retransmission and window halving etc. Also no change with "hw.ixl.ringsz=256" in loader.conf. This is really weird. Lars signature.asc Description: Message signed with OpenPGP using GPGMail
Re: ixl 40G bad performance?
On Monday, October 19, 2015, Eggert, Lars wrote: > Hi, > > On 2015-10-19, at 16:20, Luigi Rizzo > > wrote: > > > > i would look at the following: > > - c states and clock speed - make sure you never go below C1, > > and fix the clock speed to max. > > Sure these parameters also affect the 10G card, but there > > may be strange interaction that trigger the power saving > > modes in different ways > > I already have powerd_flags="-a max -b max -n max" in rc.conf, which I > hope should be enough. I suspect it might not touch the c states, but better check. The safest is disable them in the bios. > > > - interrupt moderation (may affect ping latency, > > do not remember how it is set in ixl but probably a sysctl > > ixl(4) describes two sysctls that sound like they control AIM, and they > default to off: > > hw.ixl.dynamic_tx_itr: 0 > hw.ixl.dynamic_rx_itr: 0 > > There must be some other control for the actual (fixed, not dynamic) moderation. > > - number of queues (32 is a lot i wouldn't use more than 4-8), > > may affect cpu-socket affinity > > With hw.ixl.max_queues=4 in loader.conf, performance is still unchanged. > > > - tso and flow director - i have seen bad effects of > > accelerations so i would run the iperf test with > > of these features disabled on both sides, and then enable > > them one at a time > > No change with "ifconfig -tso4 -tso6 -rxcsum -txcsum -lro". > > How do I turn off flow director? I am not sure if it is enabled I'm FreeBSD. It is in linux and almost halves the pkt rate with netmap (from 35 down to 19mpps). Maybe it is not too bad for bulk TCP. > > > - queue sizes - the driver seems to use 1024 slots which is > > about 1.5 MB queued, which in turn means you have 300us > > (and possibly half of that) to drain the queue at 40Gbit/s. > > 150-300us may seem an eternity, but if a couple of cores fall > > into c7 your budget is gone and the loss will trigger a > > retransmission and window halving etc. > > Also no change with "hw.ixl.ringsz=256" in loader.conf. Any better success with 2048 slots? 3.5 gbit is what I used to see on the ixgbe with tso disabled, probably hitting a CPU bound. Cheers Luigi > This is really weird. > > Lars > -- -+--- Prof. Luigi RIZZO, ri...@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/. Universita` di Pisa TEL +39-050-2217533 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -+--- ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ixl 40G bad performance?
Hi, in order to eliminate network or hardware weirdness, I've rerun the test with Linux 4.3rc6, where I get 13.1 Gbits/sec throughput and 52 usec flood ping latency. Not great either, but in line with earlier experiments with Mellanox NICs and an untuned Linux system. On 2015-10-19, at 17:11, Luigi Rizzo wrote: > I suspect it might not touch the c states, but better check. The safest is > disable them in the bios. I'll try that. >> hw.ixl.dynamic_tx_itr: 0 >> hw.ixl.dynamic_rx_itr: 0 >> >> > There must be some other control for the actual (fixed, not dynamic) > moderation. The only other sysctls in ixl(4) that look relevant are: hw.ixl.rx_itr The RX interrupt rate value, set to 8K by default. hw.ixl.tx_itr The TX interrupt rate value, set to 4K by default. I'll play with those. >> Also no change with "hw.ixl.ringsz=256" in loader.conf. > > Any better success with 2048 slots? > 3.5 gbit is what I used to see on the ixgbe with tso disabled, probably > hitting a CPU bound. Will try. Thanks! Lars signature.asc Description: Message signed with OpenPGP using GPGMail
Re: ixl 40G bad performance?
On Mon, Oct 19, 2015 at 8:34 AM, Eggert, Lars wrote: > Hi, > > in order to eliminate network or hardware weirdness, I've rerun the test with > Linux 4.3rc6, where I get 13.1 Gbits/sec throughput and 52 usec flood ping > latency. Not great either, but in line with earlier experiments with Mellanox > NICs and an untuned Linux system. > ... >> There must be some other control for the actual (fixed, not dynamic) >> moderation. > > The only other sysctls in ixl(4) that look relevant are: > > hw.ixl.rx_itr > The RX interrupt rate value, set to 8K by default. > > hw.ixl.tx_itr > The TX interrupt rate value, set to 4K by default. > yes those. raise to 20-50k and see what you get in terms of ping latency. Note that 4k on tx means you get to reclaim buffers in the tx queue (unless it is done opportunistically) every 250us which is dangerously close to the 300us capacity of the queue itself. cheers luigi > I'll play with those. > >>> Also no change with "hw.ixl.ringsz=256" in loader.conf. >> >> Any better success with 2048 slots? >> 3.5 gbit is what I used to see on the ixgbe with tso disabled, probably >> hitting a CPU bound. > > Will try. > > Thanks! > > Lars -- -+--- Prof. Luigi RIZZO, ri...@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/. Universita` di Pisa TEL +39-050-2217533 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -+--- ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ixl 40G bad performance?
On 10/19/15 at 08:11P, Luigi Rizzo wrote: > On Monday, October 19, 2015, Eggert, Lars wrote: > > > > > How do I turn off flow director? > > > I am not sure if it is enabled I'm FreeBSD. It is in linux and almost > halves the pkt rate with netmap (from 35 down to 19mpps). > Maybe it is not too bad for bulk TCP. > Flow director support is incomplete on FreeBSD and that's why it is disabled by default. Cheers, Hiren pgp9F3s1LYB3X.pgp Description: PGP signature
Re: Some MSI are not routed correctly
On Thursday, October 08, 2015 07:33:27 AM Maxim Sobolev wrote: > Hi John & others, > > We've came across a weird MSI routing issue on one of our newest dual > E5-2690v3 (haswell) Supermicro X10DRL-i boxes running latest 10.2-p4. It is > fitted with dual port Intel I350 card, in addition to the built-in I210 > chip that is not used. The hw.igb.num_queues is set to 4, and the driver > reports binding to the CPUs 0-3 for the first port and CPUs 4-7 for the > second, however when verified with top -P under the load, interrupts are > only delivered to the CPUs 0-3, no interrupt time is recorded on the CPUs > 4-7. systat -vm shows that all 8 queues are firing interrupts, so my guess > that for whatever reason bus_bind_intr() is not doing what's expected to do > for half of those interrupts. > > What's interesting is that on a similar box (same chassis/mobo/cpu) but > equipped with the quad-port X540-AT2 10Gig card, interrupts are routed > properly. The latter is running with hw.ix.num_queues="3". > > pcib2: port 0xcf8-0xcff on acpi0 > pci0: on pcib2 > pcib3: irq 26 at device 1.0 on pci0 > pci1: on pcib3 > igb0: mem > 0xc720-0xc72f,0xc7304000-0xc7307fff irq 26 at device 0.0 on pci1 > igb0: Using MSIX interrupts with 5 vectors > igb0: Ethernet address: a0:36:9f:76:af:20 > igb0: Bound queue 0 to cpu0 > igb0: Bound queue 1 to cpu1 > igb0: Bound queue 2 to cpu2 > igb0: Bound queue 3 to cpu3 > igb0: netmap queues/slots: TX 4/4096, RX 4/4096 > igb1: mem > 0xc710-0xc71f,0xc730-0xc7303fff irq 28 at device 0.1 on pci1 > igb1: Using MSIX interrupts with 5 vectors > igb1: Ethernet address: a0:36:9f:76:af:21 > igb1: Bound queue 0 to cpu4 > igb1: Bound queue 1 to cpu5 > igb1: Bound queue 2 to cpu6 > igb1: Bound queue 3 to cpu7 > igb1: netmap queues/slots: TX 4/4096, RX 4/4096 > > pcib2: port 0xcf8-0xcff on acpi0 > pci0: on pcib2 > pcib3: irq 26 at device 1.0 on pci0 > pci1: on pcib3 > pcib4: irq 32 at device 2.0 on pci0 > pci2: on pcib4 > pcib5: irq 40 at device 3.0 on pci0 > pci3: on pcib5 > ix0: port > 0x6020-0x603f mem 0xc7c0-0xc7df,0xc7e04000-0xc7e07fff irq 40 at > device 0.0 on pci3 > ix0: Using MSIX interrupts with 4 vectors > ix0: Bound queue 0 to cpu 0 > ix0: Bound queue 1 to cpu 1 > ix0: Bound queue 2 to cpu 2 > ix0: Ethernet address: 0c:c4:7a:5e:be:64 > ix0: PCI Express Bus: Speed 5.0GT/s Width x8 > ix0: netmap queues/slots: TX 3/4096, RX 3/4096 > ix1: port > 0x6000-0x601f mem 0xc7a0-0xc7bf,0xc7e0-0xc7e03fff irq 44 at > device 0.1 on pci3 > ix1: Using MSIX interrupts with 4 vectors > ix1: Bound queue 0 to cpu 3 > ix1: Bound queue 1 to cpu 4 > ix1: Bound queue 2 to cpu 5 > ix1: Ethernet address: 0c:c4:7a:5e:be:65 > ix1: PCI Express Bus: Speed 5.0GT/s Width x8 > ix1: netmap queues/slots: TX 3/4096, RX 3/4096 > > Some extra debug is here: > > http://sobomax.sippysoft.com/haswell_bug/bad.dmesg > http://sobomax.sippysoft.com/haswell_bug/lstopo_bad.png > http://sobomax.sippysoft.com/haswell_bug/systat_vm_bad.png > http://sobomax.sippysoft.com/haswell_bug/top_P_bad.png > > http://sobomax.sippysoft.com/haswell_bug/good.dmesg > http://sobomax.sippysoft.com/haswell_bug/lstopo_good.png > http://sobomax.sippysoft.com/haswell_bug/systat_vm_good.png > http://sobomax.sippysoft.com/haswell_bug/top_P_good.png > > Any ideas on how to debug that further are welcome. The box in the > production, but we can remove traffic during off-peak to run some > test/debug code on. Can you get procstat -S output for the interrupt threads? (Usually interrupt threads are in pid 12, so 'procstat -S 12' would suffice.) -- John Baldwin ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: MPTCP for FreeBSD repository on BitBucket/v0.51 update
Nigel... seriously... /*-^M * Copyright (c) 2012-2015^M * Swinburne University of Technology, Melbourne, Australia.^M * All rights reserved.^M *^M * This software was developed at the Centre for Advanced Internet^M * Architectures, Swinburne University of Technology, by Nigel Williams and^M * Lawrence Stewart, made possible in part by a gift from the FreeBSD^M * Foundation and The Cisco University Research Program Fund, a corporate^M * advised fund of Silicon Valley Community Foundation.^M *^M * Redistribution and use in source and binary forms, with or without^M * modification, are permitted provided that the following conditions^M * are met:^M * 1. Redistributions of source code must retain the above copyright^M *notice, this list of conditions and the following disclaimer.^M * 2. Redistributions in binary form must reproduce the above copyright^M *notice, this list of conditions and the following disclaimer in the^M *documentation and/or other materials provided with the distribution.^M *^M * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND^M * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE^M * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE^M * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE^M * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL^M * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS^M * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)^M * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT^M * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY^M * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF^M * SUCH DAMAGE.^M */^M ^M /*^M * mptcp.h^M *^M * Created on: 15/05/2012^M * Author: nwilliams^M */^M ^M #ifndef MPTCP_H_^M #define MPTCP_H_^M ^M ^M #include ^M ^M #define MPTCP_64BIT_KEY 8^M ^M typedef u_int64_t mptcp_seq;^M ^M /* MPTCP subtypes */^M #define MPTCP_SUBTYPE_MP_CAPABLE0^M #define MPTCP_SUBLEN_MP_CAPABLE_SYN 12^M #define MPTCP_SUBLEN_MP_CAPABLE_ACK 20^M ^M #define MPTCP_SUBTYPE_MP_JOIN 1^M #define MPTCP_SUBLEN_MP_JOIN_SYN12^M #define MPTCP_SUBLEN_MP_JOIN_SYNACK 16 // should be 16, but run out of option space^M #define MPTCP_SUBLEN_MP_JOIN_ACK24 // should be 24, but run out of option space^M ^M #define MPTCP_SUBTYPE_DSS 2^M #define MPTCP_SUBLEN_DSS_DATA_ACK XX^M #define MPTCP_SUBLEN_DSS_DATA_DSN XX^M ^M #define MPTCP_SUBTYPE_ADD_ADDR 3^M #define MPTCP_SUBLEN_ADD_ADDRV4 8^M #define MPTCP_SUBLEN_ADD_ADDRV6 20^M ^M #define MPTCP_SUBTYPE_REMOVE_ADDR 4^M #define MPTCP_SUBLEN_REMOVE_ADDR4^M ^M #define MPTCP_SUBTYPE_MP_PRIO 5^M ^M #define MPTCP_SUBTYPE_MP_FAIL 6^M #define MPTCP_SUBTYPELEN_MP_FAIL12^M ^M #define MPTCP_SUBTYPE_MP_FASTCLOSE 7^M #define MPTCP_SUBTYPELEN_MP_FASTCLOSE 12^M ^M #define MAX_MP_OPLEN28^M ^M /* mptcp errors */^M ^M #define EMAXSUBFLOWSREACHED 01^M #define ENOMPCB 02^M #define ENOTCPCB 03^M ^M /* mptcp funcs */^M ^M ^M #define MPTCP_SA_NAME_MAX 16 /* max scheduler discipline name length */^M ^M #endif /* MPTCP_H_ */^M On Mon, Oct 19, 2015 at 1:42 PM, Nigel Williams wrote: > Hi, > > The MPTCP code is now available as a mercurial repository: > - Repository: https://bitbucket.org/nw-swin/caia-mptcp-freebsd > - Wiki: https://bitbucket.org/nw-swin/caia-mptcp-freebsd/wiki/ > > For those interested in trying the implementation/looking at the code, > this should hopefully make the process a little easier (and save having to > patch in updates). It should also make it possible to contribute code for > those wishing to do so. > > Some details: > - Has been branched off 'freebsd-head' at 'http://hg-beta.freebsd.org/base', > and will be merged on a weekly basis. > - I will be working off this repository so it will be up-to-date with > recent changes. > - In place of patch releases, release versions will now be tagged. > - I'll also start to populate the 'Issues' section so that there is a > better picture of current bugs/things TBD. > > The version has also been updated to v0.51. See: > - http://caia.swin.edu.au/newtcp/mptcp/tools.html > - OR https://bitbucket.org/nw-swin/caia-mptcp-freebsd/wiki/Home > > Functionally-wise this hasn't changed from the previous version, but has > been merged with a recent revision of head. > > cheers, > nigel > ___ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" > ___
Re: MPTCP for FreeBSD repository on BitBucket/v0.51 update
Hi, Fixed. It should just have been that file. cheers, nigel On 20/10/15 09:50, Outback Dingo wrote: Nigel... seriously... /*-^M * Copyright (c) 2012-2015^M * Swinburne University of Technology, Melbourne, Australia.^M * All rights reserved.^M *^M * This software was developed at the Centre for Advanced Internet^M * Architectures, Swinburne University of Technology, by Nigel Williams and^M * Lawrence Stewart, made possible in part by a gift from the FreeBSD^M * Foundation and The Cisco University Research Program Fund, a corporate^M * advised fund of Silicon Valley Community Foundation.^M *^M * Redistribution and use in source and binary forms, with or without^M * modification, are permitted provided that the following conditions^M * are met:^M * 1. Redistributions of source code must retain the above copyright^M *notice, this list of conditions and the following disclaimer.^M * 2. Redistributions in binary form must reproduce the above copyright^M *notice, this list of conditions and the following disclaimer in the^M *documentation and/or other materials provided with the distribution.^M *^M * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND^M * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE^M * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE^M * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE^M * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL^M * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS^M * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)^M * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT^M * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY^M * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF^M * SUCH DAMAGE.^M */^M ^M /*^M * mptcp.h^M *^M * Created on: 15/05/2012^M * Author: nwilliams^M */^M ^M #ifndef MPTCP_H_^M #define MPTCP_H_^M ^M ^M #include ^M ^M #define MPTCP_64BIT_KEY 8^M ^M typedef u_int64_t mptcp_seq;^M ^M /* MPTCP subtypes */^M #define MPTCP_SUBTYPE_MP_CAPABLE0^M #define MPTCP_SUBLEN_MP_CAPABLE_SYN 12^M #define MPTCP_SUBLEN_MP_CAPABLE_ACK 20^M ^M #define MPTCP_SUBTYPE_MP_JOIN 1^M #define MPTCP_SUBLEN_MP_JOIN_SYN12^M #define MPTCP_SUBLEN_MP_JOIN_SYNACK 16 // should be 16, but run out of option space^M #define MPTCP_SUBLEN_MP_JOIN_ACK24 // should be 24, but run out of option space^M ^M #define MPTCP_SUBTYPE_DSS 2^M #define MPTCP_SUBLEN_DSS_DATA_ACK XX^M #define MPTCP_SUBLEN_DSS_DATA_DSN XX^M ^M #define MPTCP_SUBTYPE_ADD_ADDR 3^M #define MPTCP_SUBLEN_ADD_ADDRV4 8^M #define MPTCP_SUBLEN_ADD_ADDRV6 20^M ^M #define MPTCP_SUBTYPE_REMOVE_ADDR 4^M #define MPTCP_SUBLEN_REMOVE_ADDR4^M ^M #define MPTCP_SUBTYPE_MP_PRIO 5^M ^M #define MPTCP_SUBTYPE_MP_FAIL 6^M #define MPTCP_SUBTYPELEN_MP_FAIL12^M ^M #define MPTCP_SUBTYPE_MP_FASTCLOSE 7^M #define MPTCP_SUBTYPELEN_MP_FASTCLOSE 12^M ^M #define MAX_MP_OPLEN28^M ^M /* mptcp errors */^M ^M #define EMAXSUBFLOWSREACHED 01^M #define ENOMPCB 02^M #define ENOTCPCB 03^M ^M /* mptcp funcs */^M ^M ^M #define MPTCP_SA_NAME_MAX 16 /* max scheduler discipline name length */^M ^M #endif /* MPTCP_H_ */^M On Mon, Oct 19, 2015 at 1:42 PM, Nigel Williams mailto:njwilli...@swin.edu.au>> wrote: Hi, The MPTCP code is now available as a mercurial repository: - Repository: https://bitbucket.org/nw-swin/caia-mptcp-freebsd - Wiki: https://bitbucket.org/nw-swin/caia-mptcp-freebsd/wiki/ For those interested in trying the implementation/looking at the code, this should hopefully make the process a little easier (and save having to patch in updates). It should also make it possible to contribute code for those wishing to do so. Some details: - Has been branched off 'freebsd-head' at 'http://hg-beta.freebsd.org/base', and will be merged on a weekly basis. - I will be working off this repository so it will be up-to-date with recent changes. - In place of patch releases, release versions will now be tagged. - I'll also start to populate the 'Issues' section so that there is a better picture of current bugs/things TBD. The version has also been updated to v0.51. See: - http://caia.swin.edu.au/newtcp/mptcp/tools.html - OR https://bitbucket.org/nw-swin/caia-mptcp-freebsd/wiki/Home Functionally-wise this hasn't changed from the previous version, but has been merged with a recent revision of head. cheers, nigel ___
Re: MPTCP for FreeBSD repository on BitBucket/v0.51 update
Hahaha was this written in notepad or did someone forget to turn off dos style line encodings. -- Jason Hellenthal JJH48-ARIN On Oct 19, 2015, at 17:50, Outback Dingo wrote: Nigel... seriously... /*-^M * Copyright (c) 2012-2015^M * Swinburne University of Technology, Melbourne, Australia.^M * All rights reserved.^M *^M * This software was developed at the Centre for Advanced Internet^M * Architectures, Swinburne University of Technology, by Nigel Williams and^M * Lawrence Stewart, made possible in part by a gift from the FreeBSD^M * Foundation and The Cisco University Research Program Fund, a corporate^M * advised fund of Silicon Valley Community Foundation.^M *^M * Redistribution and use in source and binary forms, with or without^M * modification, are permitted provided that the following conditions^M * are met:^M * 1. Redistributions of source code must retain the above copyright^M *notice, this list of conditions and the following disclaimer.^M * 2. Redistributions in binary form must reproduce the above copyright^M *notice, this list of conditions and the following disclaimer in the^M *documentation and/or other materials provided with the distribution.^M *^M * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND^M * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE^M * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE^M * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE^M * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL^M * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS^M * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)^M * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT^M * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY^M * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF^M * SUCH DAMAGE.^M */^M ^M /*^M * mptcp.h^M *^M * Created on: 15/05/2012^M * Author: nwilliams^M */^M ^M #ifndef MPTCP_H_^M #define MPTCP_H_^M ^M ^M #include ^M ^M #define MPTCP_64BIT_KEY 8^M ^M typedef u_int64_t mptcp_seq;^M ^M /* MPTCP subtypes */^M #define MPTCP_SUBTYPE_MP_CAPABLE0^M #define MPTCP_SUBLEN_MP_CAPABLE_SYN 12^M #define MPTCP_SUBLEN_MP_CAPABLE_ACK 20^M ^M #define MPTCP_SUBTYPE_MP_JOIN 1^M #define MPTCP_SUBLEN_MP_JOIN_SYN12^M #define MPTCP_SUBLEN_MP_JOIN_SYNACK 16 // should be 16, but run out of option space^M #define MPTCP_SUBLEN_MP_JOIN_ACK24 // should be 24, but run out of option space^M ^M #define MPTCP_SUBTYPE_DSS 2^M #define MPTCP_SUBLEN_DSS_DATA_ACK XX^M #define MPTCP_SUBLEN_DSS_DATA_DSN XX^M ^M #define MPTCP_SUBTYPE_ADD_ADDR 3^M #define MPTCP_SUBLEN_ADD_ADDRV4 8^M #define MPTCP_SUBLEN_ADD_ADDRV6 20^M ^M #define MPTCP_SUBTYPE_REMOVE_ADDR 4^M #define MPTCP_SUBLEN_REMOVE_ADDR4^M ^M #define MPTCP_SUBTYPE_MP_PRIO 5^M ^M #define MPTCP_SUBTYPE_MP_FAIL 6^M #define MPTCP_SUBTYPELEN_MP_FAIL12^M ^M #define MPTCP_SUBTYPE_MP_FASTCLOSE 7^M #define MPTCP_SUBTYPELEN_MP_FASTCLOSE 12^M ^M #define MAX_MP_OPLEN28^M ^M /* mptcp errors */^M ^M #define EMAXSUBFLOWSREACHED 01^M #define ENOMPCB 02^M #define ENOTCPCB 03^M ^M /* mptcp funcs */^M ^M ^M #define MPTCP_SA_NAME_MAX 16 /* max scheduler discipline name length */^M ^M #endif /* MPTCP_H_ */^M On Mon, Oct 19, 2015 at 1:42 PM, Nigel Williams wrote: > Hi, > > The MPTCP code is now available as a mercurial repository: > - Repository: https://bitbucket.org/nw-swin/caia-mptcp-freebsd > - Wiki: https://bitbucket.org/nw-swin/caia-mptcp-freebsd/wiki/ > > For those interested in trying the implementation/looking at the code, > this should hopefully make the process a little easier (and save having to > patch in updates). It should also make it possible to contribute code for > those wishing to do so. > > Some details: > - Has been branched off 'freebsd-head' at 'http://hg-beta.freebsd.org/base', > and will be merged on a weekly basis. > - I will be working off this repository so it will be up-to-date with > recent changes. > - In place of patch releases, release versions will now be tagged. > - I'll also start to populate the 'Issues' section so that there is a > better picture of current bugs/things TBD. > > The version has also been updated to v0.51. See: > - http://caia.swin.edu.au/newtcp/mptcp/tools.html > - OR https://bitbucket.org/nw-swin/caia-mptcp-freebsd/wiki/Home > > Functionally-wise this hasn't changed from the previous version, but has > been merged with a recent revision of head. > > cheers, > nigel > ___ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org
Re: MPTCP for FreeBSD repository on BitBucket/v0.51 update
Outback Dingo and Jason, On 20/10/2015 10:43, Jason Hellenthal wrote: > Hahaha was this written in notepad or did someone forget to turn off dos > style line encodings. > >> On 20/10/2015 09:50, Outback Dingo wrote: >> Nigel... >> >> seriously... >> Publicly ridiculing someone for such a minor issue when a friendly private email or even better a pull request on Bitbucket would have been appropriate makes you both look pathetic. Shut up and code, or failing that shut up and let other people code, and please keep disparaging comments off the list in future. Lawrence ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: MPTCP for FreeBSD repository on BitBucket/v0.51 update
On 10/20/15 at 12:21P, Lawrence Stewart wrote: > Outback Dingo and Jason, > > On 20/10/2015 10:43, Jason Hellenthal wrote: > > Hahaha was this written in notepad or did someone forget to turn off dos > > style line encodings. > > > >> On 20/10/2015 09:50, Outback Dingo wrote: > >> Nigel... > >> > >> seriously... > >> > > Publicly ridiculing someone for such a minor issue when a friendly > private email or even better a pull request on Bitbucket would have been > appropriate makes you both look pathetic. Shut up and code, or failing > that shut up and let other people code, and please keep disparaging > comments off the list in future. So true. How could you guys pick this "nit" over the huge overhaul Nigel is doing in tcp stack to bring this useful feature into FreeBSD? Please be constructive in your responses. We want to encourage and support people who work on FreeBSD and not make fun of them. Cheers, Hiren pgpGDtZdRAcqh.pgp Description: PGP signature
Re: ixl 40G bad performance?
On Mon, Oct 19, 2015 at 8:11 AM, Luigi Rizzo wrote: > On Monday, October 19, 2015, Eggert, Lars wrote: > > > Hi, > > > > On 2015-10-19, at 16:20, Luigi Rizzo > > > wrote: > > > > > > i would look at the following: > > > - c states and clock speed - make sure you never go below C1, > > > and fix the clock speed to max. > > > Sure these parameters also affect the 10G card, but there > > > may be strange interaction that trigger the power saving > > > modes in different ways > > > > I already have powerd_flags="-a max -b max -n max" in rc.conf, which I > > hope should be enough. > > > I suspect it might not touch the c states, but better check. The safest is > disable them in the bios. > To disable C-States: sysctl dev.cpu.0.cx_lowest=C1 -- Kevin Oberman, Part time kid herder and retired Network Engineer ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
