Re: Netmap Checksum Offloading
> On 16 Jun 2016, at 08:40, Dominik Schoeffmann wrote: > > Is the checksum offloading patch for the igb(4) driver available online? > (I could not find it) > I would really like to take a look at it, espacially the context > descriptor part of it. I would also be interested in it... For looking at SCTP checksum offloading. Best regards Michael > > Best regards, > Dominik > > On 16.06.2016 02:04, Jim Thompson wrote: >> >> Luiz Otavio O Souza (loos@) developed these for igb(4) and, by extension, >> em(4) for use in netmap-fwd. >> >> He’s just gone back to Brazil with 82599 ixgb(4) hardware. I’m sure he’ll >> develop similar patches for ixgb(4) in the near future. >> >> Chelsio is also “on the list”, but I figured I’d speak to np@ about it >> first. ;-) >> We might do ixl(4) as well. >> >> Before Luiz retired to Brazil, we discussed upstreaming these to FreeBSD. >> We’re committed to make it happen, but I doubt they make 11. >> >> Jim >> >>> On Jun 15, 2016, at 6:50 PM, Navdeep Parhar wrote: >>> >>> On 06/15/2016 16:15, Andrey Yakovlev wrote: ive heard on bsdcan this year that some patches exist to add hwcsum offloading to netmap, hope to see it chelsio at least >>> >>> cxgbe/cxl is a bit sneaky and will let you override netmap (on tx only). >>> The ncxl interfaces declare themselves capable of checksumming but all >>> such capabilities are disabled by default. Just enable txcsum on the >>> interface and the hardware will do checksum insertion on tx. No way to >>> solve the rx part entirely within the driver -- netmap has to be willing >>> to accept checksum related flags from the driver. >>> >>> Regards, >>> Navdeep >>> -- ./andy 14.06.2016, 12:15, "Dominik Schoeffmann" : > Dear Netmap Developers, > > during the course of my bachelor's thesis, I modified a packet generator > called MoonGen [1] in order to utilize netmap. > One key component was to flexibly offload checksums for different kinds > of packets (IPv4, UDP, TCP). > The ixgbe netmap patch was modified [2] in order to construct context > descriptors and suitable data descriptors. This is implemented in less > than 250 LoC (including pseudo-header calculations). > The man page states, that checksum offloading is available via ethtool, > although a solution inside the netmap API might be a cleaner way for > applications to actually use these features. > Attached is a graph showing the performance implication of using > offloading in the current implementation. > As can be seen, offloading has only a minor impact. > When regarding this data (and comparing it to other frameworks), please > keep in mind, that internally a lot of per-packet effort is needed due > to the software architecture of the packet generator. > > The question being: > Would it not make sense to include checksum offloading inside of netmap > in order to accomodate applications operating on layer 3 and above? > As these programs need to calculate the checksums in software, it would > be just as fast to move these calculations to the kernel for NICs > without checksum offloading support (and the kernel would act as a > library). > The problem which currently is imposed by the fact, that netmap exports > the complete ring, is that context descriptors disrupt the data > descriptors, which is unpleasant for the application. > But you may find the data interesting nevertheless. > > Best Regards, > Dominik Schoeffmann > > [1] https://github.com/dschoeffm/MoonGen/tree/netmap > [2] https://github.com/dschoeffm/netmap/tree/mg-chksum-offloading ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" >>> >>> ___ >>> freebsd-net@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" >> > ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
[Bug 200420] [igb] igb0: Watchdog timeout -- resetting
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200420 arca...@ivanovy.net changed: What|Removed |Added CC||arca...@ivanovy.net --- Comment #1 from arca...@ivanovy.net --- I can report the same for 10.3-RELEASE-p4 and stock kernel. # cat /boot/loader.conf: kern.geom.label.gptid.enable="0" kern.ipc.nmbclusters="100" hw.pci.enable_msi="1" hw.pci.enable_msix="0" zfs_load="YES" aesni_load="YES" ichsmb_load="YES" ipmi_load="YES" beastie_disable="YES" autoboot_delay="3" # pciconf -lcv igb0@pci0:2:0:0:class=0x02 card=0x153315d9 chip=0x15338086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = 'I210 Gigabit Network Connection' class = network subclass = ethernet cap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit, vector masks enabled with 1 message cap 11[70] = MSI-X supports 5 messages Table in map 0x1c[0x0], PBA in map 0x1c[0x2000] cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR RO NS link x1(x1) speed 2.5(2.5) ASPM L1(L0s/L1) ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected ecap 0003[140] = Serial 1 002590x2 ecap 0017[1a0] = TPH Requester 1 igb1@pci0:5:0:0:class=0x02 card=0x153315d9 chip=0x15338086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = 'I210 Gigabit Network Connection' class = network subclass = ethernet cap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit, vector masks enabled with 1 message cap 11[70] = MSI-X supports 5 messages Table in map 0x1c[0x0], PBA in map 0x1c[0x2000] cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR RO NS link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1) ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected ecap 0003[140] = Serial 1 002590x3 ecap 0017[1a0] = TPH Requester 1 This configuration is fairly stable (1h so far, no watchdog timeouts on igb1). Disabling MSI altogether causes igb1 that doesn't pass any traffic and doesn't even come up. Disabling MSI-X seems to help. This is a SuperMicro X11SBA-F board (http://www.supermicro.com/products/motherboard/X11/X11SBA-F.cfm) with BIOS 1.0b (latest). -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
[Bug 200420] [igb] igb0: Watchdog timeout -- resetting
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200420 --- Comment #2 from arca...@ivanovy.net --- Spoke too soon, disabling MSI-X seems to help only marginally: Jun 16 13:29:49 fw1 kernel: igb1: Watchdog timeout -- resetting Jun 16 13:29:49 fw1 kernel: igb1: Queue(846295657) tdh = -1249464976, hw tdt = 589458993 Jun 16 13:29:49 fw1 kernel: igb1: TX(846295657) desc avail = 0,Next TX to Clean = 0 Jun 16 13:29:49 fw1 kernel: igb1: link state changed to DOWN Jun 16 13:29:53 fw1 kernel: igb1: link state changed to UP Jun 16 13:29:53 fw1 devd: Executing '/etc/rc.d/dhclient quietstart igb1' Jun 16 13:34:26 fw1 kernel: igb1: Watchdog timeout -- resetting Jun 16 13:34:26 fw1 kernel: igb1: Queue(846295657) tdh = -1249464976, hw tdt = 589458993 Jun 16 13:34:26 fw1 kernel: igb1: TX(846295657) desc avail = 0,Next TX to Clean = 0 Jun 16 13:34:26 fw1 kernel: igb1: link state changed to DOWN Jun 16 13:34:31 fw1 kernel: igb1: link state changed to UP Jun 16 13:34:31 fw1 devd: Executing '/etc/rc.d/dhclient quietstart igb1' -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
[Bug 200420] [igb] igb0: Watchdog timeout -- resetting
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200420 --- Comment #3 from arca...@ivanovy.net --- Possibly related to #200221 -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
[Bug 200420] [igb] igb0: Watchdog timeout -- resetting
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200420 --- Comment #4 from arca...@ivanovy.net --- igb0: flags=8843 metric 0 mtu 1500 options=bb igb1: flags=8843 metric 0 mtu 1500 options=bb I have TSO, VLAN_HWTSO and LRO disabled. -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
[Bug 200420] [igb] igb0: Watchdog timeout -- resetting
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200420 --- Comment #5 from arca...@ivanovy.net --- some PCI-E errors: pcib4@pci0:3:0:0: class=0x060400 card=0x chip=0x260812d8 rev=0x00 hdr=0x01 vendor = 'Pericom Semiconductor' class = bridge subclass = PCI-PCI PCI-e errors = Correctable Error Detected Unsupported Request Detected Corrected = Receiver Error Bad TLP Bad DLLP REPLAY_NUM Rollover Replay Timer Timeout Advisory Non-Fatal Error igb0@pci0:2:0:0:class=0x02 card=0x153315d9 chip=0x15338086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = 'I210 Gigabit Network Connection' class = network subclass = ethernet Corrected = Advisory Non-Fatal Error igb1@pci0:5:0:0:class=0x02 card=0x153315d9 chip=0x15338086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = 'I210 Gigabit Network Connection' class = network subclass = ethernet PCI-e errors = Correctable Error Detected Corrected = Advisory Non-Fatal Error -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
[Bug 200420] [igb] igb0: Watchdog timeout -- resetting
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200420 --- Comment #6 from arca...@ivanovy.net --- # netstat -m 2048/3772/5820 mbufs in use (current/cache/total) 2046/2514/4560/100 mbuf clusters in use (current/cache/total/max) 2046/2508 mbuf+clusters out of packet secondary zone in use (current/cache) 0/4/4/250101 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/74104 9k jumbo clusters in use (current/cache/total/max) 0/0/0/41683 16k jumbo clusters in use (current/cache/total/max) 4604K/5987K/10591K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Is netmap jumbo frames broken in STABLE?
About week ago I've patched if_ix.c, just returned back code fragment of max_frame_size determination from 2.8.3 version of ixgbe driver: /* ** Determine the correct mbuf pool ** for doing jumbo frames */ if (adapter->max_frame_size <= 2048) adapter->rx_mbuf_sz = MCLBYTES; else if (adapter->max_frame_size <= 4096) adapter->rx_mbuf_sz = MJUMPAGESIZE; else if (adapter->max_frame_size <= 9216) adapter->rx_mbuf_sz = MJUM9BYTES; else adapter->rx_mbuf_sz = MJUM16BYTES; After week of heavy testing everything is ok. This solution is acceptable for me for now, because I use eight interfaces in netmap mode only. In future I want to make support something like dev.ix.N.jumbo_mbuf_enable variable, since one of ix interfaces will be used for generic kernel stack with jumbo frame support. Doing with fragmented packet in netmap ring is also possible way, but need more user cpu cycles for processing, I guess. In any case I can test it when this patch will be merged for netmap. -- Andrew 2016-06-08 14:28 GMT+03:00 : > > Support for fragmented packets with ixgbe was recently added on the linux version of Netmap : > > https://github.com/luigirizzo/netmap/commit/fc1e77560a8a8ea93cc3594de5fae94334debcd3 > > I think the change for freebsd would be quite the same looking at https://github.com/freebsd/freebsd/blob/master/sys/dev/netmap/ixgbe_netmap.h#L396 > > After that, your userspace application simply have to check for the NS_MOREFRAG flag in the receive ring, and if it's set he knows the end of the packet will follow in the next buf. > > Tom ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: lagg(4): LOR, deadlock and panic
On Tue, 2016-06-14 at 09:26 -0600, Alan Somers wrote: > > I don't know the best answer either. But while you're in there, are > you interested in fixing any other lagg panics too? I've written > some > ATF torture tests for lagg, but I haven't checked them into head yet > because most of them quickly panic. > > -Alan We run into if_lagg and if_vlan panics at $WORK all the time in our automation. I've fixed the if_vlan panics and I'm hoping to update this review: https://reviews.freebsd.org/D5825 soon with something that accomodates drivers sleeping in the vlan_*config event handlers (which involves having a global rmlock _and_ sx in if_vlan). I was planning on doing a similar audit/fixing of if_lagg soon when I get the chance. Matt Joras ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
panic with tcp timers
Hi! At Netflix we are observing a race in TCP timers with head. The problem is a regression, that doesn't happen on stable/10. The panic usually happens after several hours at 55 Gbit/s of traffic. What happens is that tcp_timer_keep finds t_tcpcb being NULL. Some coredumps have tcpcb already initialized, with non-NULL t_tcpcb and in TCPS_ESTABLISHED state. Which means that other CPU was working on the tcpcb while the faulted one was working on the panic. So, this all looks like a use after free, which conflicts with new allocation. Comparing stable/10 and head, I see two changes that could affect that: - callout_async_drain - switch to READ lock for inp info in tcp timers That's why you are in To, Julien and Hans :) We continue investigating, and I will keep you updated. However, any help is welcome. I can share cores. -- Totus tuus, Glebius. ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
