Re: What is best TCP throughput benchmarking tool?

2018-10-22 Thread Jason Wolfe
Lev Serebryakov wrote on Sat, Oct 20, 2018 at 05:54:38PM +0300: > Hello Eugene, > > Saturday, October 20, 2018, 2:23:29 AM, you wrote: > > > You do not need to micro-control this. The wrk provides you with nice stats > > plus you have counters of "systat -ifstat 1" during long test. > > >> All

Re: Multiple cores/race conditions in IPv6 RA

2015-12-09 Thread Jason Wolfe
Word around town is Kubilay Kocak leaked this on Wed, Dec 09, 2015 at 09:02:48PM +1100: > On 9/12/2015 7:58 PM, Andrey V. Elsukov wrote: > > On 09.12.15 11:54, Kubilay Kocak wrote: > >>> some time ago Mark Johnston has published there the patch related to > >>> this problem: > >>> https://lists.fr

Re: Multiple cores/race conditions in IPv6 RA

2015-12-08 Thread Jason Wolfe
I believe I've found a way to reproduce this, simply an rtsol against 2 interfaces in isolated loops, which elicits a broadcasted RA. I am able to deadlock the system fairly quickly, which finally results in a core after __rw_wunlock_hard steps in. We have seen this deadlock in one other case

Re: Unbalanced LACP link

2015-03-17 Thread Jason Wolfe
On Mon, Mar 16, 2015 at 2:43 AM, Hans Petter Selasky wrote: > On 03/16/15 10:37, Vitalii Duk wrote: >> >> I've changed use_flowid to 0 and it helped! But isn't it setting >> significant? In a description it says "Shift flowid bits to prevent >> multiqueue collisions". > > > Hi, > > Maybe your ethe

Re: ixgbe(4) spin lock held too long

2015-01-17 Thread Jason Wolfe
On Mon, Dec 15, 2014 at 9:23 AM, John Baldwin wrote: > On Wednesday, December 10, 2014 12:47:02 PM Jason Wolfe wrote: >> John, >> >> So apparently the concurrent timer scheduling was not fixed, though it >> does seem rarer. We had about 2 weeks of stability, then last

Re: ixgbe(4) spin lock held too long

2014-10-23 Thread Jason Wolfe
On Sat, Oct 18, 2014 at 4:42 AM, John Baldwin wrote: > On Friday, October 17, 2014 11:32:13 PM Jason Wolfe wrote: >> Producing 10G of random traffic against a server with this assertion >> added took about 2 hours to panic, so if it turns out we need anything >> further it s

Re: ixgbe(4) spin lock held too long

2014-10-17 Thread Jason Wolfe
On Thu, Oct 16, 2014 at 12:23 PM, John Baldwin wrote: > > > I looked at the other trace and I don't think it disagrees with my previous > theory. I do have more KTR patches to log when we spin on locks which would > really confirm this, but I haven't tested those fully on HEAD yet. > > However, I

Re: ixgbe(4) spin lock held too long

2014-10-17 Thread Jason Wolfe
On Thu, Oct 16, 2014 at 12:23 PM, John Baldwin wrote: > > > I looked at the other trace and I don't think it disagrees with my previous > theory. I do have more KTR patches to log when we spin on locks which would > really confirm this, but I haven't tested those fully on HEAD yet. > > However, I

Re: ixgbe(4) spin lock held too long

2014-10-13 Thread Jason Wolfe
On Fri, Oct 10, 2014 at 11:19 PM, Jason Wolfe wrote: > On Fri, Oct 10, 2014 at 8:53 AM, John Baldwin wrote: > >> On Thursday, October 09, 2014 02:31:32 PM Jason Wolfe wrote: >> > On Wed, Oct 8, 2014 at 12:29 PM, John Baldwin wrote: >> > > My only other thoug

Re: ixgbe(4) spin lock held too long

2014-10-10 Thread Jason Wolfe
On Fri, Oct 10, 2014 at 8:53 AM, John Baldwin wrote: > On Thursday, October 09, 2014 02:31:32 PM Jason Wolfe wrote: > > On Wed, Oct 8, 2014 at 12:29 PM, John Baldwin wrote: > > > My only other thought is if a direct timeout routine ran for a long > time. > > > >

Re: ixgbe(4) spin lock held too long

2014-10-09 Thread Jason Wolfe
On Wed, Oct 8, 2014 at 12:29 PM, John Baldwin wrote: > My only other thought is if a direct timeout routine ran for a long time. > > I just committed a change to current that can let you capture KTR traces of > callout routines for use with schedgraph (r272757). Unfortunately, > enabling KTR_SCH

Re: ixgbe(4) spin lock held too long

2014-10-08 Thread Jason Wolfe
On Tue, Oct 7, 2014 at 11:28 AM, John Baldwin wrote: > On Tuesday, October 07, 2014 2:06:42 pm Jason Wolfe wrote: > > Hey John, > > > > Happy to do this, but the pool of boxes is about 500 large, which is the > > reason I'm able to see a crash every day or so. I

Re: ixgbe(4) spin lock held too long

2014-10-02 Thread Jason Wolfe
On Wed, Sep 10, 2014 at 8:24 AM, John Baldwin wrote: > On Monday, September 08, 2014 03:34:02 PM Eric van Gyzen wrote: > > On 09/08/2014 15:19, Sean Bruno wrote: > > > On Mon, 2014-09-08 at 12:09 -0700, Sean Bruno wrote: > > >> This sort of looks like the hardware failed to respond to us in time?

Re: kern/172675: [netinet] [patch] sysctl_tcp_hc_list (net.inet.tcp.hostcache.list) race condition causing memory corruption

2013-06-08 Thread Jason Wolfe
The following reply was made to PR kern/172675; it has been noted by GNATS. From: Jason Wolfe To: bug-follo...@freebsd.org Cc: Subject: Re: kern/172675: [netinet] [patch] sysctl_tcp_hc_list (net.inet.tcp.hostcache.list) race condition causing memory corruption Date: Sat, 08 Jun 2013 02:20:48

Re: kern/172675: [netinet] [patch] sysctl_tcp_hc_list (net.inet.tcp.hostcache.list) race condition causing memory corruption

2013-06-08 Thread Jason Wolfe
The following reply was made to PR kern/172675; it has been noted by GNATS. From: Jason Wolfe To: bug-follo...@freebsd.org Cc: Subject: Re: kern/172675: [netinet] [patch] sysctl_tcp_hc_list (net.inet.tcp.hostcache.list) race condition causing memory corruption Date: Sat, 08 Jun 2013 01:11:21

Session MTU update patch only 9

2012-11-02 Thread Jason Wolfe
Hi, While attempting to fix a PMTUD bug in 8.3-RELEASE-p4 that appears to now be resolved by r234342, I also came across r238516. It appears this fix only made it into 9/STABLE though, does anyone know why it wasn't brought back to 8? It's a fairly easy port, but I just wanted to confirm there w

Re: Dropping TCP options from retransmitted SYNs considered harmful

2012-10-13 Thread Jason Wolfe
On Fri, Oct 12, 2012 at 9:13 AM, John Baldwin wrote: > Back in 2001 FreeBSD added a hack to strip TCP options from retransmitted SYNs > starting with the 3rd SYN in this block in tcp_timer.c: > > /* > * Disable rfc1323 if we haven't got any response to > * our third SYN t

Re: 82574L hangs (with r233708 e1000 driver).

2012-08-21 Thread Jason Wolfe
On Fri, Aug 17, 2012 at 7:23 AM, Barney Cordoba wrote: > > --- On Thu, 8/9/12, Jason Wolfe wrote: >> >> Ever since r235553 the 82574L has been stable for me, >> collectively >> passing ~1.2Tb/s for the past 4 months without issue. >> We did have >>

Re: 82574L hangs (with r233708 e1000 driver).

2012-08-09 Thread Jason Wolfe
On Thu, Aug 9, 2012 at 8:25 AM, Barney Cordoba wrote: >> --- On Fri, 5/11/12, Barney Cordoba wrote: >> >> FWIW, I've got an X7SPE-HF-D525 MB with 82574L running on a >> 7.0 driver >> that seems to work pretty well. It panics once in a blue >> moon when we >> overload it (like 200Mb/s of traffic)

Re: Broadcom NetXtreme BCM5719 support

2012-07-25 Thread Jason Wolfe
On Thu, Jul 12, 2012 at 11:02 PM, Eugene M. Zheganin wrote: > Hi. > > > On 13.07.2012 04:39, Jason Wolfe wrote: >> >> bge0: mem >> 0xf6bf-0xf6bf,0xf6be-0xf6be,0xf6bd-0xf6bd irq >> 32 at device 0.0 on pci3 >> bge0: CHIP ID 0x057190

Broadcom NetXtreme BCM5719 support

2012-07-12 Thread Jason Wolfe
Hi, I have an HP ProLiant DL360p Gen8 server in my possession for testing, and it appears the Broadcom NetXtreme BCM5719 doesn't yet have support. I noticed someone asking about server support on freebsd-questions last month, but it didn't garner much attention. Played around with 8.3-RELEASE, 8-

Re: 82574L hangs (with r233708 e1000 driver).

2012-04-07 Thread Jason Wolfe
On Sat, Apr 7, 2012 at 6:37 AM, Konstantin Belousov wrote: > I bought Intel Atom motherboard DN2800MT, which has integrated if_em > LOM, reported by pciconf as > em0@pci0:1:0:0: class=0x02 card=0x20128086 chip=0x10d38086 rev=0x00 > hdr=0x00 > 82574L Gigabit Network Connection > It seems that

Re: Possible interoperability issue with TCP timestamps between FreeBSD and Linux

2012-03-31 Thread Jason Wolfe
On Sat, Mar 31, 2012 at 5:25 AM, Andre Oppermann wrote: > On 31.03.2012 00:07, Jason Wolfe wrote: >> >> So I'm seeing an issue that appears to be caused by the strict TCP >> timestamp adherence in the Linux kernel when there are connections >> being initiated by b

Possible interoperability issue with TCP timestamps between FreeBSD and Linux

2012-03-30 Thread Jason Wolfe
So I'm seeing an issue that appears to be caused by the strict TCP timestamp adherence in the Linux kernel when there are connections being initiated by both sides of a Linux and FreeBSD pair of servers. What is looks like is FreeBSD is only tracking the timestamp for the remote host for connection

Re: Intel 82574L interface wedging - em7.3.2/8.2-STABLE

2012-03-23 Thread Jason Wolfe
On Fri, Mar 23, 2012 at 11:09 AM, Mike Tancsa wrote: > On 3/20/2012 2:57 PM, John Baldwin wrote: TX when link becomes active.  I've also updated it to fix resume for em and igb to DTRT when buf_ring is used, and to not include old-style start routines at all when using multiq.  It i

Re: Intel 82574L interface wedging - em7.3.2/8.2-STABLE

2012-03-20 Thread Jason Wolfe
On Thu, Mar 15, 2012 at 11:17 AM, John Baldwin wrote: > On Sunday, March 11, 2012 3:47:07 am Hooman Fazaeli wrote: >> On 3/11/2012 5:31 AM, Adrian Chadd wrote: >> > Are you able to post the patch here? >> > Maybe Jack can look at what's going on and apply it to the latest >> > intel ethernet drive

Re: Stuck in FIN_WAIT_2

2012-03-19 Thread Jason Wolfe
Is 'tcpdrop' present? Looks like it was in 5 at least, but I couldn't validate in 4. At any rate sounds like your going the upgrade route. I've had no issues with a 20s timeout and fast recycle as mentioned in 8.X also. Jason On Thu, Mar 15, 2012 at 2:43 PM, grarpamp wrote: >> net.inet.tcp.fin

Re: Intel 82574L interface wedging - em7.3.2/8.2-STABLE

2012-03-07 Thread Jason Wolfe
On Thu, Mar 1, 2012 at 12:31 PM, Jason Wolfe wrote: > So since the 7.3.0/7.3.2 code released out of the "Intel 82574L > interface wedging on em 7.1.9/7.2.3 when MSIX enabled" thread I've > been having some good results in 8.2-STABLE, and 'wedges' are much > le

Intel 82574L interface wedging - em7.3.2/8.2-STABLE

2012-03-01 Thread Jason Wolfe
So since the 7.3.0/7.3.2 code released out of the "Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled" thread I've been having some good results in 8.2-STABLE, and 'wedges' are much less common. I am however still seeing them rarely, using some fuzzy math based on uptime on the new

Re: em loses link after IPMI reset

2012-01-16 Thread Jason Wolfe
Andrey, Are you running Supermicro hardware by chance? We had a very similar issue that was caused by a buggy version of the IPMI firmware on a good number of machines with the same 82574 chips. At any rate, might give the IPMI update a try and see if that changes the behavior at all. Jason On

Re: Arg. TCP slow start killing me.

2011-11-13 Thread Jason Wolfe
On Sun, Nov 13, 2011 at 4:53 PM, Erich Weiler wrote: > I suspect my firewall *is* the cause of the packet loss, unfortunately. >> We're sending multiple streams in from multiple sources and >> destinations, but the aggregate bandwidth coming into the firewall is >> consistent no matter how many s

Re: Arg. TCP slow start killing me.

2011-11-13 Thread Jason Wolfe
Erich, Forgot to mention net.inet.tcp.delayed_ack can be a detriment in latent paths, might try setting it to 0 to see if it improves your throughput. Jason Wolfe On Sun, Nov 13, 2011 at 2:48 PM, Jason Wolfe wrote: > Erich, > > Slow start is actually just the initial ramp up limit

Re: Arg. TCP slow start killing me.

2011-11-13 Thread Jason Wolfe
ially show a sizable positive impact in the saw tooth. net.inet.tcp.inflight.enable=0 Is it possible to upgrade to 8.2-STABLE? Cubic has shown some really great improvement in my latent paths, a steady 10% overall increase in same cases. Jason Wolfe On Sun, Nov 13, 2011 at 2:16 PM, Erich Weiler

Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled

2011-11-08 Thread Jason Wolfe
On Tue, Nov 8, 2011 at 10:21 AM, Hooman Fazaeli wrote: > I have allocated more time to the problem and guess I can explain what > your problem is. > > With MSIX disabled, the driver uses fast interrupt handler (em_irq_fast) > which calls rx/tx task and then checks for link status change. This > i

Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled

2011-11-07 Thread Jason Wolfe
On Mon, Oct 31, 2011 at 1:13 AM, Hooman Fazaeli wrote: > > Attached is a patch for if_em.c. It flushes interface queue when it is > full > and link is not active. Please note that when this happens, drops are > increasing > on interface and this will trigger your scripts as before. You need to > c

Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled

2011-10-30 Thread Jason Wolfe
On Sun, Oct 30, 2011 at 1:57 AM, Hooman Fazaeli wrote: > > I finally managed to re-produce an affect similar to Jason's case. It > may not be the exact same issue, but it is a serious problem and must > be addressed. > > 1. Push out packet on em/igb with high rate. > 2. Disconnect cable and wait f

Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled

2011-10-26 Thread Jason Wolfe
On Wed, Oct 26, 2011 at 6:16 AM, Mike Tancsa wrote: > On 10/26/2011 7:33 AM, Jason Wolfe wrote: > > Hooman, > > > > I have run with dev.em.X.flow_control=0, which should have the same > result > > as hw.em.fc_setting=0, and net.inet.tcp.tso is also 0. I'm

Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled

2011-10-26 Thread Jason Wolfe
You may add the > following line to the end of em_print_debug_info function > to report link status when the problem happens: > >device_printf(dev, "Link state: %s\n", >adapter->link_active? "active": "inactive"); > > > > > On 10/26/2

Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled

2011-10-26 Thread Jason Wolfe
Hooman, I have run with dev.em.X.flow_control=0, which should have the same result as hw.em.fc_setting=0, and net.inet.tcp.tso is also 0. I'm not sure the remaining options would be able to produce the scenario I'm seeing, but I'm open to giving it a try with no options on the interfaces. I've a

Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled

2011-10-25 Thread Jason Wolfe
On Fri, Oct 7, 2011 at 2:14 PM, Jason Wolfe wrote: > Bumping rx/tx descriptors to 2048 was actually for performance reasons and > not to try to get around the issue. I did some fairly in depth testing and > found under heavy load it performed the best with those settings. > > As m

Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled

2011-10-11 Thread Jason Wolfe
On Fri, Oct 7, 2011 at 2:14 PM, Jason Wolfe wrote: > Bumping rx/tx descriptors to 2048 was actually for performance reasons and > not to try to get around the issue. I did some fairly in depth testing and > found under heavy load it performed the best with those settings. > > As m

Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled

2011-10-07 Thread Jason Wolfe
On Fri, Oct 7, 2011 at 12:55 PM, Arnaud Lacombe wrote: > Hi, > > On Fri, Oct 7, 2011 at 3:24 PM, Mike Tancsa wrote: > > When you disable MSI-X, you mean via hw.pci.enable_msix=0 across the > > board, or you disable multi-queue for the NIC, so it uses just one > > interrupt, rather than separate

Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled

2011-10-07 Thread Jason Wolfe
On Fri, Oct 7, 2011 at 12:55 PM, Arnaud Lacombe wrote: > Hi, > > On Fri, Oct 7, 2011 at 3:24 PM, Mike Tancsa wrote: > > When you disable MSI-X, you mean via hw.pci.enable_msix=0 across the > > board, or you disable multi-queue for the NIC, so it uses just one > > interrupt, rather than separate

Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled

2011-10-07 Thread Jason Wolfe
On Fri, Oct 7, 2011 at 12:24 PM, Arnaud Lacombe wrote: > Hi, > > On Fri, Oct 7, 2011 at 2:57 PM, Jason Wolfe wrote: > > Jack, > > > > Entirely possible there are multiple moving pieces here, the only bit I > know > > for certain is it's related to the di

Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled

2011-10-07 Thread Jason Wolfe
email I'm happy to re enable it on a handful of servers and collect some fresh reports. Jason On Fri, Oct 7, 2011 at 8:06 AM, Mike Tancsa wrote: > On 10/6/2011 7:15 PM, Jason Wolfe wrote: > > I'm seeing the interface wedge on a good number of systems with Intel > 82

Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled

2011-10-07 Thread Jason Wolfe
of having a hang once in maybe 3 weeks, and then that was fixed. > I have > not heard of a problem from Mike since. > > So, I'd say something else must be coming into play here, I'll stare at the > data > a bit when I get a chance and see if anything jumps out at me

Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled

2011-10-06 Thread Jason Wolfe
i0:255:5:1: class=0x06 card=0x80868086 chip=0x2da98086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI hostb19@pci0:255:5:2: class=0x06 card=0x80868086 chip=0x2daa8086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' class = bridge s