Re: ixgbe & if_igb RX ring locking

2012-10-31 Thread Alexander V. Chernikov
On 22.10.2012 05:43, Ryan Stone wrote: On Sun, Oct 21, 2012 at 2:21 PM, Alexander V. Chernikov wrote: ix:rx -> udp is also fairly obvious (here's one backtrace): The same question, where "udp" -> "ix:rx" can happen ? It can't happen directly as far as I can tell. But to trigger a deadlock,

Re: ixgbe & if_igb RX ring locking

2012-10-21 Thread Ryan Stone
On Sun, Oct 21, 2012 at 2:21 PM, Alexander V. Chernikov wrote: >> ix:rx -> udp is also fairly obvious (here's one backtrace): > The same question, where "udp" -> "ix:rx" can happen ? It can't happen directly as far as I can tell. But to trigger a deadlock, "all" that has to happen is that a thr

Re: ixgbe & if_igb RX ring locking

2012-10-21 Thread Alexander V. Chernikov
On 16.10.2012 17:20, Ryan Stone wrote: On Tue, Oct 16, 2012 at 8:47 AM, Gleb Smirnoff wrote: Can you please provide hints how can SIOCADDMULTI lead to obtaining RX lock in the stock driver? It doesn't. But it does acquire the core lock, and the core lock is acquired before the RX lock (in ix

Re: ixgbe & if_igb RX ring locking

2012-10-19 Thread Jack Vogel
On Fri, Oct 19, 2012 at 8:23 AM, Alexander V. Chernikov < melif...@freebsd.org> wrote: > On 17.10.2012 18:06, John Baldwin wrote: > >> On Monday, October 15, 2012 9:04:27 am John Baldwin wrote: >> >>> On Monday, October 15, 2012 10:10:40 am Alexander V. Chernikov wrote: >>> On 13.10.2012 23:2

Re: ixgbe & if_igb RX ring locking

2012-10-19 Thread Alexander V. Chernikov
On 17.10.2012 18:06, John Baldwin wrote: On Monday, October 15, 2012 9:04:27 am John Baldwin wrote: On Monday, October 15, 2012 10:10:40 am Alexander V. Chernikov wrote: On 13.10.2012 23:24, Jack Vogel wrote: On Sat, Oct 13, 2012 at 11:22 AM, Luigi Rizzo wrote: one option could be (same a

Re: ixgbe & if_igb RX ring locking

2012-10-19 Thread Fabien Thomas
Le 18 oct. 2012 à 20:09, Jack Vogel a écrit : > On Thu, Oct 18, 2012 at 6:20 AM, Andre Oppermann wrote: > >> On 13.10.2012 20:22, Luigi Rizzo wrote: >> >>> On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov wrote: >>> Hello list! Packets receiving code for b

Re: ixgbe & if_igb RX ring locking

2012-10-18 Thread Luigi Rizzo
On Thu, Oct 18, 2012 at 03:20:14PM +0200, Andre Oppermann wrote: > On 13.10.2012 20:22, Luigi Rizzo wrote: > >On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov wrote: > >>Hello list! > >> > >> > >>Packets receiving code for both ixgbe and if_igb looks like the following: > >> > >> >

Re: ixgbe & if_igb RX ring locking

2012-10-18 Thread Jack Vogel
On Thu, Oct 18, 2012 at 6:20 AM, Andre Oppermann wrote: > On 13.10.2012 20:22, Luigi Rizzo wrote: > >> On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov wrote: >> >>> Hello list! >>> >>> >>> Packets receiving code for both ixgbe and if_igb looks like the >>> following: >>> >>> >>> i

Re: ixgbe & if_igb RX ring locking

2012-10-18 Thread Andre Oppermann
On 13.10.2012 20:22, Luigi Rizzo wrote: On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov wrote: Hello list! Packets receiving code for both ixgbe and if_igb looks like the following: ixgbe_msix_que -- ixgbe_rxeof() { IXGBE_RX_LOCK(rxr); while {

Re: ixgbe & if_igb RX ring locking

2012-10-17 Thread John Baldwin
On Monday, October 15, 2012 9:04:27 am John Baldwin wrote: > On Monday, October 15, 2012 10:10:40 am Alexander V. Chernikov wrote: > > On 13.10.2012 23:24, Jack Vogel wrote: > > > On Sat, Oct 13, 2012 at 11:22 AM, Luigi Rizzo wrote: > > > > >> > > >> one option could be (same as it is done in the

Re: ixgbe & if_igb RX ring locking

2012-10-16 Thread Adrian Chadd
On 16 October 2012 05:38, John Baldwin wrote: > I don't follow how this is related to this thread at all (which has more to do > with ixgbe scheduling duplicate work). However, is your issue that the stack > locks (e.g. socket and protocol layer locks) are held across > if_start/if_transmit? It

Re: ixgbe & if_igb RX ring locking

2012-10-16 Thread John Baldwin
On Tuesday, October 16, 2012 8:46:49 am Alexander V. Chernikov wrote: > On 15.10.2012 22:14, John Baldwin wrote: > > On Monday, October 15, 2012 12:32:10 pm Gleb Smirnoff wrote: > >> On Mon, Oct 15, 2012 at 09:04:27AM -0400, John Baldwin wrote: > >> J> > 3) in practice taskqueue routine is a nightm

Re: ixgbe & if_igb RX ring locking

2012-10-16 Thread Ryan Stone
On Tue, Oct 16, 2012 at 8:47 AM, Gleb Smirnoff wrote: > Can you please provide hints how can SIOCADDMULTI lead to obtaining RX > lock in the stock driver? It doesn't. But it does acquire the core lock, and the core lock is acquired before the RX lock (in ixgbe_init, for instance). __

Re: ixgbe & if_igb RX ring locking

2012-10-16 Thread John Baldwin
On Monday, October 15, 2012 6:36:57 pm Adrian Chadd wrote: > The reason why I've started moving net80211 and ath _away_ from using > direct dispatch (for now) and to using a taskqueue for TX (and RX) is > because it's too freaking annoying right now to deal with all the > crazy long-held locks to g

Re: ixgbe & if_igb RX ring locking

2012-10-16 Thread Alexander V. Chernikov
On 15.10.2012 22:14, John Baldwin wrote: On Monday, October 15, 2012 12:32:10 pm Gleb Smirnoff wrote: On Mon, Oct 15, 2012 at 09:04:27AM -0400, John Baldwin wrote: J> > 3) in practice taskqueue routine is a nightmare for many people since J> > there is no way to stop "kernel {ix0 que}" thread ea

Re: ixgbe & if_igb RX ring locking

2012-10-16 Thread Gleb Smirnoff
On Tue, Oct 16, 2012 at 08:40:47AM -0400, Ryan Stone wrote: R> > Are you using stock ixgbe driver? R> R> Pay no attention to the line numbers behind the curtain. :) R> R> I don't believe that I've changed the locking order at all in the R> driver, but you are right, that wasn't taken from the sto

Re: ixgbe & if_igb RX ring locking

2012-10-16 Thread Ryan Stone
On Tue, Oct 16, 2012 at 8:12 AM, Alexander V. Chernikov wrote: > Are you using stock ixgbe driver? Pay no attention to the line numbers behind the curtain. :) I don't believe that I've changed the locking order at all in the driver, but you are right, that wasn't taken from the stock driver. ___

Re: ixgbe & if_igb RX ring locking

2012-10-16 Thread Alexander V. Chernikov
On 16.10.2012 00:48, Ryan Stone wrote: On Mon, Oct 15, 2012 at 12:29 PM, Gleb Smirnoff wrote: To me this unlock/lock looks like a legacy from times, when the driver had a single mutex for both TX and RX parts. And removing this re-locking in foo_rxeof() was one of the aims for separate TX/RX l

Re: ixgbe & if_igb RX ring locking

2012-10-15 Thread Adrian Chadd
The reason why I've started moving net80211 and ath _away_ from using direct dispatch (for now) and to using a taskqueue for TX (and RX) is because it's too freaking annoying right now to deal with all the crazy long-held locks to guarantee consistency between multiple transmitting threads. Consid

Re: ixgbe & if_igb RX ring locking

2012-10-15 Thread Ryan Stone
On Mon, Oct 15, 2012 at 12:29 PM, Gleb Smirnoff wrote: > To me this unlock/lock looks like a legacy from times, when the driver > had a single mutex for both TX and RX parts. > > And removing this re-locking in foo_rxeof() was one of the aims for separate > TX/RX locking. > > Really, lurking throu

Re: ixgbe & if_igb RX ring locking

2012-10-15 Thread John Baldwin
On Monday, October 15, 2012 12:32:10 pm Gleb Smirnoff wrote: > On Mon, Oct 15, 2012 at 09:04:27AM -0400, John Baldwin wrote: > J> > 3) in practice taskqueue routine is a nightmare for many people since > J> > there is no way to stop "kernel {ix0 que}" thread eating 100% cpu after > J> > some traf

Re: ixgbe & if_igb RX ring locking

2012-10-15 Thread Jack Vogel
On Mon, Oct 15, 2012 at 9:58 AM, Gleb Smirnoff wrote: > On Mon, Oct 15, 2012 at 09:39:24AM -0700, Jack Vogel wrote: > J> > To me this unlock/lock looks like a legacy from times, when the driver > J> > had a single mutex for both TX and RX parts. > J> > > J> > And removing this re-locking in foo_r

Re: ixgbe & if_igb RX ring locking

2012-10-15 Thread Gleb Smirnoff
On Mon, Oct 15, 2012 at 09:39:24AM -0700, Jack Vogel wrote: J> > To me this unlock/lock looks like a legacy from times, when the driver J> > had a single mutex for both TX and RX parts. J> > J> > And removing this re-locking in foo_rxeof() was one of the aims for J> > separate J> > TX/RX locking. J

Re: ixgbe & if_igb RX ring locking

2012-10-15 Thread Alexander V. Chernikov
On 15.10.2012 20:39, Jack Vogel wrote: I did not want to add it back, there were problems that constrained me to do so, although its been some time, I'd be happy to do some testing again without and see. We've got more than hundred routers/firewalls running under heavy load without this lock

Re: ixgbe & if_igb RX ring locking

2012-10-15 Thread Jack Vogel
On Mon, Oct 15, 2012 at 9:29 AM, Gleb Smirnoff wrote: > On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov wrote: > A> Packets receiving code for both ixgbe and if_igb looks like the > following: > A> ixgbe_msix_que > A> > A> -- ixgbe_rxeof() > A> { > A>IXGBE_RX_LOCK(rxr

Re: ixgbe & if_igb RX ring locking

2012-10-15 Thread Gleb Smirnoff
On Mon, Oct 15, 2012 at 09:04:27AM -0400, John Baldwin wrote: J> > 3) in practice taskqueue routine is a nightmare for many people since J> > there is no way to stop "kernel {ix0 que}" thread eating 100% cpu after J> > some traffic burst happens: once it is called it starts to schedule J> > itse

Re: ixgbe & if_igb RX ring locking

2012-10-15 Thread Gleb Smirnoff
On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov wrote: A> Packets receiving code for both ixgbe and if_igb looks like the following: A> ixgbe_msix_que A> A> -- ixgbe_rxeof() A> { A>IXGBE_RX_LOCK(rxr); A> while A> { A> get_packet; A> A>

Re: ixgbe & if_igb RX ring locking

2012-10-15 Thread John Baldwin
On Monday, October 15, 2012 10:10:40 am Alexander V. Chernikov wrote: > On 13.10.2012 23:24, Jack Vogel wrote: > > On Sat, Oct 13, 2012 at 11:22 AM, Luigi Rizzo wrote: > > >> > >> one option could be (same as it is done in the timer > >> routine in dummynet) to build a list of all the packets > >

Re: ixgbe & if_igb RX ring locking

2012-10-15 Thread Alexander V. Chernikov
On 13.10.2012 23:24, Jack Vogel wrote: On Sat, Oct 13, 2012 at 11:22 AM, Luigi Rizzo wrote: one option could be (same as it is done in the timer routine in dummynet) to build a list of all the packets that need to be sent to if_input(), and then call if_input with the entire list outside the

Re: ixgbe & if_igb RX ring locking

2012-10-14 Thread Adrian Chadd
God, yes please. Please please please please. Adrian ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: ixgbe & if_igb RX ring locking

2012-10-13 Thread Jack Vogel
On Sat, Oct 13, 2012 at 11:22 AM, Luigi Rizzo wrote: > On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov wrote: > > Hello list! > > > > > > Packets receiving code for both ixgbe and if_igb looks like the > following: > > > > > > ixgbe_msix_que > > > > -- ixgbe_rxeof() > >{ > >

Re: ixgbe & if_igb RX ring locking

2012-10-13 Thread Luigi Rizzo
On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov wrote: > Hello list! > > > Packets receiving code for both ixgbe and if_igb looks like the following: > > > ixgbe_msix_que > > -- ixgbe_rxeof() >{ > IXGBE_RX_LOCK(rxr); > while > { >get_packe

ixgbe & if_igb RX ring locking

2012-10-13 Thread Alexander V. Chernikov
Hello list! Packets receiving code for both ixgbe and if_igb looks like the following: ixgbe_msix_que -- ixgbe_rxeof() { IXGBE_RX_LOCK(rxr); while { get_packet; -- ixgbe_rx_input() { ++ IXGBE_RX_UNLOCK(rxr);