Re: [ovs-discuss] kernel panic under heavy receive load

2015-03-05 Thread Chris Dunlop
On Wed, Mar 04, 2015 at 03:56:41PM -0500, Xu (Simon) Chen wrote: > Now I think about it... Maybe I run into this problem because I turned on > Sflow collection on my OVS nodes... Yes, I was also running sflow collection. I think the bug will only bite when doing sflow collection because it's the s

Re: [ovs-discuss] kernel panic under heavy receive load

2015-03-04 Thread Pravin Shelar
On Wed, Mar 4, 2015 at 6:16 AM, Chris Dunlop wrote: > On Sat, Feb 28, 2015 at 12:49:31PM +1100, Chris Dunlop wrote: >> On Fri, Feb 27, 2015 at 08:30:42PM -0500, Xu (Simon) Chen wrote: >> > On Fri, Feb 27, 2015 at 6:14 PM, Pravin Shelar wrote: >> > > So it looks like vhost is generating shared skb

Re: [ovs-discuss] kernel panic under heavy receive load

2015-03-04 Thread Chris Dunlop
On Sat, Feb 28, 2015 at 12:49:31PM +1100, Chris Dunlop wrote: > On Fri, Feb 27, 2015 at 08:30:42PM -0500, Xu (Simon) Chen wrote: > > On Fri, Feb 27, 2015 at 6:14 PM, Pravin Shelar wrote: > > > So it looks like vhost is generating shared skb. Can you try same test > > > on latest upstream kernel? >

Re: [ovs-discuss] kernel panic under heavy receive load

2015-02-27 Thread Chris Dunlop
On Fri, Feb 27, 2015 at 08:33:13PM -0500, Xu (Simon) Chen wrote: > On Fri, Feb 27, 2015 at 7:13 PM, Chris Dunlop wrote: > > On Fri, Feb 27, 2015 at 11:08:21AM -0800, Pravin Shelar wrote: > > > Can you reproduce this bug on hypervisor to hypervisor > > > test without any VMs? > > > > The thought ju

Re: [ovs-discuss] kernel panic under heavy receive load

2015-02-27 Thread Chris Dunlop
On Fri, Feb 27, 2015 at 08:31:46PM -0500, Xu (Simon) Chen wrote: > On Fri, Feb 27, 2015 at 6:49 PM, Chris Dunlop wrote: > > On Fri, Feb 27, 2015 at 06:06:25PM -0500, Xu (Simon) Chen wrote: > > > On Friday, February 27, 2015, Chris Dunlop wrote: > > > > Simon, are you able to try your test running

Re: [ovs-discuss] kernel panic under heavy receive load

2015-02-27 Thread Chris Dunlop
On Fri, Feb 27, 2015 at 08:30:42PM -0500, Xu (Simon) Chen wrote: > On Fri, Feb 27, 2015 at 6:14 PM, Pravin Shelar wrote: > > So it looks like vhost is generating shared skb. Can you try same test > > on latest upstream kernel? > > I don't know whether it's vhost, as for me the VM receiving high v

Re: [ovs-discuss] kernel panic under heavy receive load

2015-02-27 Thread Xu (Simon) Chen
On Fri, Feb 27, 2015 at 7:13 PM, Chris Dunlop wrote: > On Fri, Feb 27, 2015 at 11:08:21AM -0800, Pravin Shelar wrote: > > On Thu, Feb 26, 2015 at 9:13 PM, Chris Dunlop > wrote: > > > Hi, > > > > > > "Me too" on Simon's BUG() described below (apologies for the top post). > > > Basically: > > > >

Re: [ovs-discuss] kernel panic under heavy receive load

2015-02-27 Thread Xu (Simon) Chen
On Fri, Feb 27, 2015 at 6:49 PM, Chris Dunlop wrote: > On Fri, Feb 27, 2015 at 06:06:25PM -0500, Xu (Simon) Chen wrote: > > On Friday, February 27, 2015, Chris Dunlop wrote: > > > Simon, are you able to try your test running direct hypervisor > > > to hypervisor? > > > > Nope... I have only see

Re: [ovs-discuss] kernel panic under heavy receive load

2015-02-27 Thread Xu (Simon) Chen
On Fri, Feb 27, 2015 at 6:14 PM, Pravin Shelar wrote: > On Fri, Feb 27, 2015 at 3:06 PM, Xu (Simon) Chen > wrote: > > > > > > On Friday, February 27, 2015, Chris Dunlop wrote: > >> > >> On Fri, Feb 27, 2015 at 11:08:21AM -0800, Pravin Shelar wrote: > >> > On Thu, Feb 26, 2015 at 9:13 PM, Chris

Re: [ovs-discuss] kernel panic under heavy receive load

2015-02-27 Thread Chris Dunlop
On Fri, Feb 27, 2015 at 11:08:21AM -0800, Pravin Shelar wrote: > On Thu, Feb 26, 2015 at 9:13 PM, Chris Dunlop wrote: > > Hi, > > > > "Me too" on Simon's BUG() described below (apologies for the top post). > > Basically: > > > > [ 7318.409796] kernel BUG at net/core/skbuff.c:1041! > > ... > > [ 73

Re: [ovs-discuss] kernel panic under heavy receive load

2015-02-27 Thread Chris Dunlop
On Fri, Feb 27, 2015 at 06:06:25PM -0500, Xu (Simon) Chen wrote: > On Friday, February 27, 2015, Chris Dunlop wrote: > > Simon, are you able to try your test running direct hypervisor > > to hypervisor? > > Nope... I have only seen this between VMs. After repeated tests, it has > got worse that

Re: [ovs-discuss] kernel panic under heavy receive load

2015-02-27 Thread Pravin Shelar
On Fri, Feb 27, 2015 at 3:06 PM, Xu (Simon) Chen wrote: > > > On Friday, February 27, 2015, Chris Dunlop wrote: >> >> On Fri, Feb 27, 2015 at 11:08:21AM -0800, Pravin Shelar wrote: >> > On Thu, Feb 26, 2015 at 9:13 PM, Chris Dunlop >> > wrote: >> > > Hi, >> > > >> > > "Me too" on Simon's BUG() d

Re: [ovs-discuss] kernel panic under heavy receive load

2015-02-27 Thread Xu (Simon) Chen
On Friday, February 27, 2015, Chris Dunlop wrote: > On Fri, Feb 27, 2015 at 11:08:21AM -0800, Pravin Shelar wrote: > > On Thu, Feb 26, 2015 at 9:13 PM, Chris Dunlop > wrote: > > > Hi, > > > > > > "Me too" on Simon's BUG() described below (apologies for the top post). > > > Basically: > > > > > >

Re: [ovs-discuss] kernel panic under heavy receive load

2015-02-27 Thread Chris Dunlop
On Fri, Feb 27, 2015 at 11:08:21AM -0800, Pravin Shelar wrote: > On Thu, Feb 26, 2015 at 9:13 PM, Chris Dunlop wrote: > > Hi, > > > > "Me too" on Simon's BUG() described below (apologies for the top post). > > Basically: > > > > [ 7318.409796] kernel BUG at net/core/skbuff.c:1041! > > ... > > [ 73

Re: [ovs-discuss] kernel panic under heavy receive load

2015-02-27 Thread Pravin Shelar
On Thu, Feb 26, 2015 at 9:13 PM, Chris Dunlop wrote: > Hi, > > "Me too" on Simon's BUG() described below (apologies for the top post). > Basically: > > [ 7318.409796] kernel BUG at net/core/skbuff.c:1041! > ... > [ 7318.591562] RIP: 0010:[] [] > pskb_expand_head+0x234/0x270 > ... > [ 7318.705710

Re: [ovs-discuss] kernel panic under heavy receive load

2015-02-26 Thread Chris Dunlop
Hi, "Me too" on Simon's BUG() described below (apologies for the top post). Basically: [ 7318.409796] kernel BUG at net/core/skbuff.c:1041! ... [ 7318.591562] RIP: 0010:[] [] pskb_expand_head+0x234/0x270 ... [ 7318.705710] [] __pskb_pull_tail+0x4c/0x330 [ 7318.711571] [] skb_checksum_help+0x1

Re: [ovs-discuss] kernel panic under heavy receive load

2015-02-12 Thread Xu (Simon) Chen
Sorry, click send too fast... I think this line causes the KP: https://github.com/torvalds/linux/blob/v3.14/net/core/skbuff.c#L1039 But this is weird, because as I read from this mailing list, OVS doesn't allow shared skb. I tried turn off GRO: it made receive side slower, but it eventually crash

[ovs-discuss] kernel panic under heavy receive load

2015-02-12 Thread Xu (Simon) Chen
Hi folks, I can now consistently reproduce a kernel panic on my system. I am using OVS 2.3.0 on 3.14.29 kernel, a sender and a receiver (two VMs) on two identical hypervisors, using VXLAN tunnel connecting the two VMs. Iperf is used inside of VMs for generating traffic. The sender side has no pro