Hi.
this bug has been fixed on github.
diff --git a/data/netdev-dpdk.c b/lib/netdev-dpdk.c
index 75c369a..ec49e3a 100644
--- a/data/netdev-dpdk.c
+++ b/lib/netdev-dpdk.c
@@ -776,6 +776,7 @@ netdev_dpdk_vhost_destruct(struct netdev *netdev_)
}
ovs_mutex_lock(&dpdk_mutex);
+ rte_free(dev->tx_q);
list_remove(&dev->list_node);
dpdk_mp_put(dev->dpdk_mp);
ovs_mutex_unlock(&dpdk_mutex);
2016-03-17 9:25 GMT+08:00 王华夏 <wanghua...@jd.com>:
> The ovs-vswitch crashed after the test run sometimes(it can be repeated ),
> and it looks like has memory leak
>
> *[Sugesh] What you mean by sometimes? How much time it takes the exhaust
> the memory? Are you using 1GB hugepages? Make sure you have allocated
> enough 1G hugepages in the system boot. And also increase the socket-mem to
> atleast 1G(--socket-mem 1024) when starting the vswitchd.*
>
>
>
> I started the ovs-vswitchd like this:
>
>
>
> */usr/local/sbin/ovs-vswitchd --dpdk -c 0x1 -n 4 --socket-mem 6000 --
> unix:/usr/local/var/run/openvswitch/db.sock --pidfile –detach*
>
>
>
> *The hugepages of the test environment:*
>
>
>
> *[root@A06-R12-302F0405-I36-100 qemu]# cat /proc/meminfo |grep -i huge*
>
> *AnonHugePages: 292864 kB*
>
> *HugePages_Total: 20480*
>
> *HugePages_Free: 17432*
>
> *HugePages_Rsvd: 0*
>
> *HugePages_Surp: 0*
>
> *Hugepagesize: 2048 kB*
>
>
>
> The out of memory occurred 2 hours later when I started my test, and It
> can replicate in different test environment.
>
>
>
> Test steps:
>
>
>
> Step 1: create a vm for port vhost-user-0
>
> Step 2: create other 15 vms likes step 1
>
> Step 3 :destroy the 15 vms created by step 2
>
> Step 4 repeat step 2 and step 3
>
>
>
> Thanks
>
>
>
> Wang Huaxia ---JD.COM
>
>
>
> *From: *Chandran, Sugesh <sugesh.chand...@intel.com>
> *Sent: *2016年3月17日 1:29
>
> *To: *王华夏 <wanghua...@jd.com>; discuss@openvswitch.org
> *Subject: *RE: [ovs-discuss] [bug] a new problem of ovs-dpdk
>
>
>
>
>
>
>
> *Regards*
>
> *_Sugesh*
>
>
>
> *From:* 王华夏 [mailto:wanghua...@jd.com]
> *Sent:* Wednesday, March 16, 2016 1:30 AM
> *To:* Chandran, Sugesh <sugesh.chand...@intel.com>;
> discuss@openvswitch.org
> *Cc:* Traynor, Kevin <kevin.tray...@intel.com>
> *Subject:* RE: [ovs-discuss] [bug] a new problem of ovs-dpdk
>
>
>
> I tested it with qemu 2.5, and change the ovs version into 2.5 release
> version (it is not the release version before). get another problem.
>
>
>
> The ovs-vswitch crashed after the test run sometimes(it can be repeated ),
> and it looks like has memory leak
>
> *[Sugesh] What you mean by sometimes? How much time it takes the exhaust
> the memory? Are you using 1GB hugepages? Make sure you have allocated
> enough 1G hugepages in the system boot. And also increase the socket-mem to
> atleast 1G(--socket-mem 1024) when starting the vswitchd.*
>
>
>
> The stacks as follows:
>
>
>
> gdb) bt
>
> #0 0x00007f72bfc2f5d7 in raise () from /lib64/libc.so.6
>
> #1 0x00007f72bfc30cc8 in abort () from /lib64/libc.so.6
>
> #2 0x0000000000636fce in ovs_abort_valist (err_no=<optimized out>,
> format=<optimized out>, args=args@entry=0x7ffe3fe2b2f8) at lib/util.c:323
>
> #3 0x0000000000637057 in ovs_abort (err_no=err_no@entry=0,
> format=format@entry=0x6dfad5 "virtual memory exhausted") at lib/util.c:315
>
> *#4 0x0000000000637072 in out_of_memory () at lib/util.c:89*
>
> #5 0x0000000000651465 in dpdk_rte_mzalloc (sz=sz@entry=3170304) at
> lib/netdev-dpdk.c:269
>
> #6 0x0000000000651a33 in netdev_dpdk_alloc_txq
> (netdev=netdev@entry=0x7f72765ec400,
> n_txqs=n_txqs@entry=1024) at lib/netdev-dpdk.c:565
>
> #7 0x000000000065283c in netdev_dpdk_init
> (netdev_=netdev_@entry=0x7f72765ec400,
> port_no=port_no@entry=4294967295, type=type@entry=DPDK_DEV_VHOST) at
> lib/netdev-dpdk.c:633
>
> #8 0x0000000000652a06 in vhost_construct_helper (netdev_=0x7f72765ec400)
> at lib/netdev-dpdk.c:668
>
> #9 netdev_dpdk_vhost_user_construct (netdev_=0x7f72765ec400) at
> lib/netdev-dpdk.c:716
>
> #10 0x00000000005d34d9 in netdev_open (name=<optimized out>,
> type=0x38962d0 "dpdkvhostuser", netdevp=netdevp@entry=0x7ffe3fe2b510) at
> lib/netdev.c:382
>
> #11 0x0000000000562aa7 in iface_do_create (port_cfg=<optimized out>,
> errp=0x7ffe3fe2b500, netdevp=<synthetic pointer>, ofp_portp=0x7ffe3fe2b4fc,
> iface_cfg=0x38c2700, br=0x283def0) at vswitchd/bridge.c:1762
>
> #12 iface_create (port_cfg=0x38c29c0, iface_cfg=0x38c2700, br=0x283def0)
> at vswitchd/bridge.c:1816
>
> #13 bridge_add_ports__ (br=br@entry=0x283def0,
> wanted_ports=wanted_ports@entry=0x283dfd0,
> with_requested_port=with_requested_port@entry=true) at
> vswitchd/bridge.c:892
>
> #14 0x0000000000565645 in bridge_add_ports (wanted_ports=0x283dfd0,
> br=0x283def0) at vswitchd/bridge.c:903
>
> #15 bridge_reconfigure (ovs_cfg=ovs_cfg@entry=0x28415c0) at
> vswitchd/bridge.c:646
>
> #16 0x0000000000568900 in bridge_run () at vswitchd/bridge.c:2975
>
> #17 0x000000000041045d in main (argc=4, argv=0x7ffe3fe2b9a8) at
> vswitchd/ovs-vswitchd.c:120
>
>
>
>
>
> Wang Huaxia ---JD.COM
>
>
>
> *From: *Chandran, Sugesh <sugesh.chand...@intel.com>
> *Sent: *2016年3月15日 19:57
> *To: *王华夏 <wanghua...@jd.com>; discuss@openvswitch.org
> *Cc: *Traynor, Kevin <kevin.tray...@intel.com>
> *Subject: *RE: [ovs-discuss] [bug] a new problem of ovs-dpdk
>
>
>
> I feel the fix for the mentioned issued already present in the DPDK 2.2.
>
>
>
>
> https://github.com/openvswitch/ovs/commit/2d9439f045eab772cd1863ccc2efe0d179064ae5
>
>
>
> So you can use QEMU 2.4.0 to test.
>
>
>
>
>
> *Regards*
>
> *_Sugesh*
>
>
>
> *From:* 王华夏 [mailto:wanghua...@jd.com <wanghua...@jd.com>]
> *Sent:* Monday, March 14, 2016 1:26 AM
> *To:* Chandran, Sugesh <sugesh.chand...@intel.com>;
> discuss@openvswitch.org
> *Subject:* RE: [ovs-discuss] [bug] a new problem of ovs-dpdk
>
>
>
> I did not used the multiqueue in this test. And in ovs user guide ,it
> pointed out :
> *For versions of QEMU v2.4.0 and later, it is currently not possible to
> unbind more than one dpdkvhostuser port from the guest kernel driver without
> causing the ovs-vswitchd process to crash. If this is a requirement for your
> use case, it is recommended either to use a version of QEMU between v2.2.0
> and v2.3.1 (inclusive”*
>
>
>
> So I tested with qemu= 2.3
>
>
>
>
>
> Wang Huaxia ---JD.COM
>
>
>
> *From: *Chandran, Sugesh <sugesh.chand...@intel.com>
> *Sent: *2016年3月12日 1:12
> *To: *王华夏 <wanghua...@jd.com>; discuss@openvswitch.org
> *Subject: *RE: [ovs-discuss] [bug] a new problem of ovs-dpdk
>
>
>
>
>
> Are you using multiqueue for this test setup??
>
> Can you please try with Qemu >=2.4 version instead of 2.3?
>
>
>
>
>
> *Regards*
>
> *_Sugesh*
>
>
>
> *From:* discuss [mailto:discuss-boun...@openvswitch.org
> <discuss-boun...@openvswitch.org>] *On Behalf Of *???
> *Sent:* Friday, March 11, 2016 9:54 AM
> *To:* discuss@openvswitch.org
> *Subject:* [ovs-discuss] [bug] a new problem of ovs-dpdk
>
>
>
> Hi all.
>
> These days ,I tested the vm by using ovs-dpdk ,and found a new problem as
> fellows:
>
>
>
> My test environment :
>
> Host:
>
> Linux version 3.10.0-229.14.1.el7.x86_64 (buil...@kbuilder.dev.centos.org)
> (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Tue Sep 15
> 15:05:51 UTC 2015
>
> Dpdk: version 2.2
>
> Ovs: version 2.5
>
> QEMU version 2.3.1,
>
>
>
> Guest : Linux version 3.10.0-229.el7.x86_64 (
> buil...@kbuilder.dev.centos.org) (gcc version 4.8.2 20140120 (Red Hat
> 4.8.2-16) (GCC) ) #1 SMP Fri Mar 6 11:36:42 UTC 2015
>
>
>
> Ovs:
>
>
>
> 1 S root 61984 1 0 80 0 - 11923 poll_s 14:29 ?
> 00:00:31 ovsdb-server -v
> --remote=punix:/usr/local/var/run/openvswitch/db.sock
> --remote=db:Open_vSwitch,Open_vSwitch,manager_options
> --private-key=db:Open_vSwitch,SSL,private_key
> --certificate=db:Open_vSwitch,SSL,certificate
> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach
>
> 5 S root 61998 1 99 80 0 - 2745735 poll_s 14:29 ?
> 12:25:52 /usr/local/sbin/ovs-vswitchd --dpdk -c 0x1 -n 4 --socket-mem 400
> -- unix:/usr/local/var/run/openvswitch/db.sock --pidfile --detach
>
>
>
> Ovs-port cfg:
>
>
>
> Bridge "br1"
>
> Port "dpdk0"
>
> Interface "dpdk0"
>
> type: dpdk
>
> Port "br1"
>
> Interface "br1"
>
> type: internal
>
> Bridge "br0"
>
> Port "br0"
>
> Interface "br0"
>
> type: internal
>
> Port "vxlan-1"
>
> Interface "vxlan-1"
>
> type: vxlan
>
> options: {remote_ip="7.0.0.2"}
>
> Port "vhost-user-0"
>
> Interface "vhost-user-0"
>
> type: dpdkvhostuser
>
>
>
>
>
> The numa huagepages and vcpu configes in xml :
>
>
>
> <memoryBacking>
>
> <hugepages>
>
> <page size='2048' unit='KiB'/>
>
> </hugepages>
>
> </memoryBacking>
>
> <vcpu placement='static'>4</vcpu>
>
> <os>
>
> <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
>
> <boot dev='hd'/>
>
> </os>
>
> <features>
>
> <acpi/>
>
> </features>
>
> <cpu>
>
> <numa>
>
> <cell id='0' cpus='0-3' memory='4000000' unit='KiB'
> memAccess='shared'/>
>
> </numa>
>
>
>
>
>
> Test steps:
>
> Step 1: create a vm for port vhost-user-0
>
> Step 2: create other 15 vms likes step 1
>
> Step 3 :destroy the 15 vms created by step 2
>
> Step 4 repeat step 2 and step 3
>
>
>
> Then sometimes I find I can’t reach the vm created by step 1, every time,
> this occurred in step 2
>
>
>
> The logs of guest as fellows:
>
>
>
> localhost kernel: virtio_net virtio0: output.0:id 222 is not a head!
>
> localhost kernel: net eth0: Unexpected TXQ (0) queue failure: -5
>
> localhost kernel: net eth0: Unexpected TXQ (0) queue failure: -5
>
> localhost kernel: net eth0: Unexpected TXQ (0) queue failure: -5
>
> then reboot the vm it can be recovered.
>
>
>
> It is obvious that this is network problem. In my opinion, this the
> problems of the hugepages. But indeed , the hugepages has been mapped when
> the ovs-dpdk started by “/usr/local/sbin/ovs-vswitchd --dpdk -c 0x1 -n 4
> --socket-mem 400 -- unix:/usr/local/var/run/openvswitch/db.sock --pidfile
> --detach”I am not sure. How the hugepages recovery when the vm is
> destroyed. If somebody met the same problem with me , I am do not know if
> this is a bug of ovs-dpdk,or dpdk???
>
>
>
> Thanks
>
> Eric wang
>
>
>
> Wang Huaxia ---JD.COM
>
>
>
> _______________________________________________
> discuss mailing list
> discuss@openvswitch.org
> http://openvswitch.org/mailman/listinfo/discuss
>
>
_______________________________________________
discuss mailing list
discuss@openvswitch.org
http://openvswitch.org/mailman/listinfo/discuss
