Hi Mark,
Thank you for your response.
We are seeing single PMD thread even after setting pmd-cpu-mask to 3 as
below after starting the vswitchd daemon.
ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=3
Our system is a 8 core ARM system. Below are the steps we are running
and the corresponding debug logs.
-----------------------------------------------------------------------------------------------------------------------
root@ls2085ardb:/tmp# mkdir -p /dev/hugepages
root@ls2085ardb:/tmp# mount -t hugetlbfs -o pagesize=1G none /dev/hugepages
root@ls2085ardb:/tmp# mkdir -p /usr/local/etc/openvswitch
root@ls2085ardb:/tmp# mkdir -p /usr/local/var/run/openvswitch
root@ls2085ardb:/tmp# rm /usr/local/etc/openvswitch/conf.db
rm: cannot remove '/usr/local/etc/openvswitch/conf.db': No such file or
directory
root@ls2085ardb:/tmp#/tmp/ovsdb/ovsdb-tool create
/usr/local/etc/openvswitch/conf.db /tmp/vswitchd/vswitch.ovsschema
root@ls2085ardb:/tmp# /tmp/ovsdb/ovsdb-server
--remote=punix:/usr/local/var/run/openvswitch/db.sock
--remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach
--log-file=/var/log/openvswitch/ovs-vswitchd.log
2019-04-02T13:24:30Z|00001|vlog|WARN|failed to open
/var/log/openvswitch/ovs-vswitchd.log for logging: No such file or directory
root@ls2085ardb:/tmp# /tmp/utilities/ovs-vsctl --no-wait init
root@ls2085ardb:/tmp# export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
root@ls2085ardb:/tmp# vswitchd/ovs-vswitchd --dpdk -c 0x1 -n 4 --socket-mem
1024 -- unix:$DB_SOCK --pidfile
--log-file=/var/log/openvswitch/ovs-vswitchd2.log
2019-04-02T13[ 157.753639] Bits 55-60 of /proc/PID/pagemap entries are about
to stop being page-shift some time soon. See the
linux/Documentation/vm/pagemap.txt for details.
:24:43Z|00001|dpdk|INFO|No -vhost_sock_dir provided - defaulting to
/usr/local/var/run/openvswitch
EAL: VFIO support initialized
EAL: cannot open /proc/self/numa_maps, consider that all memory is in socket_id 0
Processing Container = dprc.2
container device path = /sys/bus/fsl-mc/devices/dprc.2
EAL: DPAA2-Unused container at index 0
-->Initial SHM Virtual ADDR FFFD80000000
-----> DMA size 0x40000000
-----> dma_map.vaddr = 0xFFFD80000000
Zone 0: name:<RG_MP_log_history>, phys:0x81bfffd640, len:0x2080,
virt:0xfffdbfffd640, socket_id:0, flags:0
Zone 1: name:<MP_log_history>, phys:0x81bfedd2c0, len:0x1202c0,
virt:0xfffdbfedd2c0, socket_id:0, flags:0
Zone 2: name:<rte_eth_dev_data>, phys:0x81bfeada80, len:0x2f800,
virt:0xfffdbfeada80, socket_id:0, flags:0
2019-04-02T13:24:47Z|00002|vlog|WARN|failed to open
/var/log/openvswitch/ovs-vswitchd2.log for logging: No such file or directory
changing path******************
2019-04-02T13:24:47Z|00003|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
connecting...
2019-04-02T13:24:47Z|00004|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
connected
Discovered 1 CPU cores on NUMA node 0
Discovered 1 NUMA nodes and 1 CPU cores
2019-04-02T13:24:47Z|00005|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.5.0
root@ls2085ardb:/tmp#/tmp/utilities/ovs-vsctl set Open_vSwitch .
other_config:pmd-cpu-mask=3
2019-04-02T13:24:59Z|00006|memory|INFO|4492 kB peak resident set size after
16.5 seconds
root@ls2085ardb:/tmp#/tmp/utilities/ovs-vsctl add-br br0 -- set bridge br0
datapath_type=netdev
[ 191.114980] device ovs-netdev entered promiscuous mode
[ 191.126539] device br0 entered promiscuous mode
2019-04-02T13:25:16Z|00007|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath
supports recirculation
2019-04-02T13:25:16Z|00008|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS label
stack length probed as 3
2019-04-02T13:25:16Z|00009|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath
supports unique flow ids
2019-04-02T13:25:16Z|00010|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does
not support ct_state
2019-04-02T13:25:16Z|00011|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does
not support ct_zone
2019-04-02T13:25:16Z|00012|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does
not support ct_mark
2019-04-02T13:25:16Z|00013|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does
not support ct_label
2019-04-02T13:25:16Z|00014|bridge|INFO|bridge br0: added interface br0 on port
65534
2019-04-02T13:25:16Z|00015|dpif_netlink|ERR|Generic Netlink family
'ovs_datapath' does not exist. The Open vSwitch kernel module is probably not
loaded.
2019-04-02T13:25:16Z|00016|bridge|INFO|bridge br0: using datapath ID
00005e5d815f8145
2019-04-02T13:25:16Z|00017|netdev_linux|WARN|query br0 qdisc failed (Operation
not supported)
2019-04-02T13:25:16Z|00018|netdev_linux|WARN|br0: removing policing failed:
Operation not supported
2019-04-02T13:25:16Z|00019|connmgr|INFO|br0: added service controller
"punix:/usr/local/var/run/openvswitch/br0.mgmt"
2019-04-02T13:25:16Z|00020|netdev_linux|WARN|br0: removing policing failed:
Operation not supported
root@ls2085ardb:/tmp#/tmp/utilities/ovs-vsctl add-port br0 dpdk0 -- set
Interface dpdk0 type=dpdk
2019-04-02T13:25:26Z|00021|memory|INFO|peak resident set size grew 322% in last
26.9 seconds, from 4492 kB to 18944 kB
2019-04-02T13:25:26Z|00022|memory|INFO|handlers:5 ports:1 revalidators:3 rules:5
netdev_dpdk_construct:817, netdev 0xfffdbfeab240, port name dpdk0, port number
is 0
netdev_dpdk_init:677, type is DPDK_DEV_ETH, sid 0
dpdk_eth_dev_queue_setup:492, rte_eth_dev_configured with port_id 0, n_rxq 1,
n_txq 1
2019-04-02T13:25:32Z|00023|dpdk|INFO|Port 0: 00:00:00:00:00:01
do_add_port:1158, netdev_is_pmd, numa_id 0
do_add_port:1162, netdev_n_rxq(netdev) 1
do_add_port:1165, pmd (nil)
dp_netdev_set_pmds_on_numa:3082, can_have 1, dp->pmd_cmask 3d1d4f60, n_unpinned
1, NR_PMD_THREADS 2
2019-04-02T13:25:32Z|00024|dpif_netdev|INFO|Created 1 pmd threads on numa node 0
2019-04-02T13:25:32Z|00025|bridge|INFO|bridge br0: added interface dpdk0 on
port 1
2019-04-02T13:25:32Z|00026|bridge|INFO|bridge br0: using datapath ID
0000000000000001
2019-04-02T13:25:32Z|00027|netdev_linux|WARN|br0: removing policing failed:
Operation not supported
2019-04-02T13:25:32Z|00028|netdev_linux|WARN|br0: removing policing failed:
Operation not supported
2019-04-02T13:25:32Z|00001|dpif_netdev(pmd20)|INFO|Core 0 processing port
'dpdk0'
root@ls2085ardb:/tmp#/tmp/utilities/ovs-vsctl add-port br0 dpdk1 -- set
Interface dpdk1 type=dpdk
netdev_dpdk_construct:817, netdev 0xfffd97890540, port name dpdk1, port number
is 1
dpdk_eth_dev_queue_setup:492, rte_eth_dev_configured with port_id 1, n_rxq 1,
n_txq 1
2019-04-02T13:25:54Z|00029|dpdk|INFO|Port 1: 00:00:00:00:00:02
do_add_port:1158, netdev_is_pmd, numa_id 0
do_add_port:1162, netdev_n_rxq(netdev) 1
do_add_port:1165, pmd 0x3d1dd2f0
2019-04-02T13:25:54Z|00002|dpif_netdev(pmd20)|INFO|Core 0 processing port
'dpdk0'
2019-04-02T13:25:54Z|00003|dpif_netdev(pmd20)|INFO|Core 0 processing port
'dpdk1'
2019-04-02T13:25:54Z|00030|bridge|INFO|bridge br0: added interface dpdk1 on
port 2
2019-04-02T13:25:54Z|00031|netdev_linux|WARN|br0: removing policing failed:
Operation not supported
----------------------------------------------------------------------------------------------------------------------------------------------------------
Could you please let us know, if we are missing anything.
Regards,
Bhanu.
From: Kavanagh, Mark B [mailto:mark.b.kavan...@intel.com]
Sent: Tuesday, May 17, 2016 9:14 PM
To: Bhanu Chander Gaddoju; discuss@openvswitch.org
Cc: Prasad Gorja
Subject: RE: Multiple PMD threads for DPDK ports
Hi Bhanu,
You can use the pmd-cpu-mask to create multiple PMDs.
Open http://openvswitch.org/ovs-vswitchd.conf.db.5.pdf and search for
'pmd-cpu-mask'.
Cheers,
Mark
From: discuss [mailto:discuss-boun...@openvswitch.org] On Behalf Of Bhanu
Chander Gaddoju
Sent: Tuesday, May 17, 2016 7:21 AM
To: discuss@openvswitch.org<mailto:discuss@openvswitch.org>
Cc: Prasad Gorja
Subject: [ovs-discuss] Multiple PMD threads for DPDK ports
Hi All,
I am creating a OVS bridge and attaching two dpdk ports(dpdk0 and
dpdk1) to that bridge. I am using OVS version 2.5. After attaching the two
ports to the bridge, I am observing that only one PMD thread is getting created
for the two dpdk ports.
Based on the OVS code, I understand that, OVS is finding the NUMA
nodes and cores during initialization and it is creating PMD threads based on
the number of cores of NUMA nodes. In my test case, I am seeing the number of
numa nodes as 1 and cores as 1. is it possible in OVS to create a PMD thread
per dpdk port?
If possible, I would like to know how can I create two PMD threads in
this case. Could anyone help me on this.
Regards,
Bhanu.
_______________________________________________
discuss mailing list
discuss@openvswitch.org
http://openvswitch.org/mailman/listinfo/discuss
