Steven,
I managed to get Tx/Rx rings setup with 1GB hugepages. However, when I
assign an IP address to both vhost-user/virtio interfaces and initiate
a ping VPP crashes.
Any other mechanism available to test Tx/Rx path between Vhost and
Virtio? Details below.
*******On host*******
vpp#show vhost-user VirtualEthernet0/0/0
Virtio vhost-user interfaces
Global:
coalesce frames 32 time 1e-3
number of rx virtqueues in interrupt mode: 0
Interface: VirtualEthernet0/0/0 (ifindex 3)
virtio_net_hdr_sz 12
features mask (0xffffffffffffffff):
features (0x110008000):
VIRTIO_NET_F_MRG_RXBUF (15)
VIRTIO_F_INDIRECT_DESC (28)
VIRTIO_F_VERSION_1 (32)
protocol features (0x0)
socket filename /var/run/vpp/sock3.sock type server errno "Success"
rx placement:
thread 0 on vring 1, polling
tx placement: lock-free
thread 0 on vring 0
Memory regions (total 1)
region fd guest_phys_addr memory_size userspace_addr
mmap_offset mmap_addr
====== ===== ================== ================== ==================
================== ==================
0 26 0x00007f54c0000000 0x0000000040000000 0x00007f54c0000000
0x0000000000000000 0x00007faf00000000
Virtqueue 0 (TX)
qsz 256 last_avail_idx 0 last_used_idx 0
avail.flags 1 avail.idx 256 used.flags 1 used.idx 0
kickfd 27 callfd 24 errfd -1
Virtqueue 1 (RX)
qsz 256 last_avail_idx 0 last_used_idx 0
avail.flags 1 avail.idx 0 used.flags 1 used.idx 0
kickfd 28 callfd 25 errfd -1
vpp#set interface ip address VirtualEthernet0/0/0 10.1.1.1/24
************On container**********
vpp# show interface VirtioUser0/0/0
Name Idx State Counter
Count
VirtioUser0/0/0 1 up
vpp#
vpp# set interface ip address VirtioUser0/0/0 10.1.1.2/24
vpp#
vpp# ping 10.1.1.1
Statistics: 5 sent, 0 received, 100% packet loss
vpp#
************Host vpp crash with following backtrace******************
Continuing.
Program received signal SIGSEGV, Segmentation fault.
octeontx_fpa_bufpool_alloc (handle=0)
at
/var/venom/rk-vpp-1804/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/drivers/mempool/octeontx/rte_mempool_octeontx.c:57
57 return (void *)(uintptr_t)fpavf_read64((void *)(handle +
(gdb) bt
#0 octeontx_fpa_bufpool_alloc (handle=0)
at
/var/venom/rk-vpp-1804/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/drivers/mempool/octeontx/rte_mempool_octeontx.c:57
#1 octeontx_fpavf_dequeue (mp=0x7fae7fc9ab40, obj_table=0x7fb04d868880, n=528)
at
/var/venom/rk-vpp-1804/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/drivers/mempool/octeontx/rte_mempool_octeontx.c:98
#2 0x00007fb04b73bdef in rte_mempool_ops_dequeue_bulk (n=528,
obj_table=<optimized out>,
mp=0x7fae7fc9ab40)
at
/var/venom/rk-vpp-1804/vpp/build-root/install-vpp-native/dpdk/include/dpdk/rte_mempool.h:492
#3 __mempool_generic_get (cache=<optimized out>, n=<optimized out>,
obj_table=<optimized out>,
mp=<optimized out>)
at
/var/venom/rk-vpp-1804/vpp/build-root/install-vpp-native/dpdk/include/dpdk/rte_mempool.h:1271
#4 rte_mempool_generic_get (cache=<optimized out>, n=<optimized out>,
obj_table=<optimized out>, mp=<optimized out>)
at
/var/venom/rk-vpp-1804/vpp/build-root/install-vpp-native/dpdk/include/dpdk/rte_mempool.h:1306
#5 rte_mempool_get_bulk (n=528, obj_table=<optimized out>, mp=0x7fae7fc9ab40)
at
/var/venom/rk-vpp-1804/vpp/build-root/install-vpp-native/dpdk/include/dpdk/rte_mempool.h:1339
#6 dpdk_buffer_fill_free_list_avx2 (vm=0x7fb08ec69480
<vlib_global_main>, fl=0x7fb04cb2b100,
min_free_buffers=<optimized out>)
at /var/venom/rk-vpp-1804/vpp/build-data/../src/plugins/dpdk/buffer.c:228
#7 0x00007fb08e5046ea in vlib_buffer_alloc_from_free_list (index=0
'\000', n_buffers=514,
buffers=0x7fb04cb8ec58, vm=0x7fb08ec69480 <vlib_global_main>)
at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/buffer_funcs.h:306
#8 vhost_user_if_input (mode=<optimized out>, node=0x7fb04d0f5b80,
qid=<optimized out>,
vui=0x7fb04d87523c, vum=0x7fb08e9b9560 <vhost_user_main>,
vm=0x7fb08ec69480 <vlib_global_main>)
at
/var/venom/rk-vpp-1804/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1644
#9 vhost_user_input (f=<optimized out>, node=<optimized out>,
vm=<optimized out>)
at
/var/venom/rk-vpp-1804/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1947
#10 vhost_user_input_avx2 (vm=<optimized out>, node=<optimized out>,
frame=<optimized out>)
at
/var/venom/rk-vpp-1804/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1972
#11 0x00007fb08ea166b3 in dispatch_node (last_time_stamp=<optimized
out>, frame=0x0,
dispatch_state=VLIB_NODE_STATE_POLLING, type=VLIB_NODE_TYPE_INPUT,
node=0x7fb04d0f5b80,
vm=0x7fb08ec69480 <vlib_global_main>)
at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/main.c:988
#12 vlib_main_or_worker_loop (is_main=1, vm=0x7fb08ec69480 <vlib_global_main>)
at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/main.c:1505
#13 vlib_main_loop (vm=0x7fb08ec69480 <vlib_global_main>)
at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/main.c:1633
#14 vlib_main (vm=vm@entry=0x7fb08ec69480 <vlib_global_main>,
input=input@entry=0x7fb04d077fa0)
at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/main.c:1787
#15 0x00007fb08ea4d683 in thread0 (arg=140396286350464)
at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/unix/main.c:568
#16 0x00007fb08dbe15d8 in clib_calljmp ()
at /var/venom/rk-vpp-1804/vpp/build-data/../src/vppinfra/longjmp.S:110
#17 0x00007fff0726d370 in ?? ()
---Type <return> to continue, or q <return> to quit---
#18 0x00007fb08ea4e3da in vlib_unix_main (argc=<optimized out>,
argv=<optimized out>)
at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/unix/main.c:632
#19 0x0000001900000000 in ?? ()
#20 0x000000e700000000 in ?? ()
#21 0x0000000000000831 in ?? ()
#22 0x00007fb08e9aac00 in ?? () from /usr/lib/x86_64-linux-gnu/libvnet.so.0
**************Vhost-user debugs on host**********
Jun 5 19:23:35 [18916]: vhost_user_socksvr_accept_ready:1294: New
client socket for vhost interface 3, fd 23
Jun 5 19:23:35 [18916]: vhost_user_socket_read:995: if 3 msg
VHOST_USER_SET_OWNER
Jun 5 19:23:35 [18916]: vhost_user_socket_read:847: if 3 msg
VHOST_USER_GET_FEATURES - reply 0x000000015c628000
Jun 5 19:23:35 [18916]: vhost_user_socket_read:995: if 3 msg
VHOST_USER_SET_OWNER
Jun 5 19:23:35 [18916]: vhost_user_socket_read:1004: if 3 msg
VHOST_USER_SET_VRING_CALL 0
Jun 5 19:23:35 [18916]: vhost_user_socket_read:1004: if 3 msg
VHOST_USER_SET_VRING_CALL 1
Jun 5 19:23:35 [18916]: vhost_user_socket_read:852: if 3 msg
VHOST_USER_SET_FEATURES features 0x0000000110008000
Jun 5 19:23:35 [18916]: vhost_user_socket_read:877: if 3 msg
VHOST_USER_SET_MEM_TABLE nregions 1
Jun 5 19:23:35[18916]: vhost_user_socket_read:916: map memory region
0 addr 0 len 0x40000000 fd 26 mapped 0x7faf00000000 page_sz 0x40000000
Jun 5 19:23:35 [18916]: vhost_user_socket_read:932: if 3 msg
VHOST_USER_SET_VRING_NUM idx 0 num 256
Jun 5 19:23:35 [18916]: vhost_user_socket_read:1096: if 3 msg
VHOST_USER_SET_VRING_BASE idx 0 num 0
Jun 5 19:23:35 [18916]: vhost_user_socket_read:943: if 3 msg
VHOST_USER_SET_VRING_ADDR idx 0
Jun 5 19:23:35 [18916]: vhost_user_socket_read:1037: if 3 msg
VHOST_USER_SET_VRING_KICK 0
Jun 5 19:23:35 [18916]: vhost_user_socket_read:932: if 3 msg
VHOST_USER_SET_VRING_NUM idx 1 num 256
Jun 5 19:23:35 [18916]: vhost_user_socket_read:1096: if 3 msg
VHOST_USER_SET_VRING_BASE idx 1 num 0
Jun 5 19:23:35 [18916]: vhost_user_socket_read:943: if 3 msg
VHOST_USER_SET_VRING_ADDR idx 1
Jun 5 19:23:35 [18916]: vhost_user_socket_read:1037: if 3 msg
VHOST_USER_SET_VRING_KICK 1
Jun 5 19:23:35 [18916]: vhost_user_socket_read:1211: if 3
VHOST_USER_SET_VRING_ENABLE: enable queue 0
Jun 5 19:23:35[18916]: vhost_user_socket_read:1211: if 3
VHOST_USER_SET_VRING_ENABLE: enable queue 1
Thanks.
On Tue, Jun 5, 2018 at 11:31 AM, Steven Luong (sluong) <slu...@cisco.com> wrote:
> Ravi,
>
> In order to use dpdk virtio_user, you need 1GB huge page.
>
> Steven
>
> On 6/5/18, 11:17 AM, "Ravi Kerur" <rke...@gmail.com> wrote:
>
> Hi Steven,
>
> Connection is the problem. I don't see memory regions setup correctly.
> Below are some details. Currently I am using 2MB hugepages.
>
> (1) Create vhost-user server
> debug vhost-user on
> vpp# create vhost socket /var/run/vpp/sock3.sock server
> VirtualEthernet0/0/0
> vpp# set interface state VirtualEthernet0/0/0 up
> vpp#
> vpp#
>
> (2) Instantiate a container
> docker run -it --privileged -v
> /var/run/vpp/sock3.sock:/var/run/usvhost1 -v
> /dev/hugepages:/dev/hugepages dpdk-app-vpp:latest
>
> (3) Inside the container run EAL/DPDK virtio with following startup conf.
> unix {
> nodaemon
> log /var/log/vpp/vpp.log
> full-coredump
> cli-listen /run/vpp/cli.sock
> gid vpp
> }
>
> api-trace {
> on
> }
>
> api-segment {
> gid vpp
> }
>
> dpdk {
> no-pci
> vdev virtio_user0,path=/var/run/usvhost1
> }
>
> Following errors are seen due to 2MB hugepages and I think DPDK
> requires "--single-file-segments" option.
>
> /usr/bin/vpp[19]: dpdk_config:1275: EAL init args: -c 1 -n 4 --no-pci
> --vdev virtio_user0,path=/var/run/usvhost1 --huge-dir
> /run/vpp/hugepages --file-prefix vpp --master-lcore 0 --socket-mem
> 64,64
> /usr/bin/vpp[19]: dpdk_config:1275: EAL init args: -c 1 -n 4 --no-pci
> --vdev virtio_user0,path=/var/run/usvhost1 --huge-dir
> /run/vpp/hugepages --file-prefix vpp --master-lcore 0 --socket-mem
> 64,64
> EAL: 4 hugepages of size 1073741824 reserved, but no mounted hugetlbfs
> found for that size
> EAL: VFIO support initialized
> get_hugepage_file_info(): Exceed maximum of 8
> prepare_vhost_memory_user(): Failed to prepare memory for vhost-user
> DPDK physical memory layout:
>
>
> Second test case>
> (1) and (2) are same as above. I run VPP inside a container with
> following startup config
>
> unix {
> nodaemon
> log /var/log/vpp/vpp.log
> full-coredump
> cli-listen /run/vpp/cli.sock
> gid vpp
> }
>
> api-trace {
> on
> }
>
> api-segment {
> gid vpp
> }
>
> dpdk {
> no-pci
> single-file-segments
> vdev virtio_user0,path=/var/run/usvhost1
> }
>
>
> VPP fails to start with
> plugin.so
> vpp[19]: dpdk_config: unknown input `single-file-segments no-pci vd...'
> vpp[19]: dpdk_config: unknown input `single-file-segments no-pci vd...'
>
> [1]+ Done /usr/bin/vpp -c /etc/vpp/startup.conf
> root@867dc128b544:~/dpdk#
>
>
> show version (on both host and container).
> vpp v18.04-rc2~26-gac2b736~b45 built by root on 34a554d1c194 at Wed
> Apr 25 14:53:07 UTC 2018
> vpp#
>
> Thanks.
>
> On Tue, Jun 5, 2018 at 9:23 AM, Steven Luong (sluong) <slu...@cisco.com>
> wrote:
> > Ravi,
> >
> > Do this
> >
> > 1. Run VPP native vhost-user in the host. Turn on debug "debug
> vhost-user on".
> > 2. Bring up the container with the vdev virtio_user commands that you
> have as before
> > 3. show vhost-user in the host and verify that it has a shared memory
> region. If not, the connection has a problem. Collect the show vhost-user and
> debug vhost-user and send them to me and stop. If yes, proceed with step 4.
> > 4. type "trace vhost-user-input 100" in the host
> > 5. clear error, and clear interfaces in the host and the container.
> > 6. do the ping from the container.
> > 7. Collect show error, show trace, show interface, and show vhost-user
> in the host. Collect show error and show interface in the container. Put
> output in github and provide a link to view. There is no need to send a large
> file.
> >
> > Steven
> >
> > On 6/4/18, 5:50 PM, "Ravi Kerur" <rke...@gmail.com> wrote:
> >
> > Hi Steven,
> >
> > Thanks for your help. I am using vhost-user client (VPP in
> container)
> > and vhost-user server (VPP in host). I thought it should work.
> >
> > create vhost socket /var/run/vpp/sock3.sock server (On host)
> >
> > create vhost socket /var/run/usvhost1 (On container)
> >
> > Can you please point me to a document which shows how to create VPP
> > virtio_user interfaces or static configuration in
> > /etc/vpp/startup.conf?
> >
> > I have used following declarations in /etc/vpp/startup.conf
> >
> > # vdev
> virtio_user0,path=/var/run/vpp/sock3.sock,mac=52:54:00:00:04:01
> > # vdev
> virtio_user1,path=/var/run/vpp/sock4.sock,mac=52:54:00:00:04:02
> >
> > but it doesn't work.
> >
> > Thanks.
> >
> > On Mon, Jun 4, 2018 at 3:57 PM, Steven Luong (sluong)
> <slu...@cisco.com> wrote:
> > > Ravi,
> > >
> > > VPP only supports vhost-user in the device mode. In your example,
> the host, in device mode, and the container also in device mode do not make a
> happy couple. You need one of them, either the host or container, running in
> driver mode using the dpdk vdev virtio_user command in startup.conf. So you
> need something like this
> > >
> > > (host) VPP native vhost-user ----- (container) VPP DPDK vdev
> virtio_user
> > > -- or --
> > > (host) VPP DPDK vdev virtio_user ---- (container) VPP native
> vhost-user
> > >
> > > Steven
> > >
> > > On 6/4/18, 3:27 PM, "Ravi Kerur" <rke...@gmail.com> wrote:
> > >
> > > Hi Steven
> > >
> > > Though crash is not happening anymore, there is still an
> issue with Rx
> > > and Tx. To eliminate whether it is testpmd or vpp, I decided
> to run
> > >
> > > (1) VPP vhost-user server on host-x
> > > (2) Run VPP in a container on host-x and vhost-user client
> port
> > > connecting to vhost-user server.
> > >
> > > Still doesn't work. Details below. Please let me know if
> something is
> > > wrong in what I am doing.
> > >
> > >
> > > (1) VPP vhost-user as a server
> > > (2) VPP in a container virtio-user or vhost-user client
> > >
> > > (1) Create vhost-user server socket on VPP running on host.
> > >
> > > vpp#create vhost socket /var/run/vpp/sock3.sock server
> > > vpp#set interface state VirtualEthernet0/0/0 up
> > > show vhost-user VirtualEthernet0/0/0 descriptors
> > > Virtio vhost-user interfaces
> > > Global:
> > > coalesce frames 32 time 1e-3
> > > number of rx virtqueues in interrupt mode: 0
> > > Interface: VirtualEthernet0/0/0 (ifindex 3)
> > > virtio_net_hdr_sz 0
> > > features mask (0xffffffffffffffff):
> > > features (0x0):
> > > protocol features (0x0)
> > >
> > > socket filename /var/run/vpp/sock3.sock type server errno
> "Success"
> > >
> > > rx placement:
> > > tx placement: spin-lock
> > > thread 0 on vring 0
> > >
> > > Memory regions (total 0)
> > >
> > > vpp# set interface ip address VirtualEthernet0/0/0
> 192.168.1.1/24
> > > vpp#
> > >
> > > (2) Instantiate a docker container to run VPP connecting to
> sock3.server socket.
> > >
> > > docker run -it --privileged -v
> > > /var/run/vpp/sock3.sock:/var/run/usvhost1 -v
> > > /dev/hugepages:/dev/hugepages dpdk-app-vpp:latest
> > > root@4b1bd06a3225:~/dpdk#
> > > root@4b1bd06a3225:~/dpdk# ps -ef
> > > UID PID PPID C STIME TTY TIME CMD
> > > root 1 0 0 21:39 ? 00:00:00 /bin/bash
> > > root 17 1 0 21:39 ? 00:00:00 ps -ef
> > > root@4b1bd06a3225:~/dpdk#
> > >
> > > root@8efda6701ace:~/dpdk# ps -ef | grep vpp
> > > root 19 1 39 21:41 ? 00:00:03 /usr/bin/vpp -c
> /etc/vpp/startup.conf
> > > root 25 1 0 21:41 ? 00:00:00 grep --color=auto vpp
> > > root@8efda6701ace:~/dpdk#
> > >
> > > vpp#create vhost socket /var/run/usvhost1
> > > vpp#set interface state VirtualEthernet0/0/0 up
> > > vpp#show vhost-user VirtualEthernet0/0/0 descriptors
> > > Virtio vhost-user interfaces
> > > Global:
> > > coalesce frames 32 time 1e-3
> > > number of rx virtqueues in interrupt mode: 0
> > > Interface: VirtualEthernet0/0/0 (ifindex 1)
> > > virtio_net_hdr_sz 0
> > > features mask (0xffffffffffffffff):
> > > features (0x0):
> > > protocol features (0x0)
> > >
> > > socket filename /var/run/usvhost1 type client errno "Success"
> > >
> > > rx placement:
> > > tx placement: spin-lock
> > > thread 0 on vring 0
> > >
> > > Memory regions (total 0)
> > >
> > > vpp#
> > >
> > > vpp# set interface ip address VirtualEthernet0/0/0
> 192.168.1.2/24
> > > vpp#
> > >
> > > vpp# ping 192.168.1.1
> > >
> > > Statistics: 5 sent, 0 received, 100% packet loss
> > > vpp#
> > >
> > > On Thu, May 31, 2018 at 2:30 PM, Steven Luong (sluong)
> <slu...@cisco.com> wrote:
> > > > show interface and look for the counter and count columns
> for the corresponding interface.
> > > >
> > > > Steven
> > > >
> > > > On 5/31/18, 1:28 PM, "Ravi Kerur" <rke...@gmail.com> wrote:
> > > >
> > > > Hi Steven,
> > > >
> > > > You made my day, thank you. I didn't realize different
> dpdk versions
> > > > (vpp -- 18.02.1 and testpmd -- from latest git repo
> (probably 18.05)
> > > > could be the cause of the problem, I still dont
> understand why it
> > > > should as virtio/vhost messages are meant to setup
> tx/rx rings
> > > > correctly?
> > > >
> > > > I downloaded dpdk 18.02.1 stable release and at least
> vpp doesn't
> > > > crash now (for both vpp-native and dpdk vhost
> interfaces). I have one
> > > > question is there a way to read vhost-user statistics
> counter (Rx/Tx)
> > > > on vpp? I only know
> > > >
> > > > 'show vhost-user <intf>' and 'show vhost-user <intf>
> descriptors'
> > > > which doesn't show any counters.
> > > >
> > > > Thanks.
> > > >
> > > > On Thu, May 31, 2018 at 11:51 AM, Steven Luong (sluong)
> > > > <slu...@cisco.com> wrote:
> > > > > Ravi,
> > > > >
> > > > > For (1) which works, what dpdk version are you using
> in the host? Are you using the same dpdk version as VPP is using? Since you
> are using VPP latest, I think it is 18.02. Type "show dpdk version" at the
> VPP prompt to find out for sure.
> > > > >
> > > > > Steven
> > > > >
> > > > > On 5/31/18, 11:44 AM, "Ravi Kerur" <rke...@gmail.com>
> wrote:
> > > > >
> > > > > Hi Steven,
> > > > >
> > > > > i have tested following scenarios and it
> basically is not clear why
> > > > > you think DPDK is the problem? Is it possible VPP
> and DPDK use
> > > > > different virtio versions?
> > > > >
> > > > > Following are the scenarios I have tested
> > > > >
> > > > > (1) testpmd/DPDK vhost-user (running on host) and
> testpmd/DPDK
> > > > > virito-user (in a container) -- can send and
> receive packets
> > > > > (2) VPP-native vhost-user (running on host) and
> testpmd/DPDK
> > > > > virtio-user (in a container) -- VPP crashes and
> it is in VPP code
> > > > > (3) VPP-DPDK vhost user (running on host) and
> testpmd/DPDK virtio-user
> > > > > (in a container) -- VPP crashes and in DPDK
> > > > >
> > > > > Thanks.
> > > > >
> > > > > On Thu, May 31, 2018 at 10:12 AM, Steven Luong
> (sluong)
> > > > > <slu...@cisco.com> wrote:
> > > > > > Ravi,
> > > > > >
> > > > > > I've proved my point -- there is a problem in
> the way that you invoke testpmd. The shared memory region that it passes to
> the device is not accessible from the device. I don't know what the correct
> options are that you need to use. This is really a question for dpdk.
> > > > > >
> > > > > > As a further exercise, you could remove VPP in
> the host and instead run testpmd in device mode using "--vdev
> net_vhost0,iface=/var/run/vpp/sock1.sock" option. I bet you testpmd in the
> host will crash in the same place. I hope you can find out the answer from
> dpdk and tell us about it.
> > > > > >
> > > > > > Steven
> > > > > >
> > > > > > On 5/31/18, 9:31 AM, "vpp-dev@lists.fd.io on
> behalf of Ravi Kerur" <vpp-dev@lists.fd.io on behalf of rke...@gmail.com>
> wrote:
> > > > > >
> > > > > > Hi Steven,
> > > > > >
> > > > > > Thank you for your help, I removed
> sock1.sock and sock2.sock,
> > > > > > restarted vpp, atleast interfaces get
> created. However, when I start
> > > > > > dpdk/testpmd inside the container it
> crashes as well. Below are some
> > > > > > details. I am using vpp code from latest
> repo.
> > > > > >
> > > > > > (1) On host
> > > > > > show interface
> > > > > > Name Idx
> State Counter
> > > > > > Count
> > > > > > VhostEthernet2 3
> down
> > > > > > VhostEthernet3 4
> down
> > > > > > VirtualFunctionEthernet4/10/4 1
> down
> > > > > > VirtualFunctionEthernet4/10/6 2
> down
> > > > > > local0 0
> down
> > > > > > vpp#
> > > > > > vpp# set interface state VhostEthernet2 up
> > > > > > vpp# set interface state VhostEthernet3 up
> > > > > > vpp#
> > > > > > vpp# set interface l2 bridge VhostEthernet2
> 1
> > > > > > vpp# set interface l2 bridge VhostEthernet3
> 1
> > > > > > vpp#
> > > > > >
> > > > > > (2) Run tespmd inside the container
> > > > > > docker run -it --privileged -v
> > > > > > /var/run/vpp/sock1.sock:/var/run/usvhost1 -v
> > > > > > /var/run/vpp/sock2.sock:/var/run/usvhost2 -v
> > > > > > /dev/hugepages:/dev/hugepages
> dpdk-app-testpmd ./bin/testpmd -l 16-19
> > > > > > -n 4 --log-level=8 -m 64 --no-pci
> > > > > >
> --vdev=virtio_user0,path=/var/run/usvhost1,mac=54:00:00:01:01:01
> > > > > >
> --vdev=virtio_user1,path=/var/run/usvhost2,mac=54:00:00:01:01:02 --
> > > > > > -i
> > > > > > EAL: Detected 28 lcore(s)
> > > > > > EAL: Detected 2 NUMA nodes
> > > > > > EAL: Multi-process socket
> /var/run/dpdk/rte/mp_socket
> > > > > > EAL: 8192 hugepages of size 2097152
> reserved, but no mounted hugetlbfs
> > > > > > found for that size
> > > > > > EAL: Probing VFIO support...
> > > > > > EAL: VFIO support initialized
> > > > > > EAL: Setting up physically contiguous
> memory...
> > > > > > EAL: locking hot plug lock memory...
> > > > > > EAL: primary init32...
> > > > > > Interactive-mode selected
> > > > > > Warning: NUMA should be configured manually
> by using
> > > > > > --port-numa-config and --ring-numa-config
> parameters along with
> > > > > > --numa.
> > > > > > testpmd: create a new mbuf pool
> <mbuf_pool_socket_0>: n=171456,
> > > > > > size=2176, socket=0
> > > > > > testpmd: preferred mempool ops selected:
> ring_mp_mc
> > > > > > testpmd: create a new mbuf pool
> <mbuf_pool_socket_1>: n=171456,
> > > > > > size=2176, socket=1
> > > > > > testpmd: preferred mempool ops selected:
> ring_mp_mc
> > > > > > Port 0 is now not stopped
> > > > > > Port 1 is now not stopped
> > > > > > Please stop the ports first
> > > > > > Done
> > > > > > testpmd>
> > > > > >
> > > > > > (3) VPP crashes with the same issue but
> inside dpdk code
> > > > > >
> > > > > > (gdb) cont
> > > > > > Continuing.
> > > > > >
> > > > > > Program received signal SIGSEGV,
> Segmentation fault.
> > > > > > [Switching to Thread 0x7ffd0d08e700 (LWP
> 41257)]
> > > > > > rte_vhost_dequeue_burst (vid=<optimized
> out>, queue_id=<optimized
> > > > > > out>, mbuf_pool=0x7fe17fc883c0,
> > > > > > pkts=pkts@entry=0x7fffb671ebc0,
> count=count@entry=32)
> > > > > > at
> /var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_vhost/virtio_net.c:1504
> > > > > > 1504 free_entries = *((volatile
> uint16_t *)&vq->avail->idx) -
> > > > > > (gdb) bt
> > > > > > #0 rte_vhost_dequeue_burst (vid=<optimized
> out>, queue_id=<optimized out>,
> > > > > > mbuf_pool=0x7fe17fc883c0,
> pkts=pkts@entry=0x7fffb671ebc0,
> > > > > > count=count@entry=32)
> > > > > > at
> /var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_vhost/virtio_net.c:1504
> > > > > > #1 0x00007fffb4718e6f in eth_vhost_rx
> (q=0x7fe17fbbdd80, bufs=0x7fffb671ebc0,
> > > > > > nb_bufs=<optimized out>)
> > > > > > at
> /var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/drivers/net/vhost/rte_eth_vhost.c:410
> > > > > > #2 0x00007fffb441cb7c in rte_eth_rx_burst
> (nb_pkts=256,
> > > > > > rx_pkts=0x7fffb671ebc0, queue_id=0,
> > > > > > port_id=3) at
> > > > > >
> /var/venom/vpp/build-root/install-vpp-native/dpdk/include/dpdk/rte_ethdev.h:3635
> > > > > > #3 dpdk_device_input (queue_id=0,
> thread_index=<optimized out>,
> > > > > > node=0x7fffb732c700,
> > > > > > xd=0x7fffb7337240, dm=<optimized out>,
> vm=0x7fffb6703340)
> > > > > > at
> /var/venom/vpp/build-data/../src/plugins/dpdk/device/node.c:477
> > > > > > #4 dpdk_input_node_fn_avx2 (vm=<optimized
> out>, node=<optimized out>,
> > > > > > f=<optimized out>)
> > > > > > at
> /var/venom/vpp/build-data/../src/plugins/dpdk/device/node.c:658
> > > > > > #5 0x00007ffff7954d35 in dispatch_node
> > > > > > (last_time_stamp=12531752723928016,
> frame=0x0,
> > > > > > dispatch_state=VLIB_NODE_STATE_POLLING,
> type=VLIB_NODE_TYPE_INPUT,
> > > > > > node=0x7fffb732c700,
> > > > > > vm=0x7fffb6703340) at
> /var/venom/vpp/build-data/../src/vlib/main.c:988
> > > > > > #6 vlib_main_or_worker_loop (is_main=0,
> vm=0x7fffb6703340)
> > > > > > at
> /var/venom/vpp/build-data/../src/vlib/main.c:1507
> > > > > > #7 vlib_worker_loop (vm=0x7fffb6703340) at
> > > > > >
> /var/venom/vpp/build-data/../src/vlib/main.c:1641
> > > > > > #8 0x00007ffff6ad25d8 in clib_calljmp ()
> > > > > > at
> /var/venom/vpp/build-data/../src/vppinfra/longjmp.S:110
> > > > > > #9 0x00007ffd0d08ddb0 in ?? ()
> > > > > > #10 0x00007fffb4436edd in eal_thread_loop
> (arg=<optimized out>)
> > > > > > at
> /var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_eal/linuxapp/eal/eal_thread.c:153
> > > > > > #11 0x0000000000000000 in ?? ()
> > > > > > (gdb) frame 0
> > > > > > #0 rte_vhost_dequeue_burst (vid=<optimized
> out>, queue_id=<optimized out>,
> > > > > > mbuf_pool=0x7fe17fc883c0,
> pkts=pkts@entry=0x7fffb671ebc0,
> > > > > > count=count@entry=32)
> > > > > > at
> /var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_vhost/virtio_net.c:1504
> > > > > > 1504 free_entries = *((volatile
> uint16_t *)&vq->avail->idx) -
> > > > > > (gdb) p vq
> > > > > > $1 = (struct vhost_virtqueue *)
> 0x7fc3ffc84b00
> > > > > > (gdb) p vq->avail
> > > > > > $2 = (struct vring_avail *) 0x7ffbfff98000
> > > > > > (gdb) p *$2
> > > > > > Cannot access memory at address
> 0x7ffbfff98000
> > > > > > (gdb)
> > > > > >
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > On Thu, May 31, 2018 at 12:09 AM, Steven
> Luong (sluong)
> > > > > > <slu...@cisco.com> wrote:
> > > > > > > Sorry, I was expecting to see two
> VhostEthernet interfaces like this. Those VirtualFunctionEthernet are your
> physical interfaces.
> > > > > > >
> > > > > > > sh int
> > > > > > > Name Idx
> State Counter Count
> > > > > > > VhostEthernet0 1
> up
> > > > > > > VhostEthernet1 2
> up
> > > > > > > local0 0
> down
> > > > > > > DBGvpp#
> > > > > > >
> > > > > > > You have to first manually remove
> /var/run/vpp/sock1.sock and /var/run/vpp/sock2.sock before you start vpp on
> the host. dpdk does not like it if they already existed. If you successfully
> create VhostEthernet interface, try to send some traffic through it to see if
> it crashes or not.
> > > > > > >
> > > > > > > Steven
> > > > > > >
> > > > > > > On 5/30/18, 9:17 PM, "vpp-dev@lists.fd.io
> on behalf of Steven Luong (sluong)" <vpp-dev@lists.fd.io on behalf of
> slu...@cisco.com> wrote:
> > > > > > >
> > > > > > > Ravi,
> > > > > > >
> > > > > > > I don't think you can declare (2)
> works fine yet. Please bring up the dpdk vhost-user interfaces and try to
> send some traffic between them to exercise the shared memory region from dpdk
> virtio-user which may be "questionable".
> > > > > > >
> > > > > > > VirtualFunctionEthernet4/10/4
> 1 down
> > > > > > > VirtualFunctionEthernet4/10/6
> 2 down
> > > > > > >
> > > > > > > Steven
> > > > > > >
> > > > > > > On 5/30/18, 4:41 PM, "Ravi Kerur"
> <rke...@gmail.com> wrote:
> > > > > > >
> > > > > > > Hi Steve,
> > > > > > >
> > > > > > > Thank you for your inputs, I
> added feature-mask to see if it helps in
> > > > > > > setting up queues correctly, it
> didn't so I will remove it. I have
> > > > > > > tried following combination
> > > > > > >
> > > > > > > (1) VPP->vhost-user (on host) and
> DPDK/testpmd->virtio-user (in a
> > > > > > > container) -- VPP crashes
> > > > > > > (2) DPDK/testpmd->vhost-user (on
> host) and DPDK/testpmd->virtio-user
> > > > > > > (in a container) -- works fine
> > > > > > >
> > > > > > > To use DPDK vhost-user inside
> VPP, I defined configuration in
> > > > > > > startup.conf as mentioned by you
> and it looks as follows
> > > > > > >
> > > > > > > unix {
> > > > > > > nodaemon
> > > > > > > log /var/log/vpp/vpp.log
> > > > > > > full-coredump
> > > > > > > cli-listen /run/vpp/cli.sock
> > > > > > > gid vpp
> > > > > > > }
> > > > > > >
> > > > > > > api-segment {
> > > > > > > gid vpp
> > > > > > > }
> > > > > > >
> > > > > > > cpu {
> > > > > > > main-core 1
> > > > > > > corelist-workers 6-9
> > > > > > > }
> > > > > > >
> > > > > > > dpdk {
> > > > > > > dev 0000:04:10.4
> > > > > > > dev 0000:04:10.6
> > > > > > > uio-driver vfio-pci
> > > > > > > vdev
> net_vhost0,iface=/var/run/vpp/sock1.sock
> > > > > > > vdev
> net_vhost1,iface=/var/run/vpp/sock2.sock
> > > > > > > huge-dir
> /dev/hugepages_1GB
> > > > > > > socket-mem 2048,2048
> > > > > > > }
> > > > > > >
> > > > > > > From VPP logs
> > > > > > > dpdk: EAL init args: -c 3c2 -n 4
> --vdev
> > > > > > >
> net_vhost0,iface=/var/run/vpp/sock1.sock --vdev
> > > > > > >
> net_vhost1,iface=/var/run/vpp/sock2.sock --huge-dir /dev/hugepages_1GB
> > > > > > > -w 0000:04:10.4 -w 0000:04:10.6
> --master-lcore 1 --socket-mem
> > > > > > > 2048,2048
> > > > > > >
> > > > > > > However, VPP doesn't create
> interface at all
> > > > > > >
> > > > > > > vpp# show interface
> > > > > > > Name
> Idx State Counter
> > > > > > > Count
> > > > > > > VirtualFunctionEthernet4/10/4
> 1 down
> > > > > > > VirtualFunctionEthernet4/10/6
> 2 down
> > > > > > > local0
> 0 down
> > > > > > >
> > > > > > > since it is a static mapping I am
> assuming it should be created, correct?
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > On Wed, May 30, 2018 at 3:43 PM,
> Steven Luong (sluong) <slu...@cisco.com> wrote:
> > > > > > > > Ravi,
> > > > > > > >
> > > > > > > > First and foremost, get rid of
> the feature-mask option. I don't know what 0x40400000 does for you. If that
> does not help, try testing it with dpdk based vhost-user instead of VPP
> native vhost-user to make sure that they can work well with each other first.
> To use dpdk vhost-user, add a vdev command in the startup.conf for each
> vhost-user device that you have.
> > > > > > > >
> > > > > > > > dpdk { vdev
> net_vhost0,iface=/var/run/vpp/sock1.sock }
> > > > > > > >
> > > > > > > > dpdk based vhost-user interface
> is named VhostEthernet0, VhostEthernet1, etc. Make sure you use the right
> interface name to set the state to up.
> > > > > > > >
> > > > > > > > If dpdk based vhost-user does
> not work with testpmd either, it looks like some problem with the way that
> you invoke testpmd.
> > > > > > > >
> > > > > > > > If dpdk based vhost-user works
> well with the same testpmd device driver and not vpp native vhost-user, I can
> set up something similar to yours to look into it.
> > > > > > > >
> > > > > > > > The device driver, testpmd, is
> supposed to pass the shared memory region to VPP for TX/RX queues. It looks
> like VPP vhost-user might have run into a bump there with using the shared
> memory (txvq->avail).
> > > > > > > >
> > > > > > > > Steven
> > > > > > > >
> > > > > > > > PS. vhost-user is not an
> optimum interface for containers. You may want to look into using memif if
> you don't already know about it.
> > > > > > > >
> > > > > > > >
> > > > > > > > On 5/30/18, 2:06 PM, "Ravi
> Kerur" <rke...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > I am not sure what is wrong
> with the setup or a bug in vpp, vpp
> > > > > > > > crashes with
> vhost<-->virtio communication.
> > > > > > > >
> > > > > > > > (1) Vhost-interfaces are
> created and attached to bridge-domain as follows
> > > > > > > >
> > > > > > > > create vhost socket
> /var/run/vpp/sock1.sock server feature-mask 0x40400000
> > > > > > > > create vhost socket
> /var/run/vpp/sock2.sock server feature-mask 0x40400000
> > > > > > > > set interface state
> VirtualEthernet0/0/0 up
> > > > > > > > set interface state
> VirtualEthernet0/0/1 up
> > > > > > > >
> > > > > > > > set interface l2 bridge
> VirtualEthernet0/0/0 1
> > > > > > > > set interface l2 bridge
> VirtualEthernet0/0/1 1
> > > > > > > >
> > > > > > > >
> > > > > > > > (2) DPDK/testpmd is started
> in a container to talk to vpp/vhost-user
> > > > > > > > interface as follows
> > > > > > > >
> > > > > > > > docker run -it --privileged
> -v
> > > > > > > >
> /var/run/vpp/sock1.sock:/var/run/usvhost1 -v
> > > > > > > >
> /var/run/vpp/sock2.sock:/var/run/usvhost2 -v
> > > > > > > >
> /dev/hugepages:/dev/hugepages dpdk-app-testpmd ./bin/testpmd -c 0x3 -n
> > > > > > > > 4 --log-level=9 -m 64
> --no-pci --single-file-segments
> > > > > > > >
> --vdev=virtio_user0,path=/var/run/usvhost1,mac=54:00:00:01:01:01
> > > > > > > >
> --vdev=virtio_user1,path=/var/run/usvhost2,mac=54:00:00:01:01:02 --
> > > > > > > > -i
> > > > > > > >
> > > > > > > > (3) show vhost-user
> VirtualEthernet0/0/1
> > > > > > > > Virtio vhost-user interfaces
> > > > > > > > Global:
> > > > > > > > coalesce frames 32 time
> 1e-3
> > > > > > > > number of rx virtqueues
> in interrupt mode: 0
> > > > > > > > Interface:
> VirtualEthernet0/0/1 (ifindex 4)
> > > > > > > > virtio_net_hdr_sz 10
> > > > > > > > features mask (0x40400000):
> > > > > > > > features (0x0):
> > > > > > > > protocol features (0x0)
> > > > > > > >
> > > > > > > > socket filename
> /var/run/vpp/sock2.sock type server errno "Success"
> > > > > > > >
> > > > > > > > rx placement:
> > > > > > > > tx placement: spin-lock
> > > > > > > > thread 0 on vring 0
> > > > > > > > thread 1 on vring 0
> > > > > > > > thread 2 on vring 0
> > > > > > > > thread 3 on vring 0
> > > > > > > > thread 4 on vring 0
> > > > > > > >
> > > > > > > > Memory regions (total 1)
> > > > > > > > region fd
> guest_phys_addr memory_size userspace_addr
> > > > > > > > mmap_offset mmap_addr
> > > > > > > > ====== =====
> ================== ================== ==================
> > > > > > > > ==================
> ==================
> > > > > > > > 0 55
> 0x00007ff7c0000000 0x0000000040000000 0x00007ff7c0000000
> > > > > > > > 0x0000000000000000
> 0x00007ffbc0000000
> > > > > > > >
> > > > > > > > vpp# show vhost-user
> VirtualEthernet0/0/0
> > > > > > > > Virtio vhost-user interfaces
> > > > > > > > Global:
> > > > > > > > coalesce frames 32 time
> 1e-3
> > > > > > > > number of rx virtqueues
> in interrupt mode: 0
> > > > > > > > Interface:
> VirtualEthernet0/0/0 (ifindex 3)
> > > > > > > > virtio_net_hdr_sz 10
> > > > > > > > features mask (0x40400000):
> > > > > > > > features (0x0):
> > > > > > > > protocol features (0x0)
> > > > > > > >
> > > > > > > > socket filename
> /var/run/vpp/sock1.sock type server errno "Success"
> > > > > > > >
> > > > > > > > rx placement:
> > > > > > > > tx placement: spin-lock
> > > > > > > > thread 0 on vring 0
> > > > > > > > thread 1 on vring 0
> > > > > > > > thread 2 on vring 0
> > > > > > > > thread 3 on vring 0
> > > > > > > > thread 4 on vring 0
> > > > > > > >
> > > > > > > > Memory regions (total 1)
> > > > > > > > region fd
> guest_phys_addr memory_size userspace_addr
> > > > > > > > mmap_offset mmap_addr
> > > > > > > > ====== =====
> ================== ================== ==================
> > > > > > > > ==================
> ==================
> > > > > > > > 0 51
> 0x00007ff7c0000000 0x0000000040000000 0x00007ff7c0000000
> > > > > > > > 0x0000000000000000
> 0x00007ffc00000000
> > > > > > > >
> > > > > > > > (4) vpp stack trace
> > > > > > > > Program received signal
> SIGSEGV, Segmentation fault.
> > > > > > > > [Switching to Thread
> 0x7ffd0e090700 (LWP 46570)]
> > > > > > > > 0x00007ffff7414642 in
> vhost_user_if_input
> > > > > > > >
> (mode=VNET_HW_INTERFACE_RX_MODE_POLLING,
> > > > > > > > node=0x7fffb76bab00,
> qid=<optimized out>, vui=0x7fffb6739700,
> > > > > > > > vum=0x7ffff78f4480
> <vhost_user_main>, vm=0x7fffb672a9c0)
> > > > > > > > at
> /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1596
> > > > > > > > 1596 if (PREDICT_FALSE
> (txvq->avail->flags & 0xFFFE))
> > > > > > > > (gdb) bt
> > > > > > > > #0 0x00007ffff7414642 in
> vhost_user_if_input
> > > > > > > >
> (mode=VNET_HW_INTERFACE_RX_MODE_POLLING,
> > > > > > > > node=0x7fffb76bab00,
> qid=<optimized out>, vui=0x7fffb6739700,
> > > > > > > > vum=0x7ffff78f4480
> <vhost_user_main>, vm=0x7fffb672a9c0)
> > > > > > > > at
> /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1596
> > > > > > > > #1 vhost_user_input
> (f=<optimized out>, node=<optimized out>,
> > > > > > > > vm=<optimized out>)
> > > > > > > > at
> /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1947
> > > > > > > > #2 vhost_user_input_avx2
> (vm=<optimized out>, node=<optimized out>,
> > > > > > > > frame=<optimized out>)
> > > > > > > > at
> /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1972
> > > > > > > > #3 0x00007ffff7954d35 in
> dispatch_node
> > > > > > > >
> (last_time_stamp=12391212490024174, frame=0x0,
> > > > > > > >
> dispatch_state=VLIB_NODE_STATE_POLLING, type=VLIB_NODE_TYPE_INPUT,
> > > > > > > > node=0x7fffb76bab00,
> > > > > > > > vm=0x7fffb672a9c0) at
> /var/venom/vpp/build-data/../src/vlib/main.c:988
> > > > > > > > #4
> vlib_main_or_worker_loop (is_main=0, vm=0x7fffb672a9c0)
> > > > > > > > at
> /var/venom/vpp/build-data/../src/vlib/main.c:1507
> > > > > > > > #5 vlib_worker_loop
> (vm=0x7fffb672a9c0) at
> > > > > > > >
> /var/venom/vpp/build-data/../src/vlib/main.c:1641
> > > > > > > > #6 0x00007ffff6ad25d8 in
> clib_calljmp ()
> > > > > > > > at
> /var/venom/vpp/build-data/../src/vppinfra/longjmp.S:110
> > > > > > > > #7 0x00007ffd0e08fdb0 in
> ?? ()
> > > > > > > > #8 0x00007fffb4436edd in
> eal_thread_loop (arg=<optimized out>)
> > > > > > > > at
> /var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_eal/linuxapp/eal/eal_thread.c:153
> > > > > > > > #9 0x0000000000000000 in
> ?? ()
> > > > > > > > (gdb) frame 0
> > > > > > > > #0 0x00007ffff7414642 in
> vhost_user_if_input
> > > > > > > >
> (mode=VNET_HW_INTERFACE_RX_MODE_POLLING,
> > > > > > > > node=0x7fffb76bab00,
> qid=<optimized out>, vui=0x7fffb6739700,
> > > > > > > > vum=0x7ffff78f4480
> <vhost_user_main>, vm=0x7fffb672a9c0)
> > > > > > > > at
> /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1596
> > > > > > > > 1596 if (PREDICT_FALSE
> (txvq->avail->flags & 0xFFFE))
> > > > > > > > (gdb) p txvq
> > > > > > > > $1 = (vhost_user_vring_t *)
> 0x7fffb6739ac0
> > > > > > > > (gdb) p *txvq
> > > > > > > > $2 = {cacheline0 =
> 0x7fffb6739ac0 "?", qsz_mask = 255, last_avail_idx
> > > > > > > > = 0, last_used_idx = 0,
> > > > > > > > n_since_last_int = 0,
> desc = 0x7ffbfff97000, avail = 0x7ffbfff98000,
> > > > > > > > used = 0x7ffbfff99000,
> > > > > > > > int_deadline = 0, started
> = 1 '\001', enabled = 0 '\000', log_used = 0 '\000',
> > > > > > > > cacheline1 =
> 0x7fffb6739b00 "????\n", errfd = -1, callfd_idx = 10,
> > > > > > > > kickfd_idx = 14,
> > > > > > > > log_guest_addr = 0, mode
> = 1}
> > > > > > > > (gdb) p *(txvq->avail)
> > > > > > > > Cannot access memory at
> address 0x7ffbfff98000
> > > > > > > > (gdb)
> > > > > > > >
> > > > > > > > On Tue, May 29, 2018 at
> 10:47 AM, Ravi Kerur <rke...@gmail.com> wrote:
> > > > > > > > > Steve,
> > > > > > > > >
> > > > > > > > > Thanks for inputs on
> debugs and gdb. I am using gdb on my development
> > > > > > > > > system to debug the
> issue. I would like to have reliable core
> > > > > > > > > generation on the system
> on which I don't have access to install gdb.
> > > > > > > > > I installed corekeeper
> and it still doesn't generate core. I am
> > > > > > > > > running vpp inside a VM
> (VirtualBox/vagrant), not sure if I need to
> > > > > > > > > set something inside
> vagrant config file.
> > > > > > > > >
> > > > > > > > > dpkg -l corekeeper
> > > > > > > > >
> Desired=Unknown/Install/Remove/Purge/Hold
> > > > > > > > > |
> Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> > > > > > > > > |/
> Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> > > > > > > > > ||/ Name
> Version Architecture Description
> > > > > > > > >
> +++-====================-===============-===============-==============================================
> > > > > > > > > ii corekeeper
> 1.6 amd64 enable core
> > > > > > > > > files and report crashes
> to the system
> > > > > > > > >
> > > > > > > > > Thanks.
> > > > > > > > >
> > > > > > > > > On Tue, May 29, 2018 at
> 9:38 AM, Steven Luong (sluong) <slu...@cisco.com> wrote:
> > > > > > > > >> Ravi,
> > > > > > > > >>
> > > > > > > > >> I install corekeeper and
> the core file is kept in /var/crash. But why not use gdb to attach to the VPP
> process?
> > > > > > > > >> To turn on VPP
> vhost-user debug, type "debug vhost-user on" at the VPP prompt.
> > > > > > > > >>
> > > > > > > > >> Steven
> > > > > > > > >>
> > > > > > > > >> On 5/29/18, 9:10 AM,
> "vpp-dev@lists.fd.io on behalf of Ravi Kerur" <vpp-dev@lists.fd.io on behalf
> of rke...@gmail.com> wrote:
> > > > > > > > >>
> > > > > > > > >> Hi Marco,
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Tue, May 29, 2018
> at 6:30 AM, Marco Varlese <mvarl...@suse.de> wrote:
> > > > > > > > >> > Ravi,
> > > > > > > > >> >
> > > > > > > > >> > On Sun, 2018-05-27
> at 12:20 -0700, Ravi Kerur wrote:
> > > > > > > > >> >> Hello,
> > > > > > > > >> >>
> > > > > > > > >> >> I have a
> VM(16.04.4 Ubuntu x86_64) with 2 cores and 4G RAM. I have
> > > > > > > > >> >> installed VPP
> successfully on it. Later I have created vhost-user
> > > > > > > > >> >> interfaces via
> > > > > > > > >> >>
> > > > > > > > >> >> create vhost
> socket /var/run/vpp/sock1.sock server
> > > > > > > > >> >> create vhost
> socket /var/run/vpp/sock2.sock server
> > > > > > > > >> >> set interface
> state VirtualEthernet0/0/0 up
> > > > > > > > >> >> set interface
> state VirtualEthernet0/0/1 up
> > > > > > > > >> >>
> > > > > > > > >> >> set interface l2
> bridge VirtualEthernet0/0/0 1
> > > > > > > > >> >> set interface l2
> bridge VirtualEthernet0/0/1 1
> > > > > > > > >> >>
> > > > > > > > >> >> I then run
> 'DPDK/testpmd' inside a container which will use
> > > > > > > > >> >> virtio-user
> interfaces using the following command
> > > > > > > > >> >>
> > > > > > > > >> >> docker run -it
> --privileged -v
> > > > > > > > >> >>
> /var/run/vpp/sock1.sock:/var/run/usvhost1 -v
> > > > > > > > >> >>
> /var/run/vpp/sock2.sock:/var/run/usvhost2 -v
> > > > > > > > >> >>
> /dev/hugepages:/dev/hugepages dpdk-app-testpmd ./bin/testpmd -c 0x3 -n
> > > > > > > > >> >> 4 --log-level=9
> -m 64 --no-pci --single-file-segments
> > > > > > > > >> >>
> --vdev=virtio_user0,path=/var/run/usvhost1,mac=54:01:00:01:01:01
> > > > > > > > >> >>
> --vdev=virtio_user1,path=/var/run/usvhost2,mac=54:01:00:01:01:02 --
> > > > > > > > >> >> -i
> > > > > > > > >> >>
> > > > > > > > >> >> VPP Vnet crashes
> with following message
> > > > > > > > >> >>
> > > > > > > > >> >> May 27 11:44:00
> localhost vnet[6818]: received signal SIGSEGV, PC
> > > > > > > > >> >> 0x7fcca4620187,
> faulting address 0x7fcb317ac000
> > > > > > > > >> >>
> > > > > > > > >> >> Questions:
> > > > > > > > >> >> I have 'ulimit -c
> unlimited' and /etc/vpp/startup.conf has
> > > > > > > > >> >> unix {
> > > > > > > > >> >> nodaemon
> > > > > > > > >> >> log
> /var/log/vpp/vpp.log
> > > > > > > > >> >> full-coredump
> > > > > > > > >> >> cli-listen
> /run/vpp/cli.sock
> > > > > > > > >> >> gid vpp
> > > > > > > > >> >> }
> > > > > > > > >> >>
> > > > > > > > >> >> But I couldn't
> locate corefile?
> > > > > > > > >> > The location of
> the coredump file depends on your system configuration.
> > > > > > > > >> >
> > > > > > > > >> > Please, check "cat
> /proc/sys/kernel/core_pattern"
> > > > > > > > >> >
> > > > > > > > >> > If you have
> systemd-coredump in the output of the above command, then likely the
> > > > > > > > >> > location of the
> coredump files is "/var/lib/systemd/coredump/"
> > > > > > > > >> >
> > > > > > > > >> > You can also
> change the location of where your system places the coredump files:
> > > > > > > > >> > echo
> '/PATH_TO_YOU_LOCATION/core_%e.%p' | sudo tee /proc/sys/kernel/core_pattern
> > > > > > > > >> >
> > > > > > > > >> > See if that
> helps...
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >> Initially
> '/proc/sys/kernel/core_pattern' was set to 'core'. I changed
> > > > > > > > >> it to
> 'systemd-coredump'. Still no core generated. VPP crashes
> > > > > > > > >>
> > > > > > > > >> May 29 08:54:34
> localhost vnet[4107]: received signal SIGSEGV, PC
> > > > > > > > >> 0x7f0167751187,
> faulting address 0x7efff43ac000
> > > > > > > > >> May 29 08:54:34
> localhost systemd[1]: vpp.service: Main process
> > > > > > > > >> exited, code=killed,
> status=6/ABRT
> > > > > > > > >> May 29 08:54:34
> localhost systemd[1]: vpp.service: Unit entered failed state.
> > > > > > > > >> May 29 08:54:34
> localhost systemd[1]: vpp.service: Failed with result 'signal'.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> cat
> /proc/sys/kernel/core_pattern
> > > > > > > > >> systemd-coredump
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> ulimit -a
> > > > > > > > >> core file size
> (blocks, -c) unlimited
> > > > > > > > >> data seg size
> (kbytes, -d) unlimited
> > > > > > > > >> scheduling priority
> (-e) 0
> > > > > > > > >> file size
> (blocks, -f) unlimited
> > > > > > > > >> pending signals
> (-i) 15657
> > > > > > > > >> max locked memory
> (kbytes, -l) 64
> > > > > > > > >> max memory size
> (kbytes, -m) unlimited
> > > > > > > > >> open files
> (-n) 1024
> > > > > > > > >> pipe size
> (512 bytes, -p) 8
> > > > > > > > >> POSIX message queues
> (bytes, -q) 819200
> > > > > > > > >> real-time priority
> (-r) 0
> > > > > > > > >> stack size
> (kbytes, -s) 8192
> > > > > > > > >> cpu time
> (seconds, -t) unlimited
> > > > > > > > >> max user processes
> (-u) 15657
> > > > > > > > >> virtual memory
> (kbytes, -v) unlimited
> > > > > > > > >> file locks
> (-x) unlimited
> > > > > > > > >>
> > > > > > > > >> cd
> /var/lib/systemd/coredump/
> > > > > > > > >>
> root@localhost:/var/lib/systemd/coredump# ls
> > > > > > > > >>
> root@localhost:/var/lib/systemd/coredump#
> > > > > > > > >>
> > > > > > > > >> >>
> > > > > > > > >> >> (2) How to enable
> debugs? I have used 'make build' but no additional
> > > > > > > > >> >> logs other than
> those shown below
> > > > > > > > >> >>
> > > > > > > > >> >>
> > > > > > > > >> >> VPP logs from
> /var/log/syslog is shown below
> > > > > > > > >> >> cat
> /var/log/syslog
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: vlib_plugin_early_init:361:
> > > > > > > > >> >> plugin path
> /usr/lib/vpp_plugins:/usr/lib64/vpp_plugins
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> abf_plugin.so (ACL based Forwarding)
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> acl_plugin.so (Access Control Lists)
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> avf_plugin.so (Intel Adaptive Virtual Function (AVF) Device
> > > > > > > > >> >> Plugin)
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:191: Loaded
> > > > > > > > >> >> plugin:
> cdp_plugin.so
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> dpdk_plugin.so (Data Plane Development Kit (DPDK))
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> flowprobe_plugin.so (Flow per Packet)
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> gbp_plugin.so (Group Based Policy)
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> gtpu_plugin.so (GTPv1-U)
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> igmp_plugin.so (IGMP messaging)
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> ila_plugin.so (Identifier-locator addressing for IPv6)
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> ioam_plugin.so (Inbound OAM)
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:117: Plugin
> > > > > > > > >> >> disabled
> (default): ixge_plugin.so
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> l2e_plugin.so (L2 Emulation)
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> lacp_plugin.so (Link Aggregation Control Protocol)
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> lb_plugin.so (Load Balancer)
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> memif_plugin.so (Packet Memory Interface (experimetal))
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> nat_plugin.so (Network Address Translation)
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> pppoe_plugin.so (PPPoE)
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> srv6ad_plugin.so (Dynamic SRv6 proxy)
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> srv6am_plugin.so (Masquerading SRv6 proxy)
> > > > > > > > >> >> May 27 11:40:28
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> srv6as_plugin.so (Static SRv6 proxy)
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> stn_plugin.so (VPP Steals the NIC for Container integration)
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> tlsmbedtls_plugin.so (mbedtls based TLS Engine)
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: load_one_plugin:189: Loaded
> > > > > > > > >> >> plugin:
> tlsopenssl_plugin.so (openssl based TLS Engine)
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: /usr/bin/vpp[6818]:
> > > > > > > > >> >>
> load_one_vat_plugin:67: Loaded plugin: dpdk_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > > > > > >> >> Loaded plugin:
> dpdk_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: /usr/bin/vpp[6818]:
> > > > > > > > >> >>
> load_one_vat_plugin:67: Loaded plugin: lb_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: /usr/bin/vpp[6818]:
> > > > > > > > >> >>
> load_one_vat_plugin:67: Loaded plugin: flowprobe_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: /usr/bin/vpp[6818]:
> > > > > > > > >> >>
> load_one_vat_plugin:67: Loaded plugin: stn_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: /usr/bin/vpp[6818]:
> > > > > > > > >> >>
> load_one_vat_plugin:67: Loaded plugin: nat_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: /usr/bin/vpp[6818]:
> > > > > > > > >> >>
> load_one_vat_plugin:67: Loaded plugin: udp_ping_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: /usr/bin/vpp[6818]:
> > > > > > > > >> >>
> load_one_vat_plugin:67: Loaded plugin: pppoe_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: /usr/bin/vpp[6818]:
> > > > > > > > >> >>
> load_one_vat_plugin:67: Loaded plugin: lacp_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > > > > > >> >> Loaded plugin:
> lb_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: /usr/bin/vpp[6818]:
> > > > > > > > >> >>
> load_one_vat_plugin:67: Loaded plugin: acl_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: /usr/bin/vpp[6818]:
> > > > > > > > >> >>
> load_one_vat_plugin:67: Loaded plugin: ioam_export_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: /usr/bin/vpp[6818]:
> > > > > > > > >> >>
> load_one_vat_plugin:67: Loaded plugin: ioam_trace_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: /usr/bin/vpp[6818]:
> > > > > > > > >> >>
> load_one_vat_plugin:67: Loaded plugin:
> > > > > > > > >> >>
> vxlan_gpe_ioam_export_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: /usr/bin/vpp[6818]:
> > > > > > > > >> >>
> load_one_vat_plugin:67: Loaded plugin: gtpu_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: /usr/bin/vpp[6818]:
> > > > > > > > >> >>
> load_one_vat_plugin:67: Loaded plugin: cdp_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: /usr/bin/vpp[6818]:
> > > > > > > > >> >>
> load_one_vat_plugin:67: Loaded plugin: ioam_vxlan_gpe_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: /usr/bin/vpp[6818]:
> > > > > > > > >> >>
> load_one_vat_plugin:67: Loaded plugin: memif_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: /usr/bin/vpp[6818]:
> > > > > > > > >> >>
> load_one_vat_plugin:67: Loaded plugin: ioam_pot_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > > > > > >> >> Loaded plugin:
> flowprobe_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > > > > > >> >> Loaded plugin:
> stn_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > > > > > >> >> Loaded plugin:
> nat_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > > > > > >> >> Loaded plugin:
> udp_ping_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > > > > > >> >> Loaded plugin:
> pppoe_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > > > > > >> >> Loaded plugin:
> lacp_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > > > > > >> >> Loaded plugin:
> acl_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > > > > > >> >> Loaded plugin:
> ioam_export_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > > > > > >> >> Loaded plugin:
> ioam_trace_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > > > > > >> >> Loaded plugin:
> vxlan_gpe_ioam_export_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > > > > > >> >> Loaded plugin:
> gtpu_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > > > > > >> >> Loaded plugin:
> cdp_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > > > > > >> >> Loaded plugin:
> ioam_vxlan_gpe_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > > > > > >> >> Loaded plugin:
> memif_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > > > > > >> >> Loaded plugin:
> ioam_pot_test_plugin.so
> > > > > > > > >> >> May 27 11:40:29
> localhost vpp[6818]: /usr/bin/vpp[6818]: dpdk: EAL
> > > > > > > > >> >> init args: -c 1
> -n 4 --no-pci --huge-dir /dev/hugepages --master-lcore
> > > > > > > > >> >> 0 --socket-mem
> 256,0
> > > > > > > > >> >> May 27 11:40:29
> localhost /usr/bin/vpp[6818]: dpdk: EAL init args: -c
> > > > > > > > >> >> 1 -n 4 --no-pci
> --huge-dir /dev/hugepages --master-lcore 0
> > > > > > > > >> >> --socket-mem 256,0
> > > > > > > > >> >> May 27 11:40:29
> localhost vnet[6818]: dpdk_ipsec_process:1019: not
> > > > > > > > >> >> enough DPDK
> crypto resources, default to OpenSSL
> > > > > > > > >> >> May 27 11:43:19
> localhost vnet[6818]: show vhost-user: unknown input `detail
> > > > > > > > >> >> May 27 11:44:00
> localhost vnet[6818]: received signal SIGSEGV, PC
> > > > > > > > >> >> 0x7fcca4620187,
> faulting address 0x7fcb317ac000
> > > > > > > > >> >> May 27 11:44:00
> localhost systemd[1]: vpp.service: Main process
> > > > > > > > >> >> exited,
> code=killed, status=6/ABRT
> > > > > > > > >> >> May 27 11:44:00
> localhost systemd[1]: vpp.service: Unit entered failed state.
> > > > > > > > >> >> May 27 11:44:00
> localhost systemd[1]: vpp.service: Failed with result
> > > > > > > > >> >> 'signal'.
> > > > > > > > >> >> May 27 11:44:00
> localhost systemd[1]: vpp.service: Service hold-off
> > > > > > > > >> >> time over,
> scheduling restart
> > > > > > > > >> >>
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> Ravi
> > > > > > > > >>
> > > > > > > > >> >>
> > > > > > > > >> >>
> > > > > > > > >> >> Thanks.
> > > > > > > > >> > Cheers,
> > > > > > > > >> > Marco
> > > > > > > > >> >>
> > > > > > > > >> >>
> > > > > > > > >> >>
> > > > > > > > >> > --
> > > > > > > > >> > Marco V
> > > > > > > > >> >
> > > > > > > > >> > SUSE LINUX GmbH |
> GF: Felix Imendörffer, Jane Smithard, Graham Norton
> > > > > > > > >> > HRB 21284 (AG
> Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links:
You receive all messages sent to this group.
View/Reply Online (#9534): https://lists.fd.io/g/vpp-dev/message/9534
View All Messages In Topic (24): https://lists.fd.io/g/vpp-dev/topic/20346431
Mute This Topic: https://lists.fd.io/mt/20346431/21656
New Topic: https://lists.fd.io/g/vpp-dev/post
Change Your Subscription: https://lists.fd.io/g/vpp-dev/editsub/21656
Group Home: https://lists.fd.io/g/vpp-dev
Contact Group Owner: vpp-dev+ow...@lists.fd.io
Terms of Service: https://lists.fd.io/static/tos
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub
Email sent to: arch...@mail-archive.com
-=-=-=-=-=-=-=-=-=-=-=-
