Damjan, Steven,

Kindly let me know if anything I have messed up?

I have compiled VPP on x86_64 and done everything as suggested by Steven.

Thanks.

On Tue, Jun 5, 2018 at 1:39 PM, Ravi Kerur <rke...@gmail.com> wrote:
> Hi Damjan,
>
> I am not intentional using it. I am running VPP on a x86 Ubuntu server.
>
> uname -a
> 4.9.77.2-rt61 #1 SMP PREEMPT RT Tue May 15 20:36:51 UTC 2018 x86_64
> x86_64 x86_64 GNU/Linux
>
> Thanks.
>
> On Tue, Jun 5, 2018 at 1:10 PM, Damjan Marion <dmar...@me.com> wrote:
>> Dear Ravi,
>>
>> Currently we don't support Octeon TX mempool. Are you intentionally using
>> it?
>>
>> Regards,
>>
>> Damjan
>>
>> On 5 Jun 2018, at 21:46, Ravi Kerur <rke...@gmail.com> wrote:
>>
>> Steven,
>>
>> I managed to get Tx/Rx rings setup with 1GB hugepages. However, when I
>> assign an IP address to both vhost-user/virtio interfaces and initiate
>> a ping VPP crashes.
>>
>> Any other mechanism available to test Tx/Rx path between Vhost and
>> Virtio? Details below.
>>
>>
>> *******On host*******
>> vpp#show vhost-user VirtualEthernet0/0/0
>> Virtio vhost-user interfaces
>> Global:
>>  coalesce frames 32 time 1e-3
>>  number of rx virtqueues in interrupt mode: 0
>> Interface: VirtualEthernet0/0/0 (ifindex 3)
>> virtio_net_hdr_sz 12
>> features mask (0xffffffffffffffff):
>> features (0x110008000):
>>   VIRTIO_NET_F_MRG_RXBUF (15)
>>   VIRTIO_F_INDIRECT_DESC (28)
>>   VIRTIO_F_VERSION_1 (32)
>>  protocol features (0x0)
>>
>> socket filename /var/run/vpp/sock3.sock type server errno "Success"
>>
>> rx placement:
>>   thread 0 on vring 1, polling
>> tx placement: lock-free
>>   thread 0 on vring 0
>>
>> Memory regions (total 1)
>> region fd    guest_phys_addr    memory_size        userspace_addr
>> mmap_offset        mmap_addr
>> ====== ===== ================== ================== ==================
>> ================== ==================
>>  0     26    0x00007f54c0000000 0x0000000040000000 0x00007f54c0000000
>> 0x0000000000000000 0x00007faf00000000
>>
>> Virtqueue 0 (TX)
>>  qsz 256 last_avail_idx 0 last_used_idx 0
>>  avail.flags 1 avail.idx 256 used.flags 1 used.idx 0
>>  kickfd 27 callfd 24 errfd -1
>>
>> Virtqueue 1 (RX)
>>  qsz 256 last_avail_idx 0 last_used_idx 0
>>  avail.flags 1 avail.idx 0 used.flags 1 used.idx 0
>>  kickfd 28 callfd 25 errfd -1
>>
>>
>> vpp#set interface ip address VirtualEthernet0/0/0 10.1.1.1/24
>>
>> ************On container**********
>> vpp# show interface VirtioUser0/0/0
>>              Name               Idx       State          Counter
>>    Count
>> VirtioUser0/0/0                   1         up
>> vpp#
>> vpp# set interface ip address VirtioUser0/0/0 10.1.1.2/24
>> vpp#
>> vpp# ping 10.1.1.1
>>
>> Statistics: 5 sent, 0 received, 100% packet loss
>> vpp#
>>
>>
>> ************Host vpp crash with following backtrace******************
>> Continuing.
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> octeontx_fpa_bufpool_alloc (handle=0)
>>    at
>> /var/venom/rk-vpp-1804/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/drivers/mempool/octeontx/rte_mempool_octeontx.c:57
>> 57        return (void *)(uintptr_t)fpavf_read64((void *)(handle +
>> (gdb) bt
>> #0  octeontx_fpa_bufpool_alloc (handle=0)
>>    at
>> /var/venom/rk-vpp-1804/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/drivers/mempool/octeontx/rte_mempool_octeontx.c:57
>> #1  octeontx_fpavf_dequeue (mp=0x7fae7fc9ab40, obj_table=0x7fb04d868880,
>> n=528)
>>    at
>> /var/venom/rk-vpp-1804/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/drivers/mempool/octeontx/rte_mempool_octeontx.c:98
>> #2  0x00007fb04b73bdef in rte_mempool_ops_dequeue_bulk (n=528,
>> obj_table=<optimized out>,
>>    mp=0x7fae7fc9ab40)
>>    at
>> /var/venom/rk-vpp-1804/vpp/build-root/install-vpp-native/dpdk/include/dpdk/rte_mempool.h:492
>> #3  __mempool_generic_get (cache=<optimized out>, n=<optimized out>,
>> obj_table=<optimized out>,
>>    mp=<optimized out>)
>>    at
>> /var/venom/rk-vpp-1804/vpp/build-root/install-vpp-native/dpdk/include/dpdk/rte_mempool.h:1271
>> #4  rte_mempool_generic_get (cache=<optimized out>, n=<optimized out>,
>>    obj_table=<optimized out>, mp=<optimized out>)
>>    at
>> /var/venom/rk-vpp-1804/vpp/build-root/install-vpp-native/dpdk/include/dpdk/rte_mempool.h:1306
>> #5  rte_mempool_get_bulk (n=528, obj_table=<optimized out>,
>> mp=0x7fae7fc9ab40)
>>    at
>> /var/venom/rk-vpp-1804/vpp/build-root/install-vpp-native/dpdk/include/dpdk/rte_mempool.h:1339
>> #6  dpdk_buffer_fill_free_list_avx2 (vm=0x7fb08ec69480
>> <vlib_global_main>, fl=0x7fb04cb2b100,
>>    min_free_buffers=<optimized out>)
>>    at /var/venom/rk-vpp-1804/vpp/build-data/../src/plugins/dpdk/buffer.c:228
>> #7  0x00007fb08e5046ea in vlib_buffer_alloc_from_free_list (index=0
>> '\000', n_buffers=514,
>>    buffers=0x7fb04cb8ec58, vm=0x7fb08ec69480 <vlib_global_main>)
>>    at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/buffer_funcs.h:306
>> #8  vhost_user_if_input (mode=<optimized out>, node=0x7fb04d0f5b80,
>> qid=<optimized out>,
>>    vui=0x7fb04d87523c, vum=0x7fb08e9b9560 <vhost_user_main>,
>>    vm=0x7fb08ec69480 <vlib_global_main>)
>>    at
>> /var/venom/rk-vpp-1804/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1644
>> #9  vhost_user_input (f=<optimized out>, node=<optimized out>,
>> vm=<optimized out>)
>>    at
>> /var/venom/rk-vpp-1804/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1947
>> #10 vhost_user_input_avx2 (vm=<optimized out>, node=<optimized out>,
>> frame=<optimized out>)
>>    at
>> /var/venom/rk-vpp-1804/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1972
>> #11 0x00007fb08ea166b3 in dispatch_node (last_time_stamp=<optimized
>> out>, frame=0x0,
>>    dispatch_state=VLIB_NODE_STATE_POLLING, type=VLIB_NODE_TYPE_INPUT,
>> node=0x7fb04d0f5b80,
>>    vm=0x7fb08ec69480 <vlib_global_main>)
>>    at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/main.c:988
>> #12 vlib_main_or_worker_loop (is_main=1, vm=0x7fb08ec69480
>> <vlib_global_main>)
>>    at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/main.c:1505
>> #13 vlib_main_loop (vm=0x7fb08ec69480 <vlib_global_main>)
>>    at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/main.c:1633
>> #14 vlib_main (vm=vm@entry=0x7fb08ec69480 <vlib_global_main>,
>> input=input@entry=0x7fb04d077fa0)
>>    at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/main.c:1787
>> #15 0x00007fb08ea4d683 in thread0 (arg=140396286350464)
>>    at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/unix/main.c:568
>> #16 0x00007fb08dbe15d8 in clib_calljmp ()
>>    at /var/venom/rk-vpp-1804/vpp/build-data/../src/vppinfra/longjmp.S:110
>> #17 0x00007fff0726d370 in ?? ()
>> ---Type <return> to continue, or q <return> to quit---
>> #18 0x00007fb08ea4e3da in vlib_unix_main (argc=<optimized out>,
>> argv=<optimized out>)
>>    at /var/venom/rk-vpp-1804/vpp/build-data/../src/vlib/unix/main.c:632
>> #19 0x0000001900000000 in ?? ()
>> #20 0x000000e700000000 in ?? ()
>> #21 0x0000000000000831 in ?? ()
>> #22 0x00007fb08e9aac00 in ?? () from /usr/lib/x86_64-linux-gnu/libvnet.so.0
>>
>> **************Vhost-user debugs on host**********
>> Jun  5 19:23:35 [18916]: vhost_user_socksvr_accept_ready:1294: New
>> client socket for vhost interface 3, fd 23
>> Jun  5 19:23:35 [18916]: vhost_user_socket_read:995: if 3 msg
>> VHOST_USER_SET_OWNER
>> Jun  5 19:23:35 [18916]: vhost_user_socket_read:847: if 3 msg
>> VHOST_USER_GET_FEATURES - reply 0x000000015c628000
>> Jun  5 19:23:35 [18916]: vhost_user_socket_read:995: if 3 msg
>> VHOST_USER_SET_OWNER
>> Jun  5 19:23:35 [18916]: vhost_user_socket_read:1004: if 3 msg
>> VHOST_USER_SET_VRING_CALL 0
>> Jun  5 19:23:35 [18916]: vhost_user_socket_read:1004: if 3 msg
>> VHOST_USER_SET_VRING_CALL 1
>> Jun  5 19:23:35 [18916]: vhost_user_socket_read:852: if 3 msg
>> VHOST_USER_SET_FEATURES features 0x0000000110008000
>> Jun  5 19:23:35 [18916]: vhost_user_socket_read:877: if 3 msg
>> VHOST_USER_SET_MEM_TABLE nregions 1
>> Jun  5 19:23:35[18916]: vhost_user_socket_read:916: map memory region
>> 0 addr 0 len 0x40000000 fd 26 mapped 0x7faf00000000 page_sz 0x40000000
>> Jun  5 19:23:35 [18916]: vhost_user_socket_read:932: if 3 msg
>> VHOST_USER_SET_VRING_NUM idx 0 num 256
>> Jun  5 19:23:35 [18916]: vhost_user_socket_read:1096: if 3 msg
>> VHOST_USER_SET_VRING_BASE idx 0 num 0
>> Jun  5 19:23:35 [18916]: vhost_user_socket_read:943: if 3 msg
>> VHOST_USER_SET_VRING_ADDR idx 0
>> Jun  5 19:23:35 [18916]: vhost_user_socket_read:1037: if 3 msg
>> VHOST_USER_SET_VRING_KICK 0
>> Jun  5 19:23:35 [18916]: vhost_user_socket_read:932: if 3 msg
>> VHOST_USER_SET_VRING_NUM idx 1 num 256
>> Jun  5 19:23:35 [18916]: vhost_user_socket_read:1096: if 3 msg
>> VHOST_USER_SET_VRING_BASE idx 1 num 0
>> Jun  5 19:23:35 [18916]: vhost_user_socket_read:943: if 3 msg
>> VHOST_USER_SET_VRING_ADDR idx 1
>> Jun  5 19:23:35 [18916]: vhost_user_socket_read:1037: if 3 msg
>> VHOST_USER_SET_VRING_KICK 1
>> Jun  5 19:23:35 [18916]: vhost_user_socket_read:1211: if 3
>> VHOST_USER_SET_VRING_ENABLE: enable queue 0
>> Jun  5 19:23:35[18916]: vhost_user_socket_read:1211: if 3
>> VHOST_USER_SET_VRING_ENABLE: enable queue 1
>>
>> Thanks.
>>
>> On Tue, Jun 5, 2018 at 11:31 AM, Steven Luong (sluong) <slu...@cisco.com>
>> wrote:
>>
>> Ravi,
>>
>> In order to use dpdk virtio_user, you need 1GB huge page.
>>
>> Steven
>>
>> On 6/5/18, 11:17 AM, "Ravi Kerur" <rke...@gmail.com> wrote:
>>
>>    Hi Steven,
>>
>>    Connection is the problem. I don't see memory regions setup correctly.
>>    Below are some details. Currently I am using 2MB hugepages.
>>
>>    (1) Create vhost-user server
>>    debug vhost-user on
>>    vpp# create vhost socket /var/run/vpp/sock3.sock server
>>    VirtualEthernet0/0/0
>>    vpp# set interface state VirtualEthernet0/0/0 up
>>    vpp#
>>    vpp#
>>
>>    (2) Instantiate a container
>>    docker run -it --privileged -v
>>    /var/run/vpp/sock3.sock:/var/run/usvhost1 -v
>>    /dev/hugepages:/dev/hugepages dpdk-app-vpp:latest
>>
>>    (3) Inside the container run EAL/DPDK virtio with following startup conf.
>>    unix {
>>      nodaemon
>>      log /var/log/vpp/vpp.log
>>      full-coredump
>>      cli-listen /run/vpp/cli.sock
>>      gid vpp
>>    }
>>
>>    api-trace {
>>      on
>>    }
>>
>>    api-segment {
>>      gid vpp
>>    }
>>
>>    dpdk {
>>            no-pci
>>            vdev virtio_user0,path=/var/run/usvhost1
>>    }
>>
>>    Following errors are seen due to 2MB hugepages and I think DPDK
>>    requires "--single-file-segments" option.
>>
>>    /usr/bin/vpp[19]: dpdk_config:1275: EAL init args: -c 1 -n 4 --no-pci
>>    --vdev virtio_user0,path=/var/run/usvhost1 --huge-dir
>>    /run/vpp/hugepages --file-prefix vpp --master-lcore 0 --socket-mem
>>    64,64
>>    /usr/bin/vpp[19]: dpdk_config:1275: EAL init args: -c 1 -n 4 --no-pci
>>    --vdev virtio_user0,path=/var/run/usvhost1 --huge-dir
>>    /run/vpp/hugepages --file-prefix vpp --master-lcore 0 --socket-mem
>>    64,64
>>    EAL: 4 hugepages of size 1073741824 reserved, but no mounted hugetlbfs
>>    found for that size
>>    EAL: VFIO support initialized
>>    get_hugepage_file_info(): Exceed maximum of 8
>>    prepare_vhost_memory_user(): Failed to prepare memory for vhost-user
>>    DPDK physical memory layout:
>>
>>
>>    Second test case>
>>    (1) and (2) are same as above. I run VPP inside a container with
>>    following startup config
>>
>>    unix {
>>      nodaemon
>>      log /var/log/vpp/vpp.log
>>      full-coredump
>>      cli-listen /run/vpp/cli.sock
>>      gid vpp
>>    }
>>
>>    api-trace {
>>      on
>>    }
>>
>>    api-segment {
>>      gid vpp
>>    }
>>
>>    dpdk {
>>            no-pci
>>            single-file-segments
>>            vdev virtio_user0,path=/var/run/usvhost1
>>    }
>>
>>
>>    VPP fails to start with
>>    plugin.so
>>    vpp[19]: dpdk_config: unknown input `single-file-segments no-pci vd...'
>>    vpp[19]: dpdk_config: unknown input `single-file-segments no-pci vd...'
>>
>>    [1]+  Done                    /usr/bin/vpp -c /etc/vpp/startup.conf
>>    root@867dc128b544:~/dpdk#
>>
>>
>>    show version (on both host and container).
>>    vpp v18.04-rc2~26-gac2b736~b45 built by root on 34a554d1c194 at Wed
>>    Apr 25 14:53:07 UTC 2018
>>    vpp#
>>
>>    Thanks.
>>
>>    On Tue, Jun 5, 2018 at 9:23 AM, Steven Luong (sluong) <slu...@cisco.com>
>> wrote:
>>
>> Ravi,
>>
>> Do this
>>
>> 1. Run VPP native vhost-user in the host. Turn on debug "debug vhost-user
>> on".
>> 2. Bring up the container with the vdev virtio_user commands that you have
>> as before
>> 3. show vhost-user in the host and verify that it has a shared memory
>> region. If not, the connection has a problem. Collect the show vhost-user
>> and debug vhost-user and send them to me and stop. If yes, proceed with step
>> 4.
>> 4. type "trace vhost-user-input 100" in the host
>> 5. clear error, and clear interfaces in the host and the container.
>> 6. do the ping from the container.
>> 7. Collect show error, show trace, show interface, and show vhost-user in
>> the host. Collect show error and show interface in the container. Put output
>> in github and provide a link to view. There is no need to send a large file.
>>
>> Steven
>>
>> On 6/4/18, 5:50 PM, "Ravi Kerur" <rke...@gmail.com> wrote:
>>
>>    Hi Steven,
>>
>>    Thanks for your help. I am using vhost-user client (VPP in container)
>>    and vhost-user server (VPP in host). I thought it should work.
>>
>>    create vhost socket /var/run/vpp/sock3.sock server (On host)
>>
>>    create vhost socket /var/run/usvhost1 (On container)
>>
>>    Can you please point me to a document which shows how to create VPP
>>    virtio_user interfaces or static configuration in
>>    /etc/vpp/startup.conf?
>>
>>    I have used following declarations in /etc/vpp/startup.conf
>>
>>    # vdev virtio_user0,path=/var/run/vpp/sock3.sock,mac=52:54:00:00:04:01
>>    # vdev virtio_user1,path=/var/run/vpp/sock4.sock,mac=52:54:00:00:04:02
>>
>>    but it doesn't work.
>>
>>    Thanks.
>>
>>    On Mon, Jun 4, 2018 at 3:57 PM, Steven Luong (sluong) <slu...@cisco.com>
>> wrote:
>>
>> Ravi,
>>
>> VPP only supports vhost-user in the device mode. In your example, the host,
>> in device mode, and the container also in device mode do not make a happy
>> couple. You need one of them, either the host or container, running in
>> driver mode using the dpdk vdev virtio_user command in startup.conf. So you
>> need something like this
>>
>> (host) VPP native vhost-user ----- (container) VPP DPDK vdev virtio_user
>>                          -- or --
>> (host) VPP DPDK vdev virtio_user ---- (container) VPP native vhost-user
>>
>> Steven
>>
>> On 6/4/18, 3:27 PM, "Ravi Kerur" <rke...@gmail.com> wrote:
>>
>>    Hi Steven
>>
>>    Though crash is not happening anymore, there is still an issue with Rx
>>    and Tx. To eliminate whether it is testpmd or vpp, I decided to run
>>
>>    (1) VPP vhost-user server on host-x
>>    (2) Run VPP in a container on host-x and vhost-user client port
>>    connecting to vhost-user server.
>>
>>    Still doesn't work. Details below. Please let me know if something is
>>    wrong in what I am doing.
>>
>>
>>    (1) VPP vhost-user as a server
>>    (2) VPP in a container virtio-user or vhost-user client
>>
>>    (1) Create vhost-user server socket on VPP running on host.
>>
>>    vpp#create vhost socket /var/run/vpp/sock3.sock server
>>    vpp#set interface state VirtualEthernet0/0/0 up
>>    show vhost-user VirtualEthernet0/0/0 descriptors
>>    Virtio vhost-user interfaces
>>    Global:
>>    coalesce frames 32 time 1e-3
>>    number of rx virtqueues in interrupt mode: 0
>>    Interface: VirtualEthernet0/0/0 (ifindex 3)
>>    virtio_net_hdr_sz 0
>>    features mask (0xffffffffffffffff):
>>    features (0x0):
>>    protocol features (0x0)
>>
>>    socket filename /var/run/vpp/sock3.sock type server errno "Success"
>>
>>    rx placement:
>>    tx placement: spin-lock
>>    thread 0 on vring 0
>>
>>    Memory regions (total 0)
>>
>>    vpp# set interface ip address VirtualEthernet0/0/0 192.168.1.1/24
>>    vpp#
>>
>>    (2) Instantiate a docker container to run VPP connecting to sock3.server
>> socket.
>>
>>    docker run -it --privileged -v
>>    /var/run/vpp/sock3.sock:/var/run/usvhost1 -v
>>    /dev/hugepages:/dev/hugepages dpdk-app-vpp:latest
>>    root@4b1bd06a3225:~/dpdk#
>>    root@4b1bd06a3225:~/dpdk# ps -ef
>>    UID PID PPID C STIME TTY TIME CMD
>>    root 1 0 0 21:39 ? 00:00:00 /bin/bash
>>    root 17 1 0 21:39 ? 00:00:00 ps -ef
>>    root@4b1bd06a3225:~/dpdk#
>>
>>    root@8efda6701ace:~/dpdk# ps -ef | grep vpp
>>    root 19 1 39 21:41 ? 00:00:03 /usr/bin/vpp -c /etc/vpp/startup.conf
>>    root 25 1 0 21:41 ? 00:00:00 grep --color=auto vpp
>>    root@8efda6701ace:~/dpdk#
>>
>>    vpp#create vhost socket /var/run/usvhost1
>>    vpp#set interface state VirtualEthernet0/0/0 up
>>    vpp#show vhost-user VirtualEthernet0/0/0 descriptors
>>    Virtio vhost-user interfaces
>>    Global:
>>    coalesce frames 32 time 1e-3
>>    number of rx virtqueues in interrupt mode: 0
>>    Interface: VirtualEthernet0/0/0 (ifindex 1)
>>    virtio_net_hdr_sz 0
>>    features mask (0xffffffffffffffff):
>>    features (0x0):
>>    protocol features (0x0)
>>
>>    socket filename /var/run/usvhost1 type client errno "Success"
>>
>>    rx placement:
>>    tx placement: spin-lock
>>    thread 0 on vring 0
>>
>>    Memory regions (total 0)
>>
>>    vpp#
>>
>>    vpp# set interface ip address VirtualEthernet0/0/0 192.168.1.2/24
>>    vpp#
>>
>>    vpp# ping 192.168.1.1
>>
>>    Statistics: 5 sent, 0 received, 100% packet loss
>>    vpp#
>>
>>    On Thu, May 31, 2018 at 2:30 PM, Steven Luong (sluong) <slu...@cisco.com>
>> wrote:
>>
>> show interface and look for the counter and count columns for the
>> corresponding interface.
>>
>> Steven
>>
>> On 5/31/18, 1:28 PM, "Ravi Kerur" <rke...@gmail.com> wrote:
>>
>>    Hi Steven,
>>
>>    You made my day, thank you. I didn't realize different dpdk versions
>>    (vpp -- 18.02.1 and testpmd -- from latest git repo (probably 18.05)
>>    could be the cause of the problem, I still dont understand why it
>>    should as virtio/vhost messages are meant to setup tx/rx rings
>>    correctly?
>>
>>    I downloaded dpdk 18.02.1 stable release and at least vpp doesn't
>>    crash now (for both vpp-native and dpdk vhost interfaces). I have one
>>    question is there a way to read vhost-user statistics counter (Rx/Tx)
>>    on vpp? I only know
>>
>>    'show vhost-user <intf>' and 'show vhost-user <intf> descriptors'
>>    which doesn't show any counters.
>>
>>    Thanks.
>>
>>    On Thu, May 31, 2018 at 11:51 AM, Steven Luong (sluong)
>>    <slu...@cisco.com> wrote:
>>
>> Ravi,
>>
>> For (1) which works, what dpdk version are you using in the host? Are you
>> using the same dpdk version as VPP is using? Since you are using VPP latest,
>> I think it is 18.02. Type "show dpdk version" at the VPP prompt to find out
>> for sure.
>>
>> Steven
>>
>> On 5/31/18, 11:44 AM, "Ravi Kerur" <rke...@gmail.com> wrote:
>>
>>    Hi Steven,
>>
>>    i have tested following scenarios and it basically is not clear why
>>    you think DPDK is the problem? Is it possible VPP and DPDK use
>>    different virtio versions?
>>
>>    Following are the scenarios I have tested
>>
>>    (1) testpmd/DPDK vhost-user (running on host) and testpmd/DPDK
>>    virito-user (in a container) -- can send and receive packets
>>    (2) VPP-native vhost-user (running on host) and testpmd/DPDK
>>    virtio-user (in a container) -- VPP crashes and it is in VPP code
>>    (3) VPP-DPDK vhost user (running on host) and testpmd/DPDK virtio-user
>>    (in a container) -- VPP crashes and in DPDK
>>
>>    Thanks.
>>
>>    On Thu, May 31, 2018 at 10:12 AM, Steven Luong (sluong)
>>    <slu...@cisco.com> wrote:
>>
>> Ravi,
>>
>> I've proved my point -- there is a problem in the way that you invoke
>> testpmd. The shared memory region that it passes to the device is not
>> accessible from the device. I don't know what the correct options are that
>> you need to use. This is really a question for dpdk.
>>
>> As a further exercise, you could remove VPP in the host and instead run
>> testpmd in device mode using "--vdev
>> net_vhost0,iface=/var/run/vpp/sock1.sock" option. I bet you testpmd in the
>> host will crash in the same place. I hope you can find out the answer from
>> dpdk and tell us about it.
>>
>> Steven
>>
>> On 5/31/18, 9:31 AM, "vpp-dev@lists.fd.io on behalf of Ravi Kerur"
>> <vpp-dev@lists.fd.io on behalf of rke...@gmail.com> wrote:
>>
>>    Hi Steven,
>>
>>    Thank you for your help, I removed sock1.sock and sock2.sock,
>>    restarted vpp, atleast interfaces get created. However, when I start
>>    dpdk/testpmd inside the container it crashes as well. Below are some
>>    details. I am using vpp code from latest repo.
>>
>>    (1) On host
>>    show interface
>>                  Name               Idx       State          Counter
>>        Count
>>    VhostEthernet2                    3        down
>>    VhostEthernet3                    4        down
>>    VirtualFunctionEthernet4/10/4     1        down
>>    VirtualFunctionEthernet4/10/6     2        down
>>    local0                            0        down
>>    vpp#
>>    vpp# set interface state VhostEthernet2 up
>>    vpp# set interface state VhostEthernet3 up
>>    vpp#
>>    vpp# set interface l2 bridge VhostEthernet2 1
>>    vpp# set interface l2 bridge VhostEthernet3 1
>>    vpp#
>>
>>    (2) Run tespmd inside the container
>>    docker run -it --privileged -v
>>    /var/run/vpp/sock1.sock:/var/run/usvhost1 -v
>>    /var/run/vpp/sock2.sock:/var/run/usvhost2 -v
>>    /dev/hugepages:/dev/hugepages dpdk-app-testpmd ./bin/testpmd -l 16-19
>>    -n 4 --log-level=8 -m 64 --no-pci
>>    --vdev=virtio_user0,path=/var/run/usvhost1,mac=54:00:00:01:01:01
>>    --vdev=virtio_user1,path=/var/run/usvhost2,mac=54:00:00:01:01:02 --
>>    -i
>>    EAL: Detected 28 lcore(s)
>>    EAL: Detected 2 NUMA nodes
>>    EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>>    EAL: 8192 hugepages of size 2097152 reserved, but no mounted hugetlbfs
>>    found for that size
>>    EAL: Probing VFIO support...
>>    EAL: VFIO support initialized
>>    EAL: Setting up physically contiguous memory...
>>    EAL: locking hot plug lock memory...
>>    EAL: primary init32...
>>    Interactive-mode selected
>>    Warning: NUMA should be configured manually by using
>>    --port-numa-config and --ring-numa-config parameters along with
>>    --numa.
>>    testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456,
>>    size=2176, socket=0
>>    testpmd: preferred mempool ops selected: ring_mp_mc
>>    testpmd: create a new mbuf pool <mbuf_pool_socket_1>: n=171456,
>>    size=2176, socket=1
>>    testpmd: preferred mempool ops selected: ring_mp_mc
>>    Port 0 is now not stopped
>>    Port 1 is now not stopped
>>    Please stop the ports first
>>    Done
>>    testpmd>
>>
>>    (3) VPP crashes with the same issue but inside dpdk code
>>
>>    (gdb) cont
>>    Continuing.
>>
>>    Program received signal SIGSEGV, Segmentation fault.
>>    [Switching to Thread 0x7ffd0d08e700 (LWP 41257)]
>>    rte_vhost_dequeue_burst (vid=<optimized out>, queue_id=<optimized
>>    out>, mbuf_pool=0x7fe17fc883c0,
>>        pkts=pkts@entry=0x7fffb671ebc0, count=count@entry=32)
>>        at
>> /var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_vhost/virtio_net.c:1504
>>    1504        free_entries = *((volatile uint16_t *)&vq->avail->idx) -
>>    (gdb) bt
>>    #0  rte_vhost_dequeue_burst (vid=<optimized out>, queue_id=<optimized
>> out>,
>>        mbuf_pool=0x7fe17fc883c0, pkts=pkts@entry=0x7fffb671ebc0,
>>    count=count@entry=32)
>>        at
>> /var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_vhost/virtio_net.c:1504
>>    #1  0x00007fffb4718e6f in eth_vhost_rx (q=0x7fe17fbbdd80,
>> bufs=0x7fffb671ebc0,
>>        nb_bufs=<optimized out>)
>>        at
>> /var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/drivers/net/vhost/rte_eth_vhost.c:410
>>    #2  0x00007fffb441cb7c in rte_eth_rx_burst (nb_pkts=256,
>>    rx_pkts=0x7fffb671ebc0, queue_id=0,
>>        port_id=3) at
>>
>> /var/venom/vpp/build-root/install-vpp-native/dpdk/include/dpdk/rte_ethdev.h:3635
>>    #3  dpdk_device_input (queue_id=0, thread_index=<optimized out>,
>>    node=0x7fffb732c700,
>>        xd=0x7fffb7337240, dm=<optimized out>, vm=0x7fffb6703340)
>>        at /var/venom/vpp/build-data/../src/plugins/dpdk/device/node.c:477
>>    #4  dpdk_input_node_fn_avx2 (vm=<optimized out>, node=<optimized out>,
>>    f=<optimized out>)
>>        at /var/venom/vpp/build-data/../src/plugins/dpdk/device/node.c:658
>>    #5  0x00007ffff7954d35 in dispatch_node
>>    (last_time_stamp=12531752723928016, frame=0x0,
>>        dispatch_state=VLIB_NODE_STATE_POLLING, type=VLIB_NODE_TYPE_INPUT,
>>    node=0x7fffb732c700,
>>        vm=0x7fffb6703340) at
>> /var/venom/vpp/build-data/../src/vlib/main.c:988
>>    #6  vlib_main_or_worker_loop (is_main=0, vm=0x7fffb6703340)
>>        at /var/venom/vpp/build-data/../src/vlib/main.c:1507
>>    #7  vlib_worker_loop (vm=0x7fffb6703340) at
>>    /var/venom/vpp/build-data/../src/vlib/main.c:1641
>>    #8  0x00007ffff6ad25d8 in clib_calljmp ()
>>        at /var/venom/vpp/build-data/../src/vppinfra/longjmp.S:110
>>    #9  0x00007ffd0d08ddb0 in ?? ()
>>    #10 0x00007fffb4436edd in eal_thread_loop (arg=<optimized out>)
>>        at
>> /var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_eal/linuxapp/eal/eal_thread.c:153
>>    #11 0x0000000000000000 in ?? ()
>>    (gdb) frame 0
>>    #0  rte_vhost_dequeue_burst (vid=<optimized out>, queue_id=<optimized
>> out>,
>>        mbuf_pool=0x7fe17fc883c0, pkts=pkts@entry=0x7fffb671ebc0,
>>    count=count@entry=32)
>>        at
>> /var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_vhost/virtio_net.c:1504
>>    1504        free_entries = *((volatile uint16_t *)&vq->avail->idx) -
>>    (gdb) p vq
>>    $1 = (struct vhost_virtqueue *) 0x7fc3ffc84b00
>>    (gdb) p vq->avail
>>    $2 = (struct vring_avail *) 0x7ffbfff98000
>>    (gdb) p *$2
>>    Cannot access memory at address 0x7ffbfff98000
>>    (gdb)
>>
>>
>>    Thanks.
>>
>>    On Thu, May 31, 2018 at 12:09 AM, Steven Luong (sluong)
>>    <slu...@cisco.com> wrote:
>>
>> Sorry, I was expecting to see two VhostEthernet interfaces like this. Those
>> VirtualFunctionEthernet are your physical interfaces.
>>
>> sh int
>>              Name               Idx       State          Counter
>> Count
>> VhostEthernet0                    1         up
>> VhostEthernet1                    2         up
>> local0                            0        down
>> DBGvpp#
>>
>> You have to first manually remove /var/run/vpp/sock1.sock and
>> /var/run/vpp/sock2.sock before you start vpp on the host. dpdk does not like
>> it if they already existed. If you successfully create VhostEthernet
>> interface, try to send some traffic through it to see if it crashes or not.
>>
>> Steven
>>
>> On 5/30/18, 9:17 PM, "vpp-dev@lists.fd.io on behalf of Steven Luong
>> (sluong)" <vpp-dev@lists.fd.io on behalf of slu...@cisco.com> wrote:
>>
>>    Ravi,
>>
>>    I don't think you can declare (2) works fine yet. Please bring up the
>> dpdk vhost-user interfaces and try to send some traffic between them to
>> exercise the shared memory region from dpdk virtio-user which may be
>> "questionable".
>>
>>        VirtualFunctionEthernet4/10/4     1        down
>>        VirtualFunctionEthernet4/10/6     2        down
>>
>>    Steven
>>
>>    On 5/30/18, 4:41 PM, "Ravi Kerur" <rke...@gmail.com> wrote:
>>
>>        Hi Steve,
>>
>>        Thank you for your inputs, I added feature-mask to see if it helps in
>>        setting up queues correctly, it didn't so I will remove it. I have
>>        tried following combination
>>
>>        (1) VPP->vhost-user (on host) and DPDK/testpmd->virtio-user (in a
>>        container)  -- VPP crashes
>>        (2) DPDK/testpmd->vhost-user (on host) and DPDK/testpmd->virtio-user
>>        (in a container) -- works fine
>>
>>        To use DPDK vhost-user inside VPP, I defined configuration in
>>        startup.conf as mentioned by you and it looks as follows
>>
>>        unix {
>>          nodaemon
>>          log /var/log/vpp/vpp.log
>>          full-coredump
>>          cli-listen /run/vpp/cli.sock
>>          gid vpp
>>        }
>>
>>        api-segment {
>>          gid vpp
>>        }
>>
>>        cpu {
>>                main-core 1
>>                corelist-workers 6-9
>>        }
>>
>>        dpdk {
>>                dev 0000:04:10.4
>>                dev 0000:04:10.6
>>                uio-driver vfio-pci
>>                vdev net_vhost0,iface=/var/run/vpp/sock1.sock
>>                vdev net_vhost1,iface=/var/run/vpp/sock2.sock
>>                huge-dir /dev/hugepages_1GB
>>                socket-mem 2048,2048
>>        }
>>
>>        From VPP logs
>>        dpdk: EAL init args: -c 3c2 -n 4 --vdev
>>        net_vhost0,iface=/var/run/vpp/sock1.sock --vdev
>>        net_vhost1,iface=/var/run/vpp/sock2.sock --huge-dir
>> /dev/hugepages_1GB
>>        -w 0000:04:10.4 -w 0000:04:10.6 --master-lcore 1 --socket-mem
>>        2048,2048
>>
>>        However, VPP doesn't create interface at all
>>
>>        vpp# show interface
>>                      Name               Idx       State          Counter
>>            Count
>>        VirtualFunctionEthernet4/10/4     1        down
>>        VirtualFunctionEthernet4/10/6     2        down
>>        local0                            0        down
>>
>>        since it is a static mapping I am assuming it should be created,
>> correct?
>>
>>        Thanks.
>>
>>        On Wed, May 30, 2018 at 3:43 PM, Steven Luong (sluong)
>> <slu...@cisco.com> wrote:
>>
>> Ravi,
>>
>> First and foremost, get rid of the feature-mask option. I don't know what
>> 0x40400000 does for you. If that does not help, try testing it with dpdk
>> based vhost-user instead of VPP native vhost-user to make sure that they can
>> work well with each other first. To use dpdk vhost-user, add a vdev command
>> in the startup.conf for each vhost-user device that you have.
>>
>> dpdk { vdev net_vhost0,iface=/var/run/vpp/sock1.sock }
>>
>> dpdk based vhost-user interface is named VhostEthernet0, VhostEthernet1,
>> etc. Make sure you use the right interface name to set the state to up.
>>
>> If dpdk based vhost-user does not work with testpmd either, it looks like
>> some problem with the way that you invoke testpmd.
>>
>> If dpdk based vhost-user works well with the same testpmd device driver and
>> not vpp native vhost-user, I can set up something similar to yours to look
>> into it.
>>
>> The device driver, testpmd, is supposed to pass the shared memory region to
>> VPP for TX/RX queues. It looks like VPP vhost-user might have run into a
>> bump there with using the shared memory (txvq->avail).
>>
>> Steven
>>
>> PS. vhost-user is not an optimum interface for containers. You may want to
>> look into using memif if you don't already know about it.
>>
>>
>> On 5/30/18, 2:06 PM, "Ravi Kerur" <rke...@gmail.com> wrote:
>>
>>    I am not sure what is wrong with the setup or a bug in vpp, vpp
>>    crashes with vhost<-->virtio communication.
>>
>>    (1) Vhost-interfaces are created and attached to bridge-domain as follows
>>
>>    create vhost socket /var/run/vpp/sock1.sock server feature-mask
>> 0x40400000
>>    create vhost socket /var/run/vpp/sock2.sock server feature-mask
>> 0x40400000
>>    set interface state VirtualEthernet0/0/0 up
>>    set interface state VirtualEthernet0/0/1 up
>>
>>    set interface l2 bridge VirtualEthernet0/0/0 1
>>    set interface l2 bridge VirtualEthernet0/0/1 1
>>
>>
>>    (2) DPDK/testpmd is started in a container to talk to vpp/vhost-user
>>    interface as follows
>>
>>    docker run -it --privileged -v
>>    /var/run/vpp/sock1.sock:/var/run/usvhost1 -v
>>    /var/run/vpp/sock2.sock:/var/run/usvhost2 -v
>>    /dev/hugepages:/dev/hugepages dpdk-app-testpmd ./bin/testpmd -c 0x3 -n
>>    4 --log-level=9 -m 64 --no-pci --single-file-segments
>>    --vdev=virtio_user0,path=/var/run/usvhost1,mac=54:00:00:01:01:01
>>    --vdev=virtio_user1,path=/var/run/usvhost2,mac=54:00:00:01:01:02 --
>>    -i
>>
>>    (3) show vhost-user VirtualEthernet0/0/1
>>    Virtio vhost-user interfaces
>>    Global:
>>      coalesce frames 32 time 1e-3
>>      number of rx virtqueues in interrupt mode: 0
>>    Interface: VirtualEthernet0/0/1 (ifindex 4)
>>    virtio_net_hdr_sz 10
>>     features mask (0x40400000):
>>     features (0x0):
>>      protocol features (0x0)
>>
>>     socket filename /var/run/vpp/sock2.sock type server errno "Success"
>>
>>     rx placement:
>>     tx placement: spin-lock
>>       thread 0 on vring 0
>>       thread 1 on vring 0
>>       thread 2 on vring 0
>>       thread 3 on vring 0
>>       thread 4 on vring 0
>>
>>     Memory regions (total 1)
>>     region fd    guest_phys_addr    memory_size        userspace_addr
>>    mmap_offset        mmap_addr
>>     ====== ===== ================== ================== ==================
>>    ================== ==================
>>      0     55    0x00007ff7c0000000 0x0000000040000000 0x00007ff7c0000000
>>    0x0000000000000000 0x00007ffbc0000000
>>
>>    vpp# show vhost-user VirtualEthernet0/0/0
>>    Virtio vhost-user interfaces
>>    Global:
>>      coalesce frames 32 time 1e-3
>>      number of rx virtqueues in interrupt mode: 0
>>    Interface: VirtualEthernet0/0/0 (ifindex 3)
>>    virtio_net_hdr_sz 10
>>     features mask (0x40400000):
>>     features (0x0):
>>      protocol features (0x0)
>>
>>     socket filename /var/run/vpp/sock1.sock type server errno "Success"
>>
>>     rx placement:
>>     tx placement: spin-lock
>>       thread 0 on vring 0
>>       thread 1 on vring 0
>>       thread 2 on vring 0
>>       thread 3 on vring 0
>>       thread 4 on vring 0
>>
>>     Memory regions (total 1)
>>     region fd    guest_phys_addr    memory_size        userspace_addr
>>    mmap_offset        mmap_addr
>>     ====== ===== ================== ================== ==================
>>    ================== ==================
>>      0     51    0x00007ff7c0000000 0x0000000040000000 0x00007ff7c0000000
>>    0x0000000000000000 0x00007ffc00000000
>>
>>    (4) vpp stack trace
>>    Program received signal SIGSEGV, Segmentation fault.
>>    [Switching to Thread 0x7ffd0e090700 (LWP 46570)]
>>    0x00007ffff7414642 in vhost_user_if_input
>>    (mode=VNET_HW_INTERFACE_RX_MODE_POLLING,
>>        node=0x7fffb76bab00, qid=<optimized out>, vui=0x7fffb6739700,
>>        vum=0x7ffff78f4480 <vhost_user_main>, vm=0x7fffb672a9c0)
>>        at
>> /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1596
>>    1596      if (PREDICT_FALSE (txvq->avail->flags & 0xFFFE))
>>    (gdb) bt
>>    #0  0x00007ffff7414642 in vhost_user_if_input
>>    (mode=VNET_HW_INTERFACE_RX_MODE_POLLING,
>>        node=0x7fffb76bab00, qid=<optimized out>, vui=0x7fffb6739700,
>>        vum=0x7ffff78f4480 <vhost_user_main>, vm=0x7fffb672a9c0)
>>        at
>> /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1596
>>    #1  vhost_user_input (f=<optimized out>, node=<optimized out>,
>>    vm=<optimized out>)
>>        at
>> /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1947
>>    #2  vhost_user_input_avx2 (vm=<optimized out>, node=<optimized out>,
>>    frame=<optimized out>)
>>        at
>> /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1972
>>    #3  0x00007ffff7954d35 in dispatch_node
>>    (last_time_stamp=12391212490024174, frame=0x0,
>>        dispatch_state=VLIB_NODE_STATE_POLLING, type=VLIB_NODE_TYPE_INPUT,
>>    node=0x7fffb76bab00,
>>        vm=0x7fffb672a9c0) at
>> /var/venom/vpp/build-data/../src/vlib/main.c:988
>>    #4  vlib_main_or_worker_loop (is_main=0, vm=0x7fffb672a9c0)
>>        at /var/venom/vpp/build-data/../src/vlib/main.c:1507
>>    #5  vlib_worker_loop (vm=0x7fffb672a9c0) at
>>    /var/venom/vpp/build-data/../src/vlib/main.c:1641
>>    #6  0x00007ffff6ad25d8 in clib_calljmp ()
>>        at /var/venom/vpp/build-data/../src/vppinfra/longjmp.S:110
>>    #7  0x00007ffd0e08fdb0 in ?? ()
>>    #8  0x00007fffb4436edd in eal_thread_loop (arg=<optimized out>)
>>        at
>> /var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_eal/linuxapp/eal/eal_thread.c:153
>>    #9  0x0000000000000000 in ?? ()
>>    (gdb) frame 0
>>    #0  0x00007ffff7414642 in vhost_user_if_input
>>    (mode=VNET_HW_INTERFACE_RX_MODE_POLLING,
>>        node=0x7fffb76bab00, qid=<optimized out>, vui=0x7fffb6739700,
>>        vum=0x7ffff78f4480 <vhost_user_main>, vm=0x7fffb672a9c0)
>>        at
>> /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1596
>>    1596      if (PREDICT_FALSE (txvq->avail->flags & 0xFFFE))
>>    (gdb) p txvq
>>    $1 = (vhost_user_vring_t *) 0x7fffb6739ac0
>>    (gdb) p *txvq
>>    $2 = {cacheline0 = 0x7fffb6739ac0 "?", qsz_mask = 255, last_avail_idx
>>    = 0, last_used_idx = 0,
>>      n_since_last_int = 0, desc = 0x7ffbfff97000, avail = 0x7ffbfff98000,
>>    used = 0x7ffbfff99000,
>>      int_deadline = 0, started = 1 '\001', enabled = 0 '\000', log_used = 0
>> '\000',
>>      cacheline1 = 0x7fffb6739b00 "????\n", errfd = -1, callfd_idx = 10,
>>    kickfd_idx = 14,
>>      log_guest_addr = 0, mode = 1}
>>    (gdb) p *(txvq->avail)
>>    Cannot access memory at address 0x7ffbfff98000
>>    (gdb)
>>
>>    On Tue, May 29, 2018 at 10:47 AM, Ravi Kerur <rke...@gmail.com> wrote:
>>
>> Steve,
>>
>> Thanks for inputs on debugs and gdb. I am using gdb on my development
>> system to debug the issue. I would like to have reliable core
>> generation on the system on which I don't have access to install gdb.
>> I installed corekeeper and it still doesn't generate core. I am
>> running vpp inside a VM (VirtualBox/vagrant), not sure if I need to
>> set something inside vagrant config file.
>>
>> dpkg -l corekeeper
>> Desired=Unknown/Install/Remove/Purge/Hold
>> |
>> Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
>> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
>> ||/ Name                 Version         Architecture    Description
>> +++-====================-===============-===============-==============================================
>> ii  corekeeper           1.6             amd64           enable core
>> files and report crashes to the system
>>
>> Thanks.
>>
>> On Tue, May 29, 2018 at 9:38 AM, Steven Luong (sluong) <slu...@cisco.com>
>> wrote:
>>
>> Ravi,
>>
>> I install corekeeper and the core file is kept in /var/crash. But why not
>> use gdb to attach to the VPP process?
>> To turn on VPP vhost-user debug, type "debug vhost-user on" at the VPP
>> prompt.
>>
>> Steven
>>
>> On 5/29/18, 9:10 AM, "vpp-dev@lists.fd.io on behalf of Ravi Kerur"
>> <vpp-dev@lists.fd.io on behalf of rke...@gmail.com> wrote:
>>
>>    Hi Marco,
>>
>>
>>    On Tue, May 29, 2018 at 6:30 AM, Marco Varlese <mvarl...@suse.de> wrote:
>>
>> Ravi,
>>
>> On Sun, 2018-05-27 at 12:20 -0700, Ravi Kerur wrote:
>>
>> Hello,
>>
>> I have a VM(16.04.4 Ubuntu x86_64) with 2 cores and 4G RAM. I have
>> installed VPP successfully on it. Later I have created vhost-user
>> interfaces via
>>
>> create vhost socket /var/run/vpp/sock1.sock server
>> create vhost socket /var/run/vpp/sock2.sock server
>> set interface state VirtualEthernet0/0/0 up
>> set interface state VirtualEthernet0/0/1 up
>>
>> set interface l2 bridge VirtualEthernet0/0/0 1
>> set interface l2 bridge VirtualEthernet0/0/1 1
>>
>> I then run 'DPDK/testpmd' inside a container which will use
>> virtio-user interfaces using the following command
>>
>> docker run -it --privileged -v
>> /var/run/vpp/sock1.sock:/var/run/usvhost1 -v
>> /var/run/vpp/sock2.sock:/var/run/usvhost2 -v
>> /dev/hugepages:/dev/hugepages dpdk-app-testpmd ./bin/testpmd -c 0x3 -n
>> 4 --log-level=9 -m 64 --no-pci --single-file-segments
>> --vdev=virtio_user0,path=/var/run/usvhost1,mac=54:01:00:01:01:01
>> --vdev=virtio_user1,path=/var/run/usvhost2,mac=54:01:00:01:01:02 --
>> -i
>>
>> VPP Vnet crashes with following message
>>
>> May 27 11:44:00 localhost vnet[6818]: received signal SIGSEGV, PC
>> 0x7fcca4620187, faulting address 0x7fcb317ac000
>>
>> Questions:
>> I have 'ulimit -c unlimited' and /etc/vpp/startup.conf has
>> unix {
>>  nodaemon
>>  log /var/log/vpp/vpp.log
>>  full-coredump
>>  cli-listen /run/vpp/cli.sock
>>  gid vpp
>> }
>>
>> But I couldn't locate corefile?
>>
>> The location of the coredump file depends on your system configuration.
>>
>> Please, check "cat /proc/sys/kernel/core_pattern"
>>
>> If you have systemd-coredump in the output of the above command, then likely
>> the
>> location of the coredump files is "/var/lib/systemd/coredump/"
>>
>> You can also change the location of where your system places the coredump
>> files:
>> echo '/PATH_TO_YOU_LOCATION/core_%e.%p' | sudo tee
>> /proc/sys/kernel/core_pattern
>>
>> See if that helps...
>>
>>
>>    Initially '/proc/sys/kernel/core_pattern' was set to 'core'. I changed
>>    it to 'systemd-coredump'. Still no core generated. VPP crashes
>>
>>    May 29 08:54:34 localhost vnet[4107]: received signal SIGSEGV, PC
>>    0x7f0167751187, faulting address 0x7efff43ac000
>>    May 29 08:54:34 localhost systemd[1]: vpp.service: Main process
>>    exited, code=killed, status=6/ABRT
>>    May 29 08:54:34 localhost systemd[1]: vpp.service: Unit entered failed
>> state.
>>    May 29 08:54:34 localhost systemd[1]: vpp.service: Failed with result
>> 'signal'.
>>
>>
>>    cat /proc/sys/kernel/core_pattern
>>    systemd-coredump
>>
>>
>>    ulimit -a
>>    core file size          (blocks, -c) unlimited
>>    data seg size           (kbytes, -d) unlimited
>>    scheduling priority             (-e) 0
>>    file size               (blocks, -f) unlimited
>>    pending signals                 (-i) 15657
>>    max locked memory       (kbytes, -l) 64
>>    max memory size         (kbytes, -m) unlimited
>>    open files                      (-n) 1024
>>    pipe size            (512 bytes, -p) 8
>>    POSIX message queues     (bytes, -q) 819200
>>    real-time priority              (-r) 0
>>    stack size              (kbytes, -s) 8192
>>    cpu time               (seconds, -t) unlimited
>>    max user processes              (-u) 15657
>>    virtual memory          (kbytes, -v) unlimited
>>    file locks                      (-x) unlimited
>>
>>    cd /var/lib/systemd/coredump/
>>    root@localhost:/var/lib/systemd/coredump# ls
>>    root@localhost:/var/lib/systemd/coredump#
>>
>>
>> (2) How to enable debugs? I have used 'make build' but no additional
>> logs other than those shown below
>>
>>
>> VPP logs from /var/log/syslog is shown below
>> cat /var/log/syslog
>> May 27 11:40:28 localhost vpp[6818]: vlib_plugin_early_init:361:
>> plugin path /usr/lib/vpp_plugins:/usr/lib64/vpp_plugins
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: abf_plugin.so (ACL based Forwarding)
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: acl_plugin.so (Access Control Lists)
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: avf_plugin.so (Intel Adaptive Virtual Function (AVF) Device
>> Plugin)
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:191: Loaded
>> plugin: cdp_plugin.so
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: dpdk_plugin.so (Data Plane Development Kit (DPDK))
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: flowprobe_plugin.so (Flow per Packet)
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: gbp_plugin.so (Group Based Policy)
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: gtpu_plugin.so (GTPv1-U)
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: igmp_plugin.so (IGMP messaging)
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: ila_plugin.so (Identifier-locator addressing for IPv6)
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: ioam_plugin.so (Inbound OAM)
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:117: Plugin
>> disabled (default): ixge_plugin.so
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: l2e_plugin.so (L2 Emulation)
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: lacp_plugin.so (Link Aggregation Control Protocol)
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: lb_plugin.so (Load Balancer)
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: memif_plugin.so (Packet Memory Interface (experimetal))
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: nat_plugin.so (Network Address Translation)
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: pppoe_plugin.so (PPPoE)
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: srv6ad_plugin.so (Dynamic SRv6 proxy)
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: srv6am_plugin.so (Masquerading SRv6 proxy)
>> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: srv6as_plugin.so (Static SRv6 proxy)
>> May 27 11:40:29 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: stn_plugin.so (VPP Steals the NIC for Container integration)
>> May 27 11:40:29 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: tlsmbedtls_plugin.so (mbedtls based TLS Engine)
>> May 27 11:40:29 localhost vpp[6818]: load_one_plugin:189: Loaded
>> plugin: tlsopenssl_plugin.so (openssl based TLS Engine)
>> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
>> load_one_vat_plugin:67: Loaded plugin: dpdk_test_plugin.so
>> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
>> Loaded plugin: dpdk_test_plugin.so
>> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
>> load_one_vat_plugin:67: Loaded plugin: lb_test_plugin.so
>> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
>> load_one_vat_plugin:67: Loaded plugin: flowprobe_test_plugin.so
>> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
>> load_one_vat_plugin:67: Loaded plugin: stn_test_plugin.so
>> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
>> load_one_vat_plugin:67: Loaded plugin: nat_test_plugin.so
>> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
>> load_one_vat_plugin:67: Loaded plugin: udp_ping_test_plugin.so
>> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
>> load_one_vat_plugin:67: Loaded plugin: pppoe_test_plugin.so
>> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
>> load_one_vat_plugin:67: Loaded plugin: lacp_test_plugin.so
>> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
>> Loaded plugin: lb_test_plugin.so
>> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
>> load_one_vat_plugin:67: Loaded plugin: acl_test_plugin.so
>> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
>> load_one_vat_plugin:67: Loaded plugin: ioam_export_test_plugin.so
>> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
>> load_one_vat_plugin:67: Loaded plugin: ioam_trace_test_plugin.so
>> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
>> load_one_vat_plugin:67: Loaded plugin:
>> vxlan_gpe_ioam_export_test_plugin.so
>> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
>> load_one_vat_plugin:67: Loaded plugin: gtpu_test_plugin.so
>> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
>> load_one_vat_plugin:67: Loaded plugin: cdp_test_plugin.so
>> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
>> load_one_vat_plugin:67: Loaded plugin: ioam_vxlan_gpe_test_plugin.so
>> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
>> load_one_vat_plugin:67: Loaded plugin: memif_test_plugin.so
>> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
>> load_one_vat_plugin:67: Loaded plugin: ioam_pot_test_plugin.so
>> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
>> Loaded plugin: flowprobe_test_plugin.so
>> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
>> Loaded plugin: stn_test_plugin.so
>> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
>> Loaded plugin: nat_test_plugin.so
>> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
>> Loaded plugin: udp_ping_test_plugin.so
>> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
>> Loaded plugin: pppoe_test_plugin.so
>> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
>> Loaded plugin: lacp_test_plugin.so
>> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
>> Loaded plugin: acl_test_plugin.so
>> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
>> Loaded plugin: ioam_export_test_plugin.so
>> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
>> Loaded plugin: ioam_trace_test_plugin.so
>> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
>> Loaded plugin: vxlan_gpe_ioam_export_test_plugin.so
>> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
>> Loaded plugin: gtpu_test_plugin.so
>> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
>> Loaded plugin: cdp_test_plugin.so
>> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
>> Loaded plugin: ioam_vxlan_gpe_test_plugin.so
>> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
>> Loaded plugin: memif_test_plugin.so
>> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
>> Loaded plugin: ioam_pot_test_plugin.so
>> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]: dpdk: EAL
>> init args: -c 1 -n 4 --no-pci --huge-dir /dev/hugepages --master-lcore
>> 0 --socket-mem 256,0
>> May 27 11:40:29 localhost /usr/bin/vpp[6818]: dpdk: EAL init args: -c
>> 1 -n 4 --no-pci --huge-dir /dev/hugepages --master-lcore 0
>> --socket-mem 256,0
>> May 27 11:40:29 localhost vnet[6818]: dpdk_ipsec_process:1019: not
>> enough DPDK crypto resources, default to OpenSSL
>> May 27 11:43:19 localhost vnet[6818]: show vhost-user: unknown input `detail
>> May 27 11:44:00 localhost vnet[6818]: received signal SIGSEGV, PC
>> 0x7fcca4620187, faulting address 0x7fcb317ac000
>> May 27 11:44:00 localhost systemd[1]: vpp.service: Main process
>> exited, code=killed, status=6/ABRT
>> May 27 11:44:00 localhost systemd[1]: vpp.service: Unit entered failed
>> state.
>> May 27 11:44:00 localhost systemd[1]: vpp.service: Failed with result
>> 'signal'.
>> May 27 11:44:00 localhost systemd[1]: vpp.service: Service hold-off
>> time over, scheduling restart
>>
>>
>>    Thanks,
>>    Ravi
>>
>>
>>
>> Thanks.
>>
>> Cheers,
>> Marco
>>
>>
>>
>>
>> --
>> Marco V
>>
>> SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
>> HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> 
>

-=-=-=-=-=-=-=-=-=-=-=-
Links:

You receive all messages sent to this group.

View/Reply Online (#9537): https://lists.fd.io/g/vpp-dev/message/9537
View All Messages In Topic (27): https://lists.fd.io/g/vpp-dev/topic/20346431
Mute This Topic: https://lists.fd.io/mt/20346431/21656
New Topic: https://lists.fd.io/g/vpp-dev/post

Change Your Subscription: https://lists.fd.io/g/vpp-dev/editsub/21656
Group Home: https://lists.fd.io/g/vpp-dev
Contact Group Owner: vpp-dev+ow...@lists.fd.io
Terms of Service: https://lists.fd.io/static/tos
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub
Email sent to: arch...@mail-archive.com
-=-=-=-=-=-=-=-=-=-=-=-

Reply via email to