Hi Steven,
You made my day, thank you. I didn't realize different dpdk versions
(vpp -- 18.02.1 and testpmd -- from latest git repo (probably 18.05)
could be the cause of the problem, I still dont understand why it
should as virtio/vhost messages are meant to setup tx/rx rings
correctly?
I downloaded dpdk 18.02.1 stable release and at least vpp doesn't
crash now (for both vpp-native and dpdk vhost interfaces). I have one
question is there a way to read vhost-user statistics counter (Rx/Tx)
on vpp? I only know
'show vhost-user <intf>' and 'show vhost-user <intf> descriptors'
which doesn't show any counters.
Thanks.
On Thu, May 31, 2018 at 11:51 AM, Steven Luong (sluong)
<slu...@cisco.com> wrote:
> Ravi,
>
> For (1) which works, what dpdk version are you using in the host? Are you
> using the same dpdk version as VPP is using? Since you are using VPP latest,
> I think it is 18.02. Type "show dpdk version" at the VPP prompt to find out
> for sure.
>
> Steven
>
> On 5/31/18, 11:44 AM, "Ravi Kerur" <rke...@gmail.com> wrote:
>
> Hi Steven,
>
> i have tested following scenarios and it basically is not clear why
> you think DPDK is the problem? Is it possible VPP and DPDK use
> different virtio versions?
>
> Following are the scenarios I have tested
>
> (1) testpmd/DPDK vhost-user (running on host) and testpmd/DPDK
> virito-user (in a container) -- can send and receive packets
> (2) VPP-native vhost-user (running on host) and testpmd/DPDK
> virtio-user (in a container) -- VPP crashes and it is in VPP code
> (3) VPP-DPDK vhost user (running on host) and testpmd/DPDK virtio-user
> (in a container) -- VPP crashes and in DPDK
>
> Thanks.
>
> On Thu, May 31, 2018 at 10:12 AM, Steven Luong (sluong)
> <slu...@cisco.com> wrote:
> > Ravi,
> >
> > I've proved my point -- there is a problem in the way that you invoke
> testpmd. The shared memory region that it passes to the device is not
> accessible from the device. I don't know what the correct options are that
> you need to use. This is really a question for dpdk.
> >
> > As a further exercise, you could remove VPP in the host and instead run
> testpmd in device mode using "--vdev
> net_vhost0,iface=/var/run/vpp/sock1.sock" option. I bet you testpmd in the
> host will crash in the same place. I hope you can find out the answer from
> dpdk and tell us about it.
> >
> > Steven
> >
> > On 5/31/18, 9:31 AM, "vpp-dev@lists.fd.io on behalf of Ravi Kerur"
> <vpp-dev@lists.fd.io on behalf of rke...@gmail.com> wrote:
> >
> > Hi Steven,
> >
> > Thank you for your help, I removed sock1.sock and sock2.sock,
> > restarted vpp, atleast interfaces get created. However, when I start
> > dpdk/testpmd inside the container it crashes as well. Below are some
> > details. I am using vpp code from latest repo.
> >
> > (1) On host
> > show interface
> > Name Idx State Counter
> > Count
> > VhostEthernet2 3 down
> > VhostEthernet3 4 down
> > VirtualFunctionEthernet4/10/4 1 down
> > VirtualFunctionEthernet4/10/6 2 down
> > local0 0 down
> > vpp#
> > vpp# set interface state VhostEthernet2 up
> > vpp# set interface state VhostEthernet3 up
> > vpp#
> > vpp# set interface l2 bridge VhostEthernet2 1
> > vpp# set interface l2 bridge VhostEthernet3 1
> > vpp#
> >
> > (2) Run tespmd inside the container
> > docker run -it --privileged -v
> > /var/run/vpp/sock1.sock:/var/run/usvhost1 -v
> > /var/run/vpp/sock2.sock:/var/run/usvhost2 -v
> > /dev/hugepages:/dev/hugepages dpdk-app-testpmd ./bin/testpmd -l
> 16-19
> > -n 4 --log-level=8 -m 64 --no-pci
> > --vdev=virtio_user0,path=/var/run/usvhost1,mac=54:00:00:01:01:01
> > --vdev=virtio_user1,path=/var/run/usvhost2,mac=54:00:00:01:01:02 --
> > -i
> > EAL: Detected 28 lcore(s)
> > EAL: Detected 2 NUMA nodes
> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > EAL: 8192 hugepages of size 2097152 reserved, but no mounted
> hugetlbfs
> > found for that size
> > EAL: Probing VFIO support...
> > EAL: VFIO support initialized
> > EAL: Setting up physically contiguous memory...
> > EAL: locking hot plug lock memory...
> > EAL: primary init32...
> > Interactive-mode selected
> > Warning: NUMA should be configured manually by using
> > --port-numa-config and --ring-numa-config parameters along with
> > --numa.
> > testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456,
> > size=2176, socket=0
> > testpmd: preferred mempool ops selected: ring_mp_mc
> > testpmd: create a new mbuf pool <mbuf_pool_socket_1>: n=171456,
> > size=2176, socket=1
> > testpmd: preferred mempool ops selected: ring_mp_mc
> > Port 0 is now not stopped
> > Port 1 is now not stopped
> > Please stop the ports first
> > Done
> > testpmd>
> >
> > (3) VPP crashes with the same issue but inside dpdk code
> >
> > (gdb) cont
> > Continuing.
> >
> > Program received signal SIGSEGV, Segmentation fault.
> > [Switching to Thread 0x7ffd0d08e700 (LWP 41257)]
> > rte_vhost_dequeue_burst (vid=<optimized out>, queue_id=<optimized
> > out>, mbuf_pool=0x7fe17fc883c0,
> > pkts=pkts@entry=0x7fffb671ebc0, count=count@entry=32)
> > at
> /var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_vhost/virtio_net.c:1504
> > 1504 free_entries = *((volatile uint16_t *)&vq->avail->idx) -
> > (gdb) bt
> > #0 rte_vhost_dequeue_burst (vid=<optimized out>,
> queue_id=<optimized out>,
> > mbuf_pool=0x7fe17fc883c0, pkts=pkts@entry=0x7fffb671ebc0,
> > count=count@entry=32)
> > at
> /var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_vhost/virtio_net.c:1504
> > #1 0x00007fffb4718e6f in eth_vhost_rx (q=0x7fe17fbbdd80,
> bufs=0x7fffb671ebc0,
> > nb_bufs=<optimized out>)
> > at
> /var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/drivers/net/vhost/rte_eth_vhost.c:410
> > #2 0x00007fffb441cb7c in rte_eth_rx_burst (nb_pkts=256,
> > rx_pkts=0x7fffb671ebc0, queue_id=0,
> > port_id=3) at
> >
> /var/venom/vpp/build-root/install-vpp-native/dpdk/include/dpdk/rte_ethdev.h:3635
> > #3 dpdk_device_input (queue_id=0, thread_index=<optimized out>,
> > node=0x7fffb732c700,
> > xd=0x7fffb7337240, dm=<optimized out>, vm=0x7fffb6703340)
> > at
> /var/venom/vpp/build-data/../src/plugins/dpdk/device/node.c:477
> > #4 dpdk_input_node_fn_avx2 (vm=<optimized out>, node=<optimized
> out>,
> > f=<optimized out>)
> > at
> /var/venom/vpp/build-data/../src/plugins/dpdk/device/node.c:658
> > #5 0x00007ffff7954d35 in dispatch_node
> > (last_time_stamp=12531752723928016, frame=0x0,
> > dispatch_state=VLIB_NODE_STATE_POLLING,
> type=VLIB_NODE_TYPE_INPUT,
> > node=0x7fffb732c700,
> > vm=0x7fffb6703340) at
> /var/venom/vpp/build-data/../src/vlib/main.c:988
> > #6 vlib_main_or_worker_loop (is_main=0, vm=0x7fffb6703340)
> > at /var/venom/vpp/build-data/../src/vlib/main.c:1507
> > #7 vlib_worker_loop (vm=0x7fffb6703340) at
> > /var/venom/vpp/build-data/../src/vlib/main.c:1641
> > #8 0x00007ffff6ad25d8 in clib_calljmp ()
> > at /var/venom/vpp/build-data/../src/vppinfra/longjmp.S:110
> > #9 0x00007ffd0d08ddb0 in ?? ()
> > #10 0x00007fffb4436edd in eal_thread_loop (arg=<optimized out>)
> > at
> /var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_eal/linuxapp/eal/eal_thread.c:153
> > #11 0x0000000000000000 in ?? ()
> > (gdb) frame 0
> > #0 rte_vhost_dequeue_burst (vid=<optimized out>,
> queue_id=<optimized out>,
> > mbuf_pool=0x7fe17fc883c0, pkts=pkts@entry=0x7fffb671ebc0,
> > count=count@entry=32)
> > at
> /var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_vhost/virtio_net.c:1504
> > 1504 free_entries = *((volatile uint16_t *)&vq->avail->idx) -
> > (gdb) p vq
> > $1 = (struct vhost_virtqueue *) 0x7fc3ffc84b00
> > (gdb) p vq->avail
> > $2 = (struct vring_avail *) 0x7ffbfff98000
> > (gdb) p *$2
> > Cannot access memory at address 0x7ffbfff98000
> > (gdb)
> >
> >
> > Thanks.
> >
> > On Thu, May 31, 2018 at 12:09 AM, Steven Luong (sluong)
> > <slu...@cisco.com> wrote:
> > > Sorry, I was expecting to see two VhostEthernet interfaces like
> this. Those VirtualFunctionEthernet are your physical interfaces.
> > >
> > > sh int
> > > Name Idx State Counter
> Count
> > > VhostEthernet0 1 up
> > > VhostEthernet1 2 up
> > > local0 0 down
> > > DBGvpp#
> > >
> > > You have to first manually remove /var/run/vpp/sock1.sock and
> /var/run/vpp/sock2.sock before you start vpp on the host. dpdk does not like
> it if they already existed. If you successfully create VhostEthernet
> interface, try to send some traffic through it to see if it crashes or not.
> > >
> > > Steven
> > >
> > > On 5/30/18, 9:17 PM, "vpp-dev@lists.fd.io on behalf of Steven
> Luong (sluong)" <vpp-dev@lists.fd.io on behalf of slu...@cisco.com> wrote:
> > >
> > > Ravi,
> > >
> > > I don't think you can declare (2) works fine yet. Please
> bring up the dpdk vhost-user interfaces and try to send some traffic between
> them to exercise the shared memory region from dpdk virtio-user which may be
> "questionable".
> > >
> > > VirtualFunctionEthernet4/10/4 1 down
> > > VirtualFunctionEthernet4/10/6 2 down
> > >
> > > Steven
> > >
> > > On 5/30/18, 4:41 PM, "Ravi Kerur" <rke...@gmail.com> wrote:
> > >
> > > Hi Steve,
> > >
> > > Thank you for your inputs, I added feature-mask to see if
> it helps in
> > > setting up queues correctly, it didn't so I will remove
> it. I have
> > > tried following combination
> > >
> > > (1) VPP->vhost-user (on host) and
> DPDK/testpmd->virtio-user (in a
> > > container) -- VPP crashes
> > > (2) DPDK/testpmd->vhost-user (on host) and
> DPDK/testpmd->virtio-user
> > > (in a container) -- works fine
> > >
> > > To use DPDK vhost-user inside VPP, I defined
> configuration in
> > > startup.conf as mentioned by you and it looks as follows
> > >
> > > unix {
> > > nodaemon
> > > log /var/log/vpp/vpp.log
> > > full-coredump
> > > cli-listen /run/vpp/cli.sock
> > > gid vpp
> > > }
> > >
> > > api-segment {
> > > gid vpp
> > > }
> > >
> > > cpu {
> > > main-core 1
> > > corelist-workers 6-9
> > > }
> > >
> > > dpdk {
> > > dev 0000:04:10.4
> > > dev 0000:04:10.6
> > > uio-driver vfio-pci
> > > vdev net_vhost0,iface=/var/run/vpp/sock1.sock
> > > vdev net_vhost1,iface=/var/run/vpp/sock2.sock
> > > huge-dir /dev/hugepages_1GB
> > > socket-mem 2048,2048
> > > }
> > >
> > > From VPP logs
> > > dpdk: EAL init args: -c 3c2 -n 4 --vdev
> > > net_vhost0,iface=/var/run/vpp/sock1.sock --vdev
> > > net_vhost1,iface=/var/run/vpp/sock2.sock --huge-dir
> /dev/hugepages_1GB
> > > -w 0000:04:10.4 -w 0000:04:10.6 --master-lcore 1
> --socket-mem
> > > 2048,2048
> > >
> > > However, VPP doesn't create interface at all
> > >
> > > vpp# show interface
> > > Name Idx State
> Counter
> > > Count
> > > VirtualFunctionEthernet4/10/4 1 down
> > > VirtualFunctionEthernet4/10/6 2 down
> > > local0 0 down
> > >
> > > since it is a static mapping I am assuming it should be
> created, correct?
> > >
> > > Thanks.
> > >
> > > On Wed, May 30, 2018 at 3:43 PM, Steven Luong (sluong)
> <slu...@cisco.com> wrote:
> > > > Ravi,
> > > >
> > > > First and foremost, get rid of the feature-mask option.
> I don't know what 0x40400000 does for you. If that does not help, try testing
> it with dpdk based vhost-user instead of VPP native vhost-user to make sure
> that they can work well with each other first. To use dpdk vhost-user, add a
> vdev command in the startup.conf for each vhost-user device that you have.
> > > >
> > > > dpdk { vdev net_vhost0,iface=/var/run/vpp/sock1.sock }
> > > >
> > > > dpdk based vhost-user interface is named
> VhostEthernet0, VhostEthernet1, etc. Make sure you use the right interface
> name to set the state to up.
> > > >
> > > > If dpdk based vhost-user does not work with testpmd
> either, it looks like some problem with the way that you invoke testpmd.
> > > >
> > > > If dpdk based vhost-user works well with the same
> testpmd device driver and not vpp native vhost-user, I can set up something
> similar to yours to look into it.
> > > >
> > > > The device driver, testpmd, is supposed to pass the
> shared memory region to VPP for TX/RX queues. It looks like VPP vhost-user
> might have run into a bump there with using the shared memory (txvq->avail).
> > > >
> > > > Steven
> > > >
> > > > PS. vhost-user is not an optimum interface for
> containers. You may want to look into using memif if you don't already know
> about it.
> > > >
> > > >
> > > > On 5/30/18, 2:06 PM, "Ravi Kerur" <rke...@gmail.com>
> wrote:
> > > >
> > > > I am not sure what is wrong with the setup or a bug
> in vpp, vpp
> > > > crashes with vhost<-->virtio communication.
> > > >
> > > > (1) Vhost-interfaces are created and attached to
> bridge-domain as follows
> > > >
> > > > create vhost socket /var/run/vpp/sock1.sock server
> feature-mask 0x40400000
> > > > create vhost socket /var/run/vpp/sock2.sock server
> feature-mask 0x40400000
> > > > set interface state VirtualEthernet0/0/0 up
> > > > set interface state VirtualEthernet0/0/1 up
> > > >
> > > > set interface l2 bridge VirtualEthernet0/0/0 1
> > > > set interface l2 bridge VirtualEthernet0/0/1 1
> > > >
> > > >
> > > > (2) DPDK/testpmd is started in a container to talk
> to vpp/vhost-user
> > > > interface as follows
> > > >
> > > > docker run -it --privileged -v
> > > > /var/run/vpp/sock1.sock:/var/run/usvhost1 -v
> > > > /var/run/vpp/sock2.sock:/var/run/usvhost2 -v
> > > > /dev/hugepages:/dev/hugepages dpdk-app-testpmd
> ./bin/testpmd -c 0x3 -n
> > > > 4 --log-level=9 -m 64 --no-pci
> --single-file-segments
> > > >
> --vdev=virtio_user0,path=/var/run/usvhost1,mac=54:00:00:01:01:01
> > > >
> --vdev=virtio_user1,path=/var/run/usvhost2,mac=54:00:00:01:01:02 --
> > > > -i
> > > >
> > > > (3) show vhost-user VirtualEthernet0/0/1
> > > > Virtio vhost-user interfaces
> > > > Global:
> > > > coalesce frames 32 time 1e-3
> > > > number of rx virtqueues in interrupt mode: 0
> > > > Interface: VirtualEthernet0/0/1 (ifindex 4)
> > > > virtio_net_hdr_sz 10
> > > > features mask (0x40400000):
> > > > features (0x0):
> > > > protocol features (0x0)
> > > >
> > > > socket filename /var/run/vpp/sock2.sock type
> server errno "Success"
> > > >
> > > > rx placement:
> > > > tx placement: spin-lock
> > > > thread 0 on vring 0
> > > > thread 1 on vring 0
> > > > thread 2 on vring 0
> > > > thread 3 on vring 0
> > > > thread 4 on vring 0
> > > >
> > > > Memory regions (total 1)
> > > > region fd guest_phys_addr memory_size
> userspace_addr
> > > > mmap_offset mmap_addr
> > > > ====== ===== ================== ==================
> ==================
> > > > ================== ==================
> > > > 0 55 0x00007ff7c0000000 0x0000000040000000
> 0x00007ff7c0000000
> > > > 0x0000000000000000 0x00007ffbc0000000
> > > >
> > > > vpp# show vhost-user VirtualEthernet0/0/0
> > > > Virtio vhost-user interfaces
> > > > Global:
> > > > coalesce frames 32 time 1e-3
> > > > number of rx virtqueues in interrupt mode: 0
> > > > Interface: VirtualEthernet0/0/0 (ifindex 3)
> > > > virtio_net_hdr_sz 10
> > > > features mask (0x40400000):
> > > > features (0x0):
> > > > protocol features (0x0)
> > > >
> > > > socket filename /var/run/vpp/sock1.sock type
> server errno "Success"
> > > >
> > > > rx placement:
> > > > tx placement: spin-lock
> > > > thread 0 on vring 0
> > > > thread 1 on vring 0
> > > > thread 2 on vring 0
> > > > thread 3 on vring 0
> > > > thread 4 on vring 0
> > > >
> > > > Memory regions (total 1)
> > > > region fd guest_phys_addr memory_size
> userspace_addr
> > > > mmap_offset mmap_addr
> > > > ====== ===== ================== ==================
> ==================
> > > > ================== ==================
> > > > 0 51 0x00007ff7c0000000 0x0000000040000000
> 0x00007ff7c0000000
> > > > 0x0000000000000000 0x00007ffc00000000
> > > >
> > > > (4) vpp stack trace
> > > > Program received signal SIGSEGV, Segmentation fault.
> > > > [Switching to Thread 0x7ffd0e090700 (LWP 46570)]
> > > > 0x00007ffff7414642 in vhost_user_if_input
> > > > (mode=VNET_HW_INTERFACE_RX_MODE_POLLING,
> > > > node=0x7fffb76bab00, qid=<optimized out>,
> vui=0x7fffb6739700,
> > > > vum=0x7ffff78f4480 <vhost_user_main>,
> vm=0x7fffb672a9c0)
> > > > at
> /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1596
> > > > 1596 if (PREDICT_FALSE (txvq->avail->flags &
> 0xFFFE))
> > > > (gdb) bt
> > > > #0 0x00007ffff7414642 in vhost_user_if_input
> > > > (mode=VNET_HW_INTERFACE_RX_MODE_POLLING,
> > > > node=0x7fffb76bab00, qid=<optimized out>,
> vui=0x7fffb6739700,
> > > > vum=0x7ffff78f4480 <vhost_user_main>,
> vm=0x7fffb672a9c0)
> > > > at
> /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1596
> > > > #1 vhost_user_input (f=<optimized out>,
> node=<optimized out>,
> > > > vm=<optimized out>)
> > > > at
> /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1947
> > > > #2 vhost_user_input_avx2 (vm=<optimized out>,
> node=<optimized out>,
> > > > frame=<optimized out>)
> > > > at
> /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1972
> > > > #3 0x00007ffff7954d35 in dispatch_node
> > > > (last_time_stamp=12391212490024174, frame=0x0,
> > > > dispatch_state=VLIB_NODE_STATE_POLLING,
> type=VLIB_NODE_TYPE_INPUT,
> > > > node=0x7fffb76bab00,
> > > > vm=0x7fffb672a9c0) at
> /var/venom/vpp/build-data/../src/vlib/main.c:988
> > > > #4 vlib_main_or_worker_loop (is_main=0,
> vm=0x7fffb672a9c0)
> > > > at
> /var/venom/vpp/build-data/../src/vlib/main.c:1507
> > > > #5 vlib_worker_loop (vm=0x7fffb672a9c0) at
> > > > /var/venom/vpp/build-data/../src/vlib/main.c:1641
> > > > #6 0x00007ffff6ad25d8 in clib_calljmp ()
> > > > at
> /var/venom/vpp/build-data/../src/vppinfra/longjmp.S:110
> > > > #7 0x00007ffd0e08fdb0 in ?? ()
> > > > #8 0x00007fffb4436edd in eal_thread_loop
> (arg=<optimized out>)
> > > > at
> /var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_eal/linuxapp/eal/eal_thread.c:153
> > > > #9 0x0000000000000000 in ?? ()
> > > > (gdb) frame 0
> > > > #0 0x00007ffff7414642 in vhost_user_if_input
> > > > (mode=VNET_HW_INTERFACE_RX_MODE_POLLING,
> > > > node=0x7fffb76bab00, qid=<optimized out>,
> vui=0x7fffb6739700,
> > > > vum=0x7ffff78f4480 <vhost_user_main>,
> vm=0x7fffb672a9c0)
> > > > at
> /var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1596
> > > > 1596 if (PREDICT_FALSE (txvq->avail->flags &
> 0xFFFE))
> > > > (gdb) p txvq
> > > > $1 = (vhost_user_vring_t *) 0x7fffb6739ac0
> > > > (gdb) p *txvq
> > > > $2 = {cacheline0 = 0x7fffb6739ac0 "?", qsz_mask =
> 255, last_avail_idx
> > > > = 0, last_used_idx = 0,
> > > > n_since_last_int = 0, desc = 0x7ffbfff97000,
> avail = 0x7ffbfff98000,
> > > > used = 0x7ffbfff99000,
> > > > int_deadline = 0, started = 1 '\001', enabled = 0
> '\000', log_used = 0 '\000',
> > > > cacheline1 = 0x7fffb6739b00 "????\n", errfd = -1,
> callfd_idx = 10,
> > > > kickfd_idx = 14,
> > > > log_guest_addr = 0, mode = 1}
> > > > (gdb) p *(txvq->avail)
> > > > Cannot access memory at address 0x7ffbfff98000
> > > > (gdb)
> > > >
> > > > On Tue, May 29, 2018 at 10:47 AM, Ravi Kerur
> <rke...@gmail.com> wrote:
> > > > > Steve,
> > > > >
> > > > > Thanks for inputs on debugs and gdb. I am using
> gdb on my development
> > > > > system to debug the issue. I would like to have
> reliable core
> > > > > generation on the system on which I don't have
> access to install gdb.
> > > > > I installed corekeeper and it still doesn't
> generate core. I am
> > > > > running vpp inside a VM (VirtualBox/vagrant), not
> sure if I need to
> > > > > set something inside vagrant config file.
> > > > >
> > > > > dpkg -l corekeeper
> > > > > Desired=Unknown/Install/Remove/Purge/Hold
> > > > > |
> Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> > > > > |/ Err?=(none)/Reinst-required (Status,Err:
> uppercase=bad)
> > > > > ||/ Name Version
> Architecture Description
> > > > >
> +++-====================-===============-===============-==============================================
> > > > > ii corekeeper 1.6 amd64
> enable core
> > > > > files and report crashes to the system
> > > > >
> > > > > Thanks.
> > > > >
> > > > > On Tue, May 29, 2018 at 9:38 AM, Steven Luong
> (sluong) <slu...@cisco.com> wrote:
> > > > >> Ravi,
> > > > >>
> > > > >> I install corekeeper and the core file is kept
> in /var/crash. But why not use gdb to attach to the VPP process?
> > > > >> To turn on VPP vhost-user debug, type "debug
> vhost-user on" at the VPP prompt.
> > > > >>
> > > > >> Steven
> > > > >>
> > > > >> On 5/29/18, 9:10 AM, "vpp-dev@lists.fd.io on
> behalf of Ravi Kerur" <vpp-dev@lists.fd.io on behalf of rke...@gmail.com>
> wrote:
> > > > >>
> > > > >> Hi Marco,
> > > > >>
> > > > >>
> > > > >> On Tue, May 29, 2018 at 6:30 AM, Marco
> Varlese <mvarl...@suse.de> wrote:
> > > > >> > Ravi,
> > > > >> >
> > > > >> > On Sun, 2018-05-27 at 12:20 -0700, Ravi
> Kerur wrote:
> > > > >> >> Hello,
> > > > >> >>
> > > > >> >> I have a VM(16.04.4 Ubuntu x86_64) with 2
> cores and 4G RAM. I have
> > > > >> >> installed VPP successfully on it. Later I
> have created vhost-user
> > > > >> >> interfaces via
> > > > >> >>
> > > > >> >> create vhost socket
> /var/run/vpp/sock1.sock server
> > > > >> >> create vhost socket
> /var/run/vpp/sock2.sock server
> > > > >> >> set interface state VirtualEthernet0/0/0
> up
> > > > >> >> set interface state VirtualEthernet0/0/1
> up
> > > > >> >>
> > > > >> >> set interface l2 bridge
> VirtualEthernet0/0/0 1
> > > > >> >> set interface l2 bridge
> VirtualEthernet0/0/1 1
> > > > >> >>
> > > > >> >> I then run 'DPDK/testpmd' inside a
> container which will use
> > > > >> >> virtio-user interfaces using the
> following command
> > > > >> >>
> > > > >> >> docker run -it --privileged -v
> > > > >> >> /var/run/vpp/sock1.sock:/var/run/usvhost1
> -v
> > > > >> >> /var/run/vpp/sock2.sock:/var/run/usvhost2
> -v
> > > > >> >> /dev/hugepages:/dev/hugepages
> dpdk-app-testpmd ./bin/testpmd -c 0x3 -n
> > > > >> >> 4 --log-level=9 -m 64 --no-pci
> --single-file-segments
> > > > >> >>
> --vdev=virtio_user0,path=/var/run/usvhost1,mac=54:01:00:01:01:01
> > > > >> >>
> --vdev=virtio_user1,path=/var/run/usvhost2,mac=54:01:00:01:01:02 --
> > > > >> >> -i
> > > > >> >>
> > > > >> >> VPP Vnet crashes with following message
> > > > >> >>
> > > > >> >> May 27 11:44:00 localhost vnet[6818]:
> received signal SIGSEGV, PC
> > > > >> >> 0x7fcca4620187, faulting address
> 0x7fcb317ac000
> > > > >> >>
> > > > >> >> Questions:
> > > > >> >> I have 'ulimit -c unlimited' and
> /etc/vpp/startup.conf has
> > > > >> >> unix {
> > > > >> >> nodaemon
> > > > >> >> log /var/log/vpp/vpp.log
> > > > >> >> full-coredump
> > > > >> >> cli-listen /run/vpp/cli.sock
> > > > >> >> gid vpp
> > > > >> >> }
> > > > >> >>
> > > > >> >> But I couldn't locate corefile?
> > > > >> > The location of the coredump file depends
> on your system configuration.
> > > > >> >
> > > > >> > Please, check "cat
> /proc/sys/kernel/core_pattern"
> > > > >> >
> > > > >> > If you have systemd-coredump in the output
> of the above command, then likely the
> > > > >> > location of the coredump files is
> "/var/lib/systemd/coredump/"
> > > > >> >
> > > > >> > You can also change the location of where
> your system places the coredump files:
> > > > >> > echo '/PATH_TO_YOU_LOCATION/core_%e.%p' |
> sudo tee /proc/sys/kernel/core_pattern
> > > > >> >
> > > > >> > See if that helps...
> > > > >> >
> > > > >>
> > > > >> Initially '/proc/sys/kernel/core_pattern'
> was set to 'core'. I changed
> > > > >> it to 'systemd-coredump'. Still no core
> generated. VPP crashes
> > > > >>
> > > > >> May 29 08:54:34 localhost vnet[4107]:
> received signal SIGSEGV, PC
> > > > >> 0x7f0167751187, faulting address
> 0x7efff43ac000
> > > > >> May 29 08:54:34 localhost systemd[1]:
> vpp.service: Main process
> > > > >> exited, code=killed, status=6/ABRT
> > > > >> May 29 08:54:34 localhost systemd[1]:
> vpp.service: Unit entered failed state.
> > > > >> May 29 08:54:34 localhost systemd[1]:
> vpp.service: Failed with result 'signal'.
> > > > >>
> > > > >>
> > > > >> cat /proc/sys/kernel/core_pattern
> > > > >> systemd-coredump
> > > > >>
> > > > >>
> > > > >> ulimit -a
> > > > >> core file size (blocks, -c)
> unlimited
> > > > >> data seg size (kbytes, -d)
> unlimited
> > > > >> scheduling priority (-e) 0
> > > > >> file size (blocks, -f)
> unlimited
> > > > >> pending signals (-i) 15657
> > > > >> max locked memory (kbytes, -l) 64
> > > > >> max memory size (kbytes, -m)
> unlimited
> > > > >> open files (-n) 1024
> > > > >> pipe size (512 bytes, -p) 8
> > > > >> POSIX message queues (bytes, -q) 819200
> > > > >> real-time priority (-r) 0
> > > > >> stack size (kbytes, -s) 8192
> > > > >> cpu time (seconds, -t)
> unlimited
> > > > >> max user processes (-u) 15657
> > > > >> virtual memory (kbytes, -v)
> unlimited
> > > > >> file locks (-x)
> unlimited
> > > > >>
> > > > >> cd /var/lib/systemd/coredump/
> > > > >> root@localhost:/var/lib/systemd/coredump# ls
> > > > >> root@localhost:/var/lib/systemd/coredump#
> > > > >>
> > > > >> >>
> > > > >> >> (2) How to enable debugs? I have used
> 'make build' but no additional
> > > > >> >> logs other than those shown below
> > > > >> >>
> > > > >> >>
> > > > >> >> VPP logs from /var/log/syslog is shown
> below
> > > > >> >> cat /var/log/syslog
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> vlib_plugin_early_init:361:
> > > > >> >> plugin path
> /usr/lib/vpp_plugins:/usr/lib64/vpp_plugins
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: abf_plugin.so (ACL based
> Forwarding)
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: acl_plugin.so (Access Control
> Lists)
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: avf_plugin.so (Intel Adaptive
> Virtual Function (AVF) Device
> > > > >> >> Plugin)
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:191: Loaded
> > > > >> >> plugin: cdp_plugin.so
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: dpdk_plugin.so (Data Plane
> Development Kit (DPDK))
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: flowprobe_plugin.so (Flow per
> Packet)
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: gbp_plugin.so (Group Based Policy)
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: gtpu_plugin.so (GTPv1-U)
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: igmp_plugin.so (IGMP messaging)
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: ila_plugin.so (Identifier-locator
> addressing for IPv6)
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: ioam_plugin.so (Inbound OAM)
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:117: Plugin
> > > > >> >> disabled (default): ixge_plugin.so
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: l2e_plugin.so (L2 Emulation)
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: lacp_plugin.so (Link Aggregation
> Control Protocol)
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: lb_plugin.so (Load Balancer)
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: memif_plugin.so (Packet Memory
> Interface (experimetal))
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: nat_plugin.so (Network Address
> Translation)
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: pppoe_plugin.so (PPPoE)
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: srv6ad_plugin.so (Dynamic SRv6
> proxy)
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: srv6am_plugin.so (Masquerading
> SRv6 proxy)
> > > > >> >> May 27 11:40:28 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: srv6as_plugin.so (Static SRv6
> proxy)
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: stn_plugin.so (VPP Steals the NIC
> for Container integration)
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: tlsmbedtls_plugin.so (mbedtls
> based TLS Engine)
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> load_one_plugin:189: Loaded
> > > > >> >> plugin: tlsopenssl_plugin.so (openssl
> based TLS Engine)
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> /usr/bin/vpp[6818]:
> > > > >> >> load_one_vat_plugin:67: Loaded plugin:
> dpdk_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost
> /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > >> >> Loaded plugin: dpdk_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> /usr/bin/vpp[6818]:
> > > > >> >> load_one_vat_plugin:67: Loaded plugin:
> lb_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> /usr/bin/vpp[6818]:
> > > > >> >> load_one_vat_plugin:67: Loaded plugin:
> flowprobe_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> /usr/bin/vpp[6818]:
> > > > >> >> load_one_vat_plugin:67: Loaded plugin:
> stn_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> /usr/bin/vpp[6818]:
> > > > >> >> load_one_vat_plugin:67: Loaded plugin:
> nat_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> /usr/bin/vpp[6818]:
> > > > >> >> load_one_vat_plugin:67: Loaded plugin:
> udp_ping_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> /usr/bin/vpp[6818]:
> > > > >> >> load_one_vat_plugin:67: Loaded plugin:
> pppoe_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> /usr/bin/vpp[6818]:
> > > > >> >> load_one_vat_plugin:67: Loaded plugin:
> lacp_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost
> /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > >> >> Loaded plugin: lb_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> /usr/bin/vpp[6818]:
> > > > >> >> load_one_vat_plugin:67: Loaded plugin:
> acl_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> /usr/bin/vpp[6818]:
> > > > >> >> load_one_vat_plugin:67: Loaded plugin:
> ioam_export_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> /usr/bin/vpp[6818]:
> > > > >> >> load_one_vat_plugin:67: Loaded plugin:
> ioam_trace_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> /usr/bin/vpp[6818]:
> > > > >> >> load_one_vat_plugin:67: Loaded plugin:
> > > > >> >> vxlan_gpe_ioam_export_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> /usr/bin/vpp[6818]:
> > > > >> >> load_one_vat_plugin:67: Loaded plugin:
> gtpu_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> /usr/bin/vpp[6818]:
> > > > >> >> load_one_vat_plugin:67: Loaded plugin:
> cdp_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> /usr/bin/vpp[6818]:
> > > > >> >> load_one_vat_plugin:67: Loaded plugin:
> ioam_vxlan_gpe_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> /usr/bin/vpp[6818]:
> > > > >> >> load_one_vat_plugin:67: Loaded plugin:
> memif_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> /usr/bin/vpp[6818]:
> > > > >> >> load_one_vat_plugin:67: Loaded plugin:
> ioam_pot_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost
> /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > >> >> Loaded plugin: flowprobe_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost
> /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > >> >> Loaded plugin: stn_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost
> /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > >> >> Loaded plugin: nat_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost
> /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > >> >> Loaded plugin: udp_ping_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost
> /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > >> >> Loaded plugin: pppoe_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost
> /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > >> >> Loaded plugin: lacp_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost
> /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > >> >> Loaded plugin: acl_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost
> /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > >> >> Loaded plugin: ioam_export_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost
> /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > >> >> Loaded plugin: ioam_trace_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost
> /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > >> >> Loaded plugin:
> vxlan_gpe_ioam_export_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost
> /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > >> >> Loaded plugin: gtpu_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost
> /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > >> >> Loaded plugin: cdp_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost
> /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > >> >> Loaded plugin:
> ioam_vxlan_gpe_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost
> /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > >> >> Loaded plugin: memif_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost
> /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > > >> >> Loaded plugin: ioam_pot_test_plugin.so
> > > > >> >> May 27 11:40:29 localhost vpp[6818]:
> /usr/bin/vpp[6818]: dpdk: EAL
> > > > >> >> init args: -c 1 -n 4 --no-pci --huge-dir
> /dev/hugepages --master-lcore
> > > > >> >> 0 --socket-mem 256,0
> > > > >> >> May 27 11:40:29 localhost
> /usr/bin/vpp[6818]: dpdk: EAL init args: -c
> > > > >> >> 1 -n 4 --no-pci --huge-dir /dev/hugepages
> --master-lcore 0
> > > > >> >> --socket-mem 256,0
> > > > >> >> May 27 11:40:29 localhost vnet[6818]:
> dpdk_ipsec_process:1019: not
> > > > >> >> enough DPDK crypto resources, default to
> OpenSSL
> > > > >> >> May 27 11:43:19 localhost vnet[6818]:
> show vhost-user: unknown input `detail
> > > > >> >> May 27 11:44:00 localhost vnet[6818]:
> received signal SIGSEGV, PC
> > > > >> >> 0x7fcca4620187, faulting address
> 0x7fcb317ac000
> > > > >> >> May 27 11:44:00 localhost systemd[1]:
> vpp.service: Main process
> > > > >> >> exited, code=killed, status=6/ABRT
> > > > >> >> May 27 11:44:00 localhost systemd[1]:
> vpp.service: Unit entered failed state.
> > > > >> >> May 27 11:44:00 localhost systemd[1]:
> vpp.service: Failed with result
> > > > >> >> 'signal'.
> > > > >> >> May 27 11:44:00 localhost systemd[1]:
> vpp.service: Service hold-off
> > > > >> >> time over, scheduling restart
> > > > >> >>
> > > > >>
> > > > >> Thanks,
> > > > >> Ravi
> > > > >>
> > > > >> >>
> > > > >> >>
> > > > >> >> Thanks.
> > > > >> > Cheers,
> > > > >> > Marco
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> > --
> > > > >> > Marco V
> > > > >> >
> > > > >> > SUSE LINUX GmbH | GF: Felix Imendörffer,
> Jane Smithard, Graham Norton
> > > > >> > HRB 21284 (AG Nürnberg) Maxfeldstr. 5,
> D-90409, Nürnberg
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links:
You receive all messages sent to this group.
View/Reply Online (#9486): https://lists.fd.io/g/vpp-dev/message/9486
View All Messages In Topic (16): https://lists.fd.io/g/vpp-dev/topic/20346431
Mute This Topic: https://lists.fd.io/mt/20346431/21656
New Topic: https://lists.fd.io/g/vpp-dev/post
Change Your Subscription: https://lists.fd.io/g/vpp-dev/editsub/21656
Group Home: https://lists.fd.io/g/vpp-dev
Contact Group Owner: vpp-dev+ow...@lists.fd.io
Terms of Service: https://lists.fd.io/static/tos
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub
Email sent to: arch...@mail-archive.com
-=-=-=-=-=-=-=-=-=-=-=-
