Steve,
Thanks for inputs on debugs and gdb. I am using gdb on my development
system to debug the issue. I would like to have reliable core
generation on the system on which I don't have access to install gdb.
I installed corekeeper and it still doesn't generate core. I am
running vpp inside a VM (VirtualBox/vagrant), not sure if I need to
set something inside vagrant config file.
dpkg -l corekeeper
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-====================-===============-===============-==============================================
ii corekeeper 1.6 amd64 enable core
files and report crashes to the system
Thanks.
On Tue, May 29, 2018 at 9:38 AM, Steven Luong (sluong) <slu...@cisco.com> wrote:
> Ravi,
>
> I install corekeeper and the core file is kept in /var/crash. But why not use
> gdb to attach to the VPP process?
> To turn on VPP vhost-user debug, type "debug vhost-user on" at the VPP prompt.
>
> Steven
>
> On 5/29/18, 9:10 AM, "vpp-dev@lists.fd.io on behalf of Ravi Kerur"
> <vpp-dev@lists.fd.io on behalf of rke...@gmail.com> wrote:
>
> Hi Marco,
>
>
> On Tue, May 29, 2018 at 6:30 AM, Marco Varlese <mvarl...@suse.de> wrote:
> > Ravi,
> >
> > On Sun, 2018-05-27 at 12:20 -0700, Ravi Kerur wrote:
> >> Hello,
> >>
> >> I have a VM(16.04.4 Ubuntu x86_64) with 2 cores and 4G RAM. I have
> >> installed VPP successfully on it. Later I have created vhost-user
> >> interfaces via
> >>
> >> create vhost socket /var/run/vpp/sock1.sock server
> >> create vhost socket /var/run/vpp/sock2.sock server
> >> set interface state VirtualEthernet0/0/0 up
> >> set interface state VirtualEthernet0/0/1 up
> >>
> >> set interface l2 bridge VirtualEthernet0/0/0 1
> >> set interface l2 bridge VirtualEthernet0/0/1 1
> >>
> >> I then run 'DPDK/testpmd' inside a container which will use
> >> virtio-user interfaces using the following command
> >>
> >> docker run -it --privileged -v
> >> /var/run/vpp/sock1.sock:/var/run/usvhost1 -v
> >> /var/run/vpp/sock2.sock:/var/run/usvhost2 -v
> >> /dev/hugepages:/dev/hugepages dpdk-app-testpmd ./bin/testpmd -c 0x3 -n
> >> 4 --log-level=9 -m 64 --no-pci --single-file-segments
> >> --vdev=virtio_user0,path=/var/run/usvhost1,mac=54:01:00:01:01:01
> >> --vdev=virtio_user1,path=/var/run/usvhost2,mac=54:01:00:01:01:02 --
> >> -i
> >>
> >> VPP Vnet crashes with following message
> >>
> >> May 27 11:44:00 localhost vnet[6818]: received signal SIGSEGV, PC
> >> 0x7fcca4620187, faulting address 0x7fcb317ac000
> >>
> >> Questions:
> >> I have 'ulimit -c unlimited' and /etc/vpp/startup.conf has
> >> unix {
> >> nodaemon
> >> log /var/log/vpp/vpp.log
> >> full-coredump
> >> cli-listen /run/vpp/cli.sock
> >> gid vpp
> >> }
> >>
> >> But I couldn't locate corefile?
> > The location of the coredump file depends on your system configuration.
> >
> > Please, check "cat /proc/sys/kernel/core_pattern"
> >
> > If you have systemd-coredump in the output of the above command, then
> likely the
> > location of the coredump files is "/var/lib/systemd/coredump/"
> >
> > You can also change the location of where your system places the
> coredump files:
> > echo '/PATH_TO_YOU_LOCATION/core_%e.%p' | sudo tee
> /proc/sys/kernel/core_pattern
> >
> > See if that helps...
> >
>
> Initially '/proc/sys/kernel/core_pattern' was set to 'core'. I changed
> it to 'systemd-coredump'. Still no core generated. VPP crashes
>
> May 29 08:54:34 localhost vnet[4107]: received signal SIGSEGV, PC
> 0x7f0167751187, faulting address 0x7efff43ac000
> May 29 08:54:34 localhost systemd[1]: vpp.service: Main process
> exited, code=killed, status=6/ABRT
> May 29 08:54:34 localhost systemd[1]: vpp.service: Unit entered failed
> state.
> May 29 08:54:34 localhost systemd[1]: vpp.service: Failed with result
> 'signal'.
>
>
> cat /proc/sys/kernel/core_pattern
> systemd-coredump
>
>
> ulimit -a
> core file size (blocks, -c) unlimited
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 15657
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 1024
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 15657
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
> cd /var/lib/systemd/coredump/
> root@localhost:/var/lib/systemd/coredump# ls
> root@localhost:/var/lib/systemd/coredump#
>
> >>
> >> (2) How to enable debugs? I have used 'make build' but no additional
> >> logs other than those shown below
> >>
> >>
> >> VPP logs from /var/log/syslog is shown below
> >> cat /var/log/syslog
> >> May 27 11:40:28 localhost vpp[6818]: vlib_plugin_early_init:361:
> >> plugin path /usr/lib/vpp_plugins:/usr/lib64/vpp_plugins
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: abf_plugin.so (ACL based Forwarding)
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: acl_plugin.so (Access Control Lists)
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: avf_plugin.so (Intel Adaptive Virtual Function (AVF) Device
> >> Plugin)
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:191: Loaded
> >> plugin: cdp_plugin.so
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: dpdk_plugin.so (Data Plane Development Kit (DPDK))
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: flowprobe_plugin.so (Flow per Packet)
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: gbp_plugin.so (Group Based Policy)
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: gtpu_plugin.so (GTPv1-U)
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: igmp_plugin.so (IGMP messaging)
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: ila_plugin.so (Identifier-locator addressing for IPv6)
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: ioam_plugin.so (Inbound OAM)
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:117: Plugin
> >> disabled (default): ixge_plugin.so
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: l2e_plugin.so (L2 Emulation)
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: lacp_plugin.so (Link Aggregation Control Protocol)
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: lb_plugin.so (Load Balancer)
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: memif_plugin.so (Packet Memory Interface (experimetal))
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: nat_plugin.so (Network Address Translation)
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: pppoe_plugin.so (PPPoE)
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: srv6ad_plugin.so (Dynamic SRv6 proxy)
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: srv6am_plugin.so (Masquerading SRv6 proxy)
> >> May 27 11:40:28 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: srv6as_plugin.so (Static SRv6 proxy)
> >> May 27 11:40:29 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: stn_plugin.so (VPP Steals the NIC for Container integration)
> >> May 27 11:40:29 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: tlsmbedtls_plugin.so (mbedtls based TLS Engine)
> >> May 27 11:40:29 localhost vpp[6818]: load_one_plugin:189: Loaded
> >> plugin: tlsopenssl_plugin.so (openssl based TLS Engine)
> >> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
> >> load_one_vat_plugin:67: Loaded plugin: dpdk_test_plugin.so
> >> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> >> Loaded plugin: dpdk_test_plugin.so
> >> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
> >> load_one_vat_plugin:67: Loaded plugin: lb_test_plugin.so
> >> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
> >> load_one_vat_plugin:67: Loaded plugin: flowprobe_test_plugin.so
> >> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
> >> load_one_vat_plugin:67: Loaded plugin: stn_test_plugin.so
> >> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
> >> load_one_vat_plugin:67: Loaded plugin: nat_test_plugin.so
> >> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
> >> load_one_vat_plugin:67: Loaded plugin: udp_ping_test_plugin.so
> >> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
> >> load_one_vat_plugin:67: Loaded plugin: pppoe_test_plugin.so
> >> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
> >> load_one_vat_plugin:67: Loaded plugin: lacp_test_plugin.so
> >> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> >> Loaded plugin: lb_test_plugin.so
> >> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
> >> load_one_vat_plugin:67: Loaded plugin: acl_test_plugin.so
> >> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
> >> load_one_vat_plugin:67: Loaded plugin: ioam_export_test_plugin.so
> >> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
> >> load_one_vat_plugin:67: Loaded plugin: ioam_trace_test_plugin.so
> >> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
> >> load_one_vat_plugin:67: Loaded plugin:
> >> vxlan_gpe_ioam_export_test_plugin.so
> >> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
> >> load_one_vat_plugin:67: Loaded plugin: gtpu_test_plugin.so
> >> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
> >> load_one_vat_plugin:67: Loaded plugin: cdp_test_plugin.so
> >> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
> >> load_one_vat_plugin:67: Loaded plugin: ioam_vxlan_gpe_test_plugin.so
> >> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
> >> load_one_vat_plugin:67: Loaded plugin: memif_test_plugin.so
> >> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]:
> >> load_one_vat_plugin:67: Loaded plugin: ioam_pot_test_plugin.so
> >> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> >> Loaded plugin: flowprobe_test_plugin.so
> >> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> >> Loaded plugin: stn_test_plugin.so
> >> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> >> Loaded plugin: nat_test_plugin.so
> >> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> >> Loaded plugin: udp_ping_test_plugin.so
> >> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> >> Loaded plugin: pppoe_test_plugin.so
> >> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> >> Loaded plugin: lacp_test_plugin.so
> >> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> >> Loaded plugin: acl_test_plugin.so
> >> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> >> Loaded plugin: ioam_export_test_plugin.so
> >> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> >> Loaded plugin: ioam_trace_test_plugin.so
> >> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> >> Loaded plugin: vxlan_gpe_ioam_export_test_plugin.so
> >> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> >> Loaded plugin: gtpu_test_plugin.so
> >> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> >> Loaded plugin: cdp_test_plugin.so
> >> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> >> Loaded plugin: ioam_vxlan_gpe_test_plugin.so
> >> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> >> Loaded plugin: memif_test_plugin.so
> >> May 27 11:40:29 localhost /usr/bin/vpp[6818]: load_one_vat_plugin:67:
> >> Loaded plugin: ioam_pot_test_plugin.so
> >> May 27 11:40:29 localhost vpp[6818]: /usr/bin/vpp[6818]: dpdk: EAL
> >> init args: -c 1 -n 4 --no-pci --huge-dir /dev/hugepages --master-lcore
> >> 0 --socket-mem 256,0
> >> May 27 11:40:29 localhost /usr/bin/vpp[6818]: dpdk: EAL init args: -c
> >> 1 -n 4 --no-pci --huge-dir /dev/hugepages --master-lcore 0
> >> --socket-mem 256,0
> >> May 27 11:40:29 localhost vnet[6818]: dpdk_ipsec_process:1019: not
> >> enough DPDK crypto resources, default to OpenSSL
> >> May 27 11:43:19 localhost vnet[6818]: show vhost-user: unknown input
> `detail
> >> May 27 11:44:00 localhost vnet[6818]: received signal SIGSEGV, PC
> >> 0x7fcca4620187, faulting address 0x7fcb317ac000
> >> May 27 11:44:00 localhost systemd[1]: vpp.service: Main process
> >> exited, code=killed, status=6/ABRT
> >> May 27 11:44:00 localhost systemd[1]: vpp.service: Unit entered failed
> state.
> >> May 27 11:44:00 localhost systemd[1]: vpp.service: Failed with result
> >> 'signal'.
> >> May 27 11:44:00 localhost systemd[1]: vpp.service: Service hold-off
> >> time over, scheduling restart
> >>
>
> Thanks,
> Ravi
>
> >>
> >>
> >> Thanks.
> > Cheers,
> > Marco
> >>
> >>
> >>
> > --
> > Marco V
> >
> > SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
> > HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
>
>
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links:
You receive all messages sent to this group.
View/Reply Online (#9448): https://lists.fd.io/g/vpp-dev/message/9448
View All Messages In Topic (5): https://lists.fd.io/g/vpp-dev/topic/20346431
Mute This Topic: https://lists.fd.io/mt/20346431/21656
New Topic: https://lists.fd.io/g/vpp-dev/post
Change Your Subscription: https://lists.fd.io/g/vpp-dev/editsub/21656
Group Home: https://lists.fd.io/g/vpp-dev
Contact Group Owner: vpp-dev+ow...@lists.fd.io
Terms of Service: https://lists.fd.io/static/tos
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub
-=-=-=-=-=-=-=-=-=-=-=-
