Hi,
I further debugged this issue. It turns out that
vlib_buffer_get_buffer_free_list (vlib_main_t * vm, vlib_buffer_t * b,u32 *
index) returns corrupted fl (free list) pointer. This is happening because
((vlib_buffer_t *) b)->free_list_index in function vlib_get_buffer_free_list()
is corrupt. ((vlib_buffert_t *)b)->free_list_index or infact b comes from
vlib_get_buffer(). I am thinking that the (vlib_buffer_t *)b returned by
vlib_get_buffer() is incorrect. vlib_get_buffer() return offset using
vm->physmem_main.
I tried running same scenario with dbg binaries as above issue was with release
binaries. With debug binaries I saw SIGABRT raised due to assert in
vlib_get_buffer_index() called by fill_free_list() function.
always_inline u32
vlib_get_buffer_index (vlib_main_t * vm, void *p)
{
uword offset = vlib_physmem_offset_of (&vm->physmem_main, p);
ASSERT ((offset % (1 << CLIB_LOG2_CACHE_LINE_BYTES)) == 0);
return offset >> CLIB_LOG2_CACHE_LINE_BYTES;
}
So I feel some how in both the cases vm->physmem_main is not properly setup for
my vhost-driver. Both issues (with release binary and debug binary) occur only
when I assign IP address to interface inside VM1 and each time VPP host process
crashes.
Attached crash dump with debug binaries. I have attached crash dumps with
release binaries in earlier mail. Any pointer will be helpful.
Thanks,
Nitin
On 15-Sep-2017, at 7:33 PM, Saxena, Nitin
<nitin.sax...@cavium.com<mailto:nitin.sax...@cavium.com>> wrote:
Hi All,
I am trying vhost-user configuration on Cavium's aarch64 SoC using VPP v1704. I
mainly followed steps provided at following
https://wiki.fd.io/view/VPP/Use_VPP_to_connect_VMs_Using_Vhost-User_Interface
I am able to launch two VM's using qemu-system-aarch64. The host and guest
kernel version is 4.12.9. I am also able to configure two virtual interface
described in above documentation link. However as soon as I provide IP address
to first VM interface using ifconfig orip addr add command, the vpp process on
host crashes. I was watching "vppctl show_interface" where I saw one packet
received by vpp at Virtual1 interface and after then I think vpp process
crashed so vppctl got hang.
I have attached gdb crash dump "gdb_crash_log.txt". I have also attached
command I used to launch VM1 and VM2: vm1_launch.sh, vm2_launch.sh.
Any pointers will be really helpful.
Thanks,
Nitin
<bridge_setup_vswitch.sh><gdb_crash_log.txt><startup.conf><vm1_launch.sh><vm2_launch.sh>_______________________________________________
vpp-dev mailing list
vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>
https://lists.fd.io/mailman/listinfo/vpp-dev
{\rtf1\ansi\ansicpg1252\cocoartf1504\cocoasubrtf830
{\fonttbl\f0\fswiss\fcharset0 Helvetica;}
{\colortbl;\red255\green255\blue255;}
{\*\expandedcolortbl;;}
\paperw11900\paperh16840\margl1440\margr1440\vieww25100\viewh14200\viewkind0
\pard\tx566\tx1133\tx1700\tx2267\tx2834\tx3401\tx3968\tx4535\tx5102\tx5669\tx6236\tx6803\pardirnatural\partightenfactor0
\f0\fs24 \cf0 GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1\
Copyright (C) 2016 Free Software Foundation, Inc.\
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\
This is free software: you are free to change and redistribute it.\
There is NO WARRANTY, to the extent permitted by law. Type "show copying"\
and "show warranty" for details.\
This GDB was configured as "aarch64-linux-gnu".\
Type "show configuration" for configuration details.\
For bug reporting instructions, please see:\
<http://www.gnu.org/software/gdb/bugs/>.\
Find the GDB manual and other documentation resources online at:\
<http://www.gnu.org/software/gdb/documentation/>.\
For help, type "help".\
Type "apropos word" to search for commands related to "word"...\
Reading symbols from /home/nsaxena/vpp/v1704/build-root/install-vpp_debug-native/vpp/bin/vpp...done.\
(gdb) handle SIGUSR1 nostop noprint pass\
Signal Stop Print Pass to program Description\
SIGUSR1 No No Yes User defined signal 1\
(gdb) r\
Starting program: /home/nsaxena/vpp/v1704/build-root/install-vpp_debug-native/vpp/bin/vpp -c /etc/vpp/dbg-gdb-startup.conf\
warning: Unable to determine the number of hardware watchpoints available.\
warning: Unable to determine the number of hardware breakpoints available.\
[Thread debugging using libthread_db enabled]\
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".\
vlib_plugin_early_init:360: plugin path /home/nsaxena/vpp/v1704/build-root/install-vpp_debug-native/vpp/lib/vpp_plugins\
load_one_plugin:188: Loaded plugin: acl_plugin.so (Access Control Lists)\
load_one_plugin:188: Loaded plugin: dpdk_plugin.so (Data Plane Development Kit (DPDK))\
load_one_plugin:188: Loaded plugin: flowperpkt_plugin.so (Flow per Packet)\
load_one_plugin:188: Loaded plugin: ila_plugin.so (Identifier-locator addressing for IPv6)\
load_one_plugin:188: Loaded plugin: ioam_plugin.so (Inbound OAM)\
load_one_plugin:83: Not a plugin: ixge_plugin.so\
load_one_plugin:188: Loaded plugin: lb_plugin.so (Load Balancer)\
load_one_plugin:188: Loaded plugin: libsixrd_plugin.so (IPv6 Rapid Deployment on IPv4 Infrastructure (RFC5969))\
load_one_plugin:188: Loaded plugin: memif_plugin.so (Packet Memory Interface (experimetal))\
load_one_plugin:188: Loaded plugin: snat_plugin.so (Network Address Translation)\
0: dpdk_bind_devices_to_uio:1158: Unsupported PCI device 0x14e4:0x16a1 found at PCI address 0000:0b:00.0\
0: vlib_pci_bind_to_uio: Skipping PCI device 0000:13:00.1 as host interface enp19s0f1 is up\
0: dpdk_bind_devices_to_uio:1158: Unsupported PCI device 0x14e4:0x16a1 found at PCI address 0000:0b:00.1\
0: vlib_pci_bind_to_uio: Skipping PCI device 0000:13:00.0 as host interface enp19s0f0 is up\
EAL: Detected 224 lcore(s)\
EAL: Probing VFIO support...\
[New Thread 0xffff319ff1f0 (LWP 6110)]\
EAL: PCI device 0000:13:00.0 on NUMA socket 0\
EAL: Device is blacklisted, not initializing\
EAL: PCI device 0000:13:00.1 on NUMA socket 0\
EAL: Device is blacklisted, not initializing\
DPDK physical memory layout:\
Segment 0: phys:0x9f15c00000, len:2097152, virt:0xffff31a00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0\
Segment 1: phys:0x9f16000000, len:266338304, virt:0xffff0b400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0\
Segment 2: phys:0xbf1cc00000, len:268435456, virt:0xfffecfe00000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0\
[New Thread 0xffff311ff1f0 (LWP 6111)]\
0: dpdk_ipsec_process:241: DPDK Cryptodev support is disabled, default to OpenSSL IPsec\
0: dpdk_lib_init:536: DPDK drivers found no ports...\
0: dpdk_lib_init:540: DPDK drivers found 0 ports...\
_______ _ _ _____ ___\
__/ __/ _ \\ (_)__ | | / / _ \\/ _ \\\
_/ _// // / / / _ \\ | |/ / ___/ ___/\
/_/ /____(_)_/\\___/ |___/_/ /_/\
\
DBGvpp#\
Thread 1 "vpp_main" received signal SIGINT, Interrupt.\
0x0000ffffb771f424 in __GI_epoll_pwait (epfd=3, events=0xffff7749347c, maxevents=256, timeout=10,\
set=0xffffb7f8cf60 <unblock_all_signals>) at ../sysdeps/unix/sysv/linux/epoll_pwait.c:42\
42 ../sysdeps/unix/sysv/linux/epoll_pwait.c: No such file or directory.\
(gdb) b fill_free_list\
Breakpoint 1 at 0xffff76df0aec: fill_free_list. (2 locations)\
(gdb) c\
Continuing.\
sh int\
Name Idx State Counter Count\
VirtualEthernet0/0/0 1 up\
VirtualEthernet0/0/1 2 up\
local0 0 down\
DBGvpp#\
Thread 1 "vpp_main" hit Breakpoint 1, fill_free_list (vm=0xffffb7f8d050 <vlib_global_main>, fl=0xffff77495e80,\
min_free_buffers=514) at /home/nsaxena/vpp/v1704/build-data/../src/plugins/dpdk/buffer.c:167\
167 \{\
(gdb) c\
Continuing.\
0: /home/nsaxena/vpp/v1704/build-data/../src/vlib/buffer_funcs.h:73 (vlib_get_buffer_index) assertion `(offset % (1 << CLIB_LOG2_CACHE_LINE_BYTES)) == 0' fails\
\
Thread 1 "vpp_main" received signal SIGABRT, Aborted.\
0x0000ffffb768a528 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54\
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.\
(gdb) bt\
#0 0x0000ffffb768a528 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54\
#1 0x0000ffffb768b9e0 in __GI_abort () at abort.c:89\
#2 0x0000000000407dd0 in os_panic () at /home/nsaxena/vpp/v1704/build-data/../src/vpp/vnet/main.c:261\
#3 0x0000ffffb781ea54 in debugger () at /home/nsaxena/vpp/v1704/build-data/../src/vppinfra/error.c:84\
#4 0x0000ffffb781ee84 in _clib_error (how_to_die=2, function_name=0x0, line_number=0,\
fmt=0xffff77050a98 "%s:%d (%s) assertion `%s' fails") at /home/nsaxena/vpp/v1704/build-data/../src/vppinfra/error.c:143\
#5 0x0000ffff76def6ac in vlib_get_buffer_index (vm=0xffffb7f8d050 <vlib_global_main>, p=0xffff185fbf40)\
at /home/nsaxena/vpp/v1704/build-data/../src/vlib/buffer_funcs.h:73\
#6 0x0000ffff76df1264 in fill_free_list (vm=0xffffb7f8d050 <vlib_global_main>, fl=0xffff77495e80, min_free_buffers=514)\
at /home/nsaxena/vpp/v1704/build-data/../src/plugins/dpdk/buffer.c:231\
#7 0x0000ffff76df1790 in alloc_from_free_list (vm=0xffffb7f8d050 <vlib_global_main>, free_list=0xffff77495e80,\
alloc_buffers=0xffff774a56a8, n_alloc_buffers=514)\
at /home/nsaxena/vpp/v1704/build-data/../src/plugins/dpdk/buffer.c:290\
#8 0x0000ffff76df1b2c in dpdk_buffer_alloc_from_free_list (vm=0xffffb7f8d050 <vlib_global_main>, buffers=0xffff774a56a8,\
n_buffers=514, free_list_index=0) at /home/nsaxena/vpp/v1704/build-data/../src/plugins/dpdk/buffer.c:329\
#9 0x0000ffffb7cdf9d4 in vlib_buffer_alloc_from_free_list (vm=0xffffb7f8d050 <vlib_global_main>, buffers=0xffff774a56a8,\
n_buffers=514, free_list_index=0) at /home/nsaxena/vpp/v1704/build-data/../src/vlib/buffer_funcs.h:277\
#10 0x0000ffffb7ce86c4 in vhost_user_if_input (vm=0xffffb7f8d050 <vlib_global_main>, vum=0xffffb7ec9fa8 <vhost_user_main>,\
vui=0xffff7871ceec, qid=0, node=0xffff77ca6a00)\
at /home/nsaxena/vpp/v1704/build-data/../src/vnet/devices/virtio/vhost-user.c:1659\
#11 0x0000ffffb7ce9454 in vhost_user_input (vm=0xffffb7f8d050 <vlib_global_main>, node=0xffff77ca6a00, f=0x0)\
at /home/nsaxena/vpp/v1704/build-data/../src/vnet/devices/virtio/vhost-user.c:1956\
#12 0x0000ffffb7f16054 in dispatch_node (vm=0xffffb7f8d050 <vlib_global_main>, node=0xffff77ca6a00,\
type=VLIB_NODE_TYPE_INPUT, dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x0, last_time_stamp=2518697720434)\
at /home/nsaxena/vpp/v1704/build-data/../src/vlib/main.c:998\
#13 0x0000ffffb7f17cac in vlib_main_or_worker_loop (vm=0xffffb7f8d050 <vlib_global_main>, is_main=1)\
at /home/nsaxena/vpp/v1704/build-data/../src/vlib/main.c:1493\
#14 0x0000ffffb7f18564 in vlib_main_loop (vm=0xffffb7f8d050 <vlib_global_main>)\
at /home/nsaxena/vpp/v1704/build-data/../src/vlib/main.c:1608\
#15 0x0000ffffb7f18b40 in vlib_main (vm=0xffffb7f8d050 <vlib_global_main>, input=0xffff77af1fc0)\
at /home/nsaxena/vpp/v1704/build-data/../src/vlib/main.c:1736\
#16 0x0000ffffb7f5d7e4 in thread0 (arg=281473768280144) at /home/nsaxena/vpp/v1704/build-data/../src/vlib/unix/main.c:507\
#17 0x0000ffffb78329e4 in clib_calljmp () at /home/nsaxena/vpp/v1704/build-data/../src/vppinfra/longjmp.S:676\
}_______________________________________________
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev
