On Thu, May 13, 2021 at 4:11 PM David Marchand
<david.march...@redhat.com> wrote:
> On Thu, May 13, 2021 at 2:38 PM Chenbo Xia <chenbo....@intel.com> wrote:
> >
> > This patch fixes an issue of application crash because of vhost iotlb
> > not initialized when virtio has multiqueue enabled.
> >
> > iotlb messages will be sent when some queues are not enabled. If we
> > initialize iotlb in vhost_user_set_vring_num, it could happen that
> > iotlb update comes when iotlb pool of disabled queues are not
> > initialized.
>
> This makes the problem I reproduced disappear at init, but I noticed
> the segfault after restarting testpmd once.
> And a little bit after this, my vm crashed.
>
> This is not systematic, so I guess there is some condition with how
> the virtio device is initialised in the vm.
The crash is systematic (not sure what I missed yesterday, but I
always get it with simple steps below).
Full logs:
# dpdk-testpmd --vdev
net_vhost0,iface=/var/lib/vhost_sockets/vhost0,client=1,iommu-support=1,queues=2
-w 0:0:0.0 --log-level=lib.vhost.config:debug -- -ia --rxq=2
VHOST_CONFIG: vhost-user client: socket created, fd: 31
VHOST_CONFIG: failed to connect to /var/lib/vhost_sockets/vhost0: No
such file or directory
VHOST_CONFIG: /var/lib/vhost_sockets/vhost0: reconnecting...
# start vm (with virtio device in the guest OS bound to kernel kmod,
i.e. no special configuration)
# testpmd logs:
testpmd> VHOST_CONFIG: /var/lib/vhost_sockets/vhost0: connected
VHOST_CONFIG: new device, handle is 0
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_PROTOCOL_FEATURES succeeded
and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES
VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf
VHOST_CONFIG: Processing VHOST_USER_SET_PROTOCOL_FEATURES succeeded.
VHOST_CONFIG: read message VHOST_USER_GET_QUEUE_NUM
VHOST_CONFIG: Processing VHOST_USER_GET_QUEUE_NUM succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD
VHOST_CONFIG: Processing VHOST_USER_SET_SLAVE_REQ_FD succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_OWNER
VHOST_CONFIG: Processing VHOST_USER_SET_OWNER succeeded.
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:0 file:36
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:1 file:37
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_PROTOCOL_FEATURES succeeded
and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES
VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf
VHOST_CONFIG: Processing VHOST_USER_SET_PROTOCOL_FEATURES succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD
VHOST_CONFIG: Processing VHOST_USER_SET_SLAVE_REQ_FD succeeded.
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:2 file:35
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:3 file:39
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 0
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 1
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 0 to qp idx: 2
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 0 to qp idx: 3
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 0
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 1
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 0 to qp idx: 2
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 0 to qp idx: 3
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_FEATURES
VHOST_CONFIG: negotiated Virtio features: 0x370607f83
VHOST_CONFIG: (0) mergeable RX buffers off, virtio 1 on
VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_0
VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_1
VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_2
VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_3
VHOST_CONFIG: Processing VHOST_USER_SET_FEATURES succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE
VHOST_CONFIG: guest memory region size: 0x80000000
guest physical addr: 0x0
guest virtual addr: 0x7f2400000000
host virtual addr: 0x7fff40000000
mmap addr : 0x7fff40000000
mmap size : 0x80000000
mmap align: 0x40000000
mmap off : 0x0
VHOST_CONFIG: guest memory region size: 0x180000000
guest physical addr: 0x100000000
guest virtual addr: 0x7f2480000000
host virtual addr: 0x7ffdc0000000
mmap addr : 0x7ffd40000000
mmap size : 0x200000000
mmap align: 0x40000000
mmap off : 0x80000000
VHOST_CONFIG: Processing VHOST_USER_SET_MEM_TABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:0 file:42
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:0 file:43
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:1 file:36
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:1 file:44
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 0
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 1
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_FEATURES
VHOST_CONFIG: negotiated Virtio features: 0x370607f83
VHOST_CONFIG: (0) mergeable RX buffers off, virtio 1 on
VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_0
VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_1
VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_2
VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_3
VHOST_CONFIG: Processing VHOST_USER_SET_FEATURES succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:2 file:37
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:2 file:45
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:3 file:35
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:3 file:46
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) failed to map avail ring.
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) failed to map avail ring.
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) mapped address desc: 0x7fff1d886000
VHOST_CONFIG: (0) mapped address avail: 0x7fff1d887000
VHOST_CONFIG: (0) mapped address used: 0x7fff1d887240
VHOST_CONFIG: (0) log_guest_addr: 0
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) mapped address desc: 0x7fff1d89e000
VHOST_CONFIG: (0) mapped address avail: 0x7fff1d89f000
VHOST_CONFIG: (0) mapped address used: 0x7fff1d89f240
VHOST_CONFIG: (0) log_guest_addr: 0
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) mapped address desc: 0x7fff1d8da000
VHOST_CONFIG: (0) mapped address avail: 0x7fff1d8db000
VHOST_CONFIG: (0) mapped address used: 0x7fff1d8db240
VHOST_CONFIG: (0) log_guest_addr: 0
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
Port 0: queue state event
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) mapped address desc: 0x7fff1e198000
VHOST_CONFIG: (0) mapped address avail: 0x7fff1e199000
VHOST_CONFIG: (0) mapped address used: 0x7fff1e199240
VHOST_CONFIG: (0) log_guest_addr: 0
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
Port 0: queue state event
VHOST_CONFIG: virtio is now ready for processing.
Port 0: link state change event
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 0
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 1
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 2
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
Port 0: queue state event
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 3
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
Port 0: queue state event
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
So far, everything looks good.
Now I quit testpmd.
VHOST_CONFIG: free connfd = 31 for device '/var/lib/vhost_sockets/vhost0'
And I restart testpmd with the same command as above:
VHOST_CONFIG: vhost-user client: socket created, fd: 31
VHOST_CONFIG: new device, handle is 0
Port 0: 56:48:4F:53:54:00
Checking link statuses...
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_PROTOCOL_FEATURES succeeded
and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES
VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf
VHOST_CONFIG: Processing VHOST_USER_SET_PROTOCOL_FEATURES succeeded.
VHOST_CONFIG: read message VHOST_USER_GET_QUEUE_NUM
VHOST_CONFIG: Processing VHOST_USER_GET_QUEUE_NUM succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD
VHOST_CONFIG: Processing VHOST_USER_SET_SLAVE_REQ_FD succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_OWNER
VHOST_CONFIG: Processing VHOST_USER_SET_OWNER succeeded.
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:0 file:36
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:1 file:37
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_PROTOCOL_FEATURES succeeded
and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES
VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf
VHOST_CONFIG: Processing VHOST_USER_SET_PROTOCOL_FEATURES succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD
VHOST_CONFIG: Processing VHOST_USER_SET_SLAVE_REQ_FD succeeded.
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:2 file:35
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:3 file:39
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_FEATURES
VHOST_CONFIG: negotiated Virtio features: 0x370607f83
VHOST_CONFIG: (0) mergeable RX buffers off, virtio 1 on
VHOST_CONFIG: IOTLB cache name: iotlb_101328_0_0
VHOST_CONFIG: IOTLB cache name: iotlb_101328_0_1
VHOST_CONFIG: IOTLB cache name: iotlb_101328_0_2
VHOST_CONFIG: IOTLB cache name: iotlb_101328_0_3
VHOST_CONFIG: Processing VHOST_USER_SET_FEATURES succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE
VHOST_CONFIG: guest memory region size: 0x80000000
guest physical addr: 0x0
guest virtual addr: 0x7f2400000000
host virtual addr: 0x7fff40000000
mmap addr : 0x7fff40000000
mmap size : 0x80000000
mmap align: 0x40000000
mmap off : 0x0
VHOST_CONFIG: guest memory region size: 0x180000000
guest physical addr: 0x100000000
guest virtual addr: 0x7f2480000000
host virtual addr: 0x7ffdc0000000
mmap addr : 0x7ffd40000000
mmap size : 0x200000000
mmap align: 0x40000000
mmap off : 0x80000000
VHOST_CONFIG: Processing VHOST_USER_SET_MEM_TABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:0 file:42
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:0 file:43
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:1 file:36
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:1 file:44
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 0
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 1
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_FEATURES
VHOST_CONFIG: negotiated Virtio features: 0x40000000
VHOST_CONFIG: (0) mergeable RX buffers off, virtio 1 off
VHOST_CONFIG: Processing VHOST_USER_SET_FEATURES succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:2 file:35
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:2 file:37
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:3 file:39
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:3 file:45
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
Thread 9 "vhost-events" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffee7fc400 (LWP 101339)]
0x0000000000e30803 in rte_mempool_default_cache (lcore_id=4294967295,
mp=0x0) at ../lib/mempool/rte_mempool.h:1244
1244 if (mp->cache_size == 0)
Missing separate debuginfos, use: yum debuginfo-install
glibc-2.28-127.el8_3.2.x86_64 libibverbs-29.0-3.el8.x86_64
libnl3-3.5.0-1.el8.x86_64 numactl-libs-2.0.12-11.el8.x86_64
zlib-1.2.11-16.el8_2.x86_64
(gdb) bt full
#0 0x0000000000e30803 in rte_mempool_default_cache
(lcore_id=4294967295, mp=0x0) at ../lib/mempool/rte_mempool.h:1244
No locals.
#1 rte_mempool_get_bulk (n=1, obj_table=0x7fffee7f5e98, mp=0x0) at
../lib/mempool/rte_mempool.h:1533
cache = 0x0
cache = <optimized out>
#2 rte_mempool_get (obj_p=0x7fffee7f5e98, mp=0x0) at
../lib/mempool/rte_mempool.h:1561
No locals.
#3 vhost_user_iotlb_cache_insert (vq=0x1673bff00, iova=10159157248,
uaddr=140733688868864, size=4096, perm=3 '\003') at
../lib/vhost/iotlb.c:164
node = 0x1673c08c0
new_node = 0x1673671f0
ret = 0
#4 0x000000000100d51d in vhost_user_iotlb_msg (pdev=0x7fffee7f92b8,
msg=0x7fffee7f9010, main_fd=31) at ../lib/vhost/vhost_user.c:2409
vq = 0x1673bff00
dev = 0x1673c08c0
imsg = 0x7fffee7f901c
i = 2
vva = 140733688868864
len = 4096
#5 0x000000000100e31e in vhost_user_msg_handler (vid=0, fd=31) at
../lib/vhost/vhost_user.c:2882
dev = 0x1673c08c0
msg = {request = {master = 22, slave = 22}, flags = 9, size =
32, payload = {u64 = 10159157248, state = {index = 1569222656, num =
2}, addr = {index = 1569222656, flags = 2, desc_user_addr = 4096,
used_user_addr = 139800607223808, avail_user_addr =
7907112783476097539, log_guest_addr = 10159256128}, memory = {nregions
= 1569222656, padding = 2, regions = {{guest_phys_addr = 4096,
memory_size = 139800607223808, userspace_addr =
7907112783476097539, mmap_offset = 10159256128}, {guest_phys_addr =
4294967296, memory_size = 6442450944, userspace_addr =
139794743033856,
mmap_offset = 2147483648}, {guest_phys_addr = 0,
memory_size = 0, userspace_addr = 0, mmap_offset = 0},
{guest_phys_addr = 0, memory_size = 0, userspace_addr = 0, mmap_offset
= 4294967296}, {
guest_phys_addr = 0, memory_size = 0, userspace_addr
= 0, mmap_offset = 0}, {guest_phys_addr = 0, memory_size = 0,
userspace_addr = 0, mmap_offset = 0}, {guest_phys_addr = 0,
memory_size = 0,
userspace_addr = 0, mmap_offset = 0},
{guest_phys_addr = 0, memory_size = 4294967296, userspace_addr = 0,
mmap_offset = 3419188017980506112}}}, log = {mmap_size = 10159157248,
mmap_offset = 4096}, iotlb = {iova = 10159157248, size =
4096, uaddr = 139800607223808, perm = 3 '\003', type = 2 '\002'},
crypto_session = {session_id = 10159157248, op_code = 4096,
cipher_algo = 0, cipher_key_len = 3716706304, hash_algo
= 32549, digest_len = 3523936771, auth_key_len = 1841018158, aad_len =
1569321536, op_type = 2 '\002', dir = 0 '\000',
hash_mode = 0 '\000', chaining_dir = 0 '\000', ciphe_key
= 0x100000000 "\356o.\001\003", auth_key = 0x180000000 "",
cipher_key_buf =
"\000\000\000\200$\177\000\000\000\000\000\200", '\000' <repeats 51
times>,
auth_key_buf = '\000' <repeats 12 times>, "\001", '\000'
<repeats 111 times>, "\001", '\000' <repeats 15 times>,
"ces/system/node", '\000' <repeats 209 times>...}, area = {u64 =
10159157248,
size = 4096, offset = 139800607223808}, inflight =
{mmap_size = 10159157248, mmap_offset = 4096, num_queues = 28672,
queue_size = 56712}}, fds = {-1, -1, -1, -1, -1, -1, -1, -1}, fd_num =
0}
vdpa_dev = 0x0
ret = 0
unlock_required = 0
handled = false
request = 22
i = 4
#6 0x0000000000e41f75 in vhost_user_read_cb (connfd=31,
dat=0x6e28510, remove=0x7fffee7f9394) at ../lib/vhost/socket.c:309
conn = 0x6e28510
vsocket = 0x6e1f140
ret = 1
#7 0x0000000000e0f9c4 in fdset_event_dispatch (arg=0x66f9a60
<vhost_user+8192>) at ../lib/vhost/fd_man.c:286
i = 1
pfd = 0x66f9a68 <vhost_user+8200>
pfdentry = 0x66fba88 <vhost_user+16424>
rcb = 0xe41f3c <vhost_user_read_cb>
wcb = 0x0
dat = 0x6e28510
fd = 31
numfds = 2
...
And qemu crashes right after this:
(gdb) bt full
#0 0x000056269a497160 in vhost_device_iotlb_miss
(dev=dev@entry=0x56269d6ed800, iova=10159497216, write=<optimized
out>)
at
/usr/src/debug/qemu-kvm-4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64/hw/virtio/vhost.c:944
iotlb = <optimized out>
uaddr = <optimized out>
len = <optimized out>
ret = -14
_rcu_read_auto = 0x1
#1 0x000056269a499361 in vhost_backend_handle_iotlb_msg
(imsg=0x7ffd5eddeea0, dev=0x56269d6ed800) at
/usr/src/debug/qemu-kvm-4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64/hw/virtio/vhost-backend.c:351
ret = <optimized out>
ret = <optimized out>
#2 vhost_backend_handle_iotlb_msg (dev=dev@entry=0x56269d6ed800,
imsg=imsg@entry=0x7ffd5eddeea0) at
/usr/src/debug/qemu-kvm-4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64/hw/virtio/vhost-backend.c:344
ret = 0
#3 0x000056269a499f0b in slave_read (opaque=0x56269d6ed800) at
/usr/src/debug/qemu-kvm-4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64/hw/virtio/vhost-user.c:1048
dev = 0x56269d6ed800
u = 0x56269cd3f7b0
hdr = {request = VHOST_USER_GET_FEATURES, flags = 1, size = 32}
payload = {u64 = 10159497216, state = {index = 1569562624, num
= 2}, addr = {index = 1569562624, flags = 2, desc_user_addr = 0,
used_user_addr = 0, avail_user_addr = 259, log_guest_addr = 0}, memory
= {
nregions = 1569562624, padding = 2, regions =
{{guest_phys_addr = 0, memory_size = 0, userspace_addr = 259,
mmap_offset = 0}, {guest_phys_addr = 0, memory_size = 0,
userspace_addr = 0,
mmap_offset = 0}, {guest_phys_addr = 0, memory_size =
0, userspace_addr = 0, mmap_offset = 0}, {guest_phys_addr = 0,
memory_size = 0, userspace_addr = 0, mmap_offset = 0},
{guest_phys_addr = 0,
memory_size = 0, userspace_addr = 0, mmap_offset = 0},
{guest_phys_addr = 0, memory_size = 0, userspace_addr = 0, mmap_offset
= 0}, {guest_phys_addr = 0, memory_size = 0, userspace_addr = 0,
mmap_offset = 0}, {guest_phys_addr = 0, memory_size =
0, userspace_addr = 0, mmap_offset = 0}}}, log = {mmap_size =
10159497216, mmap_offset = 0}, iotlb = {iova = 10159497216, size = 0,
uaddr = 0, perm = 3 '\003', type = 1 '\001'}, config =
{offset = 1569562624, size = 2, flags = 0, region = '\000' <repeats 12
times>, "\003\001", '\000' <repeats 241 times>}, session = {
session_id = 10159497216, session_setup_data = {op_code =
0, cipher_alg = 0, key_len = 0, hash_alg = 0, hash_result_len = 259,
auth_key_len = 0, add_len = 0, op_type = 0 '\000',
direction = 0 '\000', hash_mode = 0 '\000',
alg_chain_order = 0 '\000', cipher_key = 0x0, auth_key = 0x0}, key =
'\000' <repeats 63 times>, auth_key = '\000' <repeats 511 times>},
area = {
u64 = 10159497216, size = 0, offset = 0}, inflight =
{mmap_size = 10159497216, mmap_offset = 0, num_queues = 0, queue_size
= 0}}
size = <optimized out>
ret = 0
iov = {iov_base = 0x7ffd5eddee04, iov_len = 12}
msgh = {msg_name = 0x0, msg_namelen = 0, msg_iov =
0x7ffd5eddee10, msg_iovlen = 1, msg_control = 0x7ffd5eddf120,
msg_controllen = 0, msg_flags = 0}
fd = {-1, -1, -1, -1, -1, -1, -1, -1}
control =
"^\a\022\000\000\000\000\000\226\070\270\t\000\000\000\000\334\345\061\234&V\000\000\000x\v\322.\265\273mP\347\061\234&V\000\000\000\000\000\000\000\000\000"
cmsg = <optimized out>
i = <optimized out>
fdsize = 0
...
--
David Marchand
