Hi,
I tried to reproduce without success(see attached log).
I fail to reproduce because buf_iova fits into 32 bits in my case:
(gdb) p /x *tx_pkts[0]
$4 = {
cacheline0 = 0x77b19ec0,
buf_addr = 0x77b19f40,
buf_iova = 0x49519f40,
rearm_data = 0x77b19ed0,
However, looking at your report, something like this would work for you:
diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h
index 9d4aba11a3..38efbc517a 100644
--- a/drivers/net/virtio/virtqueue.h
+++ b/drivers/net/virtio/virtqueue.h
@@ -124,7 +124,7 @@ virtqueue_store_flags_packed(struct
vring_packed_desc *dp,
* (virtio-pci and virtio-user).
*/
#define VIRTIO_MBUF_ADDR(mb, vq) \
- ((uint64_t)(*(uintptr_t *)((uintptr_t)(mb) +
(vq)->mbuf_addr_offset)))
+ (*(uint64_t *)((uintptr_t)(mb) + (vq)->mbuf_addr_offset))
The problem is that it would likely break Virtio-user en 32bits mode, as
this is how it was initially implemented, and got fixed few years ago,
as David hinted to me:
commit 260aae9ad9621e3e758f1443abb8fcbc25ece07c
Author: Jianfeng Tan <jianfeng....@intel.com>
Date: Wed Apr 19 02:30:33 2017 +0000
net/virtio-user: fix address on 32-bit system
virtio-user cannot work on 32-bit system as higher 32-bit of the
addr field (64-bit) in the desc is filled with non-zero value
which should not happen for a 32-bit system.
In case of virtio-user, we use buf_addr of mbuf to fill the
virtqueue desc addr. This is a regression bug. For 32-bit system,
the first 4 bytes of mbuf is buf_addr, with following 8 bytes for
buf_phyaddr. With below wrong definition, both buf_addr and lower
4 bytes buf_phyaddr are obtained to fill the virtqueue desc.
#define VIRTIO_MBUF_ADDR(mb, vq) \
(*(uint64_t *)((uintptr_t)(mb) + (vq)->offset))
Fixes: 25f80d108780 ("net/virtio: fix packet corruption")
Cc: sta...@dpdk.org
Signed-off-by: Jianfeng Tan <jianfeng....@intel.com>
Acked-by: Yuanhan Liu <yuanhan....@linux.intel.com>
If my understanding is correct, on 32 bits, when mbuf->buf_addr is used
(Virtio-user), we need to mask out the higher 4 bytes, while when using
Virtio-pci we need the full 64 bits (as the physical addresses used as
IOVA on the guest are 64 bits).
Regards,
Maxime
On 9/13/23 15:24, Roger Melton (rmelton) wrote:
+Chris Brezovec
Hi Maxime,
Chris from our team is attending the DPDK Summit in Dublin this week.
If you have some time available, we'd appreciate it if he could meet
with you to discuss the 32bit virtio issue we are seeing.
Regards,
Roger Melton
On 9/6/23 2:57 PM, Dave Johnson (davejo) wrote:
Hi Maxime,
This email is regarding the following commit:
https://github.com/DPDK/dpdk/commit/ba55c94a7ebc386d2288d6578ed57aad6cb92657
A query had been sent previously on this topic (see below) indicating
this commit appears to have broken the 32-bit testpmd app and impacted
one of our products that runs as a 32-bit DPDK application. We
consequently backed the commit out of our product but would prefer to
get a fix for it. In the earlier exchange, you had asked if we were
using virtio-pci or virtio-user (we are using virtio-pci) and asked
for logs which Sampath provided. It’s been a while, so let me now if
you need me to send resend those logs or need any other information.
FWIW, I reproduced this using testpmd and noticed that this part of
the change seems to be the interesting part (in
drivers/net/virtio/virtqueue.h):
/**
* Return the IOVA (or virtual address in case of virtio-user) of mbuf
* data buffer.
*
* The address is firstly casted to the word size (sizeof(uintptr_t))
* before casting it to uint64_t. This is to make it work with different
* combination of word size (64 bit and 32 bit) and virtio device
* (virtio-pci and virtio-user).
*/
#define VIRTIO_MBUF_ADDR(mb, vq) \
((uint64_t)(*(uintptr_t *)((uintptr_t)(mb) +
(vq)->mbuf_addr_offset))
If I revert just this part of the changeset (by re-using the
VIRTIO_MBUF_ADDR to return buf_iova which matches what it had used
previously), then 32-bit testpmd is able to receive traffic again:
#define VIRTIO_MBUF_ADDR(mb, vq) (mb->buf_iova)
Looking at the address produced by each of these, I see the address is
the same except that the casting results in the upper bits getting
cleared:
Address from patch (nonworking case) = 0x58e7c900
Address using buf_iova (working case) = 0x158e7c900
::
Address from patch (nonworking case) = 0x58e7bfc0
Address using buf_iova (working case) = 0x158e7bfc0
::
Address from patch (nonworking case) = 0x58e7b680
Address using buf_iova (working case) = 0x158e7b680
::
Regards, Dave
*From: *Sampath Peechu (speechu) <spee...@cisco.com>
*Date: *Monday, January 30, 2023 at 3:29 PM
*To: *Maxime Coquelin <maxime.coque...@redhat.com>,
chenbo....@intel.com <chenbo....@intel.com>, dev@dpdk.org <dev@dpdk.org>
*Cc: *Roger Melton (rmelton) <rmel...@cisco.com>, Malcolm Bumgardner
(mbumgard) <mbumg...@cisco.com>
*Subject: *Re: Commit broke 32-bit testpmd app
Hi Maxime,
Could you please let us know if you got a chance to look at the debugs
logs I provided?
Thanks,
Sampath
*From: *Sampath Peechu (speechu) <spee...@cisco.com>
*Date: *Tuesday, December 6, 2022 at 1:08 PM
*To: *Maxime Coquelin <maxime.coque...@redhat.com>,
chenbo....@intel.com <chenbo....@intel.com>, dev@dpdk.org <dev@dpdk.org>
*Cc: *Roger Melton (rmelton) <rmel...@cisco.com>
*Subject: *Re: Commit broke 32-bit testpmd app
Hi Maxime,
Did you get a chance to look into this?
Please let me know if you need anything else.
Thanks,
Sampath
*From: *Sampath Peechu (speechu) <spee...@cisco.com>
*Date: *Wednesday, November 23, 2022 at 5:15 PM
*To: *Maxime Coquelin <maxime.coque...@redhat.com>,
chenbo....@intel.com <chenbo....@intel.com>, dev@dpdk.org <dev@dpdk.org>
*Cc: *Roger Melton (rmelton) <rmel...@cisco.com>
*Subject: *Re: Commit broke 32-bit testpmd app
Hi Maxime,
I’m attaching the following for reference.
* Instructions for Centos8 test setup
* Diffs between the working and non-working versions (working
version has the problem commit backed out)
* Working logs (stats show that ping packets from neighbor VM can be
seen with both 64-bit and 32-bit apps)
* Non-working logs (stats show that ping packets from neighbor VM
are seen with 64-bit app but NOT seen with 32-bit app)
============================
$ sudo ./usertools/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:07:00.0 'Virtio network device 1041' drv=igb_uio unused=
0000:08:00.0 'Virtio network device 1041' drv=igb_uio unused=
Network devices using kernel driver
===================================
0000:01:00.0 'Virtio network device 1041' if=enp1s0 drv=virtio-pci
unused=igb_uio *Active*
…
===========================
Thanks,
Sampath
*From: *Maxime Coquelin <maxime.coque...@redhat.com>
*Date: *Tuesday, November 22, 2022 at 4:24 AM
*To: *Sampath Peechu (speechu) <spee...@cisco.com>,
chenbo....@intel.com <chenbo....@intel.com>, dev@dpdk.org <dev@dpdk.org>
*Cc: *Roger Melton (rmelton) <rmel...@cisco.com>
*Subject: *Re: Commit broke 32-bit testpmd app
Hi,
In my initial reply (see below), I also asked if you had logs to share.
And wondered whether it happens with Virtio PCI or Virtio-user?
Regards,
Maxime
On 11/16/22 00:30, Sampath Peechu (speechu) wrote:
> ++ dev@dpdk.org <mailto:dev@dpdk.org <mailto:dev@dpdk.org>>
>
> *From: *Maxime Coquelin <maxime.coque...@redhat.com>
> *Date: *Tuesday, November 15, 2022 at 3:19 AM
> *To: *Sampath Peechu (speechu) <spee...@cisco.com>,
chenbo....@intel.com
> <chenbo....@intel.com>
> *Cc: *Roger Melton (rmelton) <rmel...@cisco.com>
> *Subject: *Re: Commit broke 32-bit testpmd app
>
> Hi Sampath,
>
>
> Please add dev@dpdk.org, the upstream mailing list, if this is related
> to the upstream DPDK project.If it is using RHEL DPDK package, please
> use the appropriate support channels.
>
> On 11/14/22 23:55, Sampath Peechu (speechu) wrote:
> > Hi Virtio Maintainers team,
> >
> > This email is regarding the following commit.
> >
> >
>
https://github.com/DPDK/dpdk/commit/ba55c94a7ebc386d2288d6578ed57aad6cb92657 <https://github.com/DPDK/dpdk/commit/ba55c94a7ebc386d2288d6578ed57aad6cb92657> <https://github.com/DPDK/dpdk/commit/ba55c94a7ebc386d2288d6578ed57aad6cb92657 <https://github.com/DPDK/dpdk/commit/ba55c94a7ebc386d2288d6578ed57aad6cb92657>>
> >
> > The above commit appears to have broken the 32-bit testpmd app (and
> > consequently impacted one of our products that runs as a 32-bit DPDK
> > app). The 64-bit testpmd app does not appear to be impacted though.
>
> We'll need some logs to understand what is going on.
> Does it happen with virtio-pci or virtio-user?
>
> Regards,
> Maxime
>
> > With the commit in place, we didn’t see any packets going through at
> > all. After backing out the commit and rebuilding the 32-bit
testpmd app
> > in our test setup, we were able to pass traffic as expected.
> >
> > Could you please let us know if this is a known issue? And if
there is a
> > fix available for it?
> >
> > Thank you,
> >
> > Sampath Peechu
> >
> > Cisco Systems
> >
>
[root@localhost dpdk]# file ./build/app/dpdk-testpmd
./build/app/dpdk-testpmd: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, BuildID[sha1]=55cf73daf9530b97290b252e48da9390163d42b7, for GNU/Linux 3.2.0, with debug_info, not stripped
[root@localhost dpdk]# uname -a
Linux localhost.localdomain 6.2.9-300.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 30 22:32:58 UTC 2023 x86_64 GNU/Linux
[root@localhost dpdk]# ./build/app/dpdk-testpmd -- -i
EAL: Detected CPU lcores: 3
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Probe PCI driver: net_virtio (1af4:1041) device: 0000:01:00.0 (socket -1)
eth_virtio_pci_init(): Failed to init PCI device
EAL: Requested device 0000:01:00.0 cannot be used
EAL: Probe PCI driver: net_virtio (1af4:1041) device: 0000:07:00.0 (socket -1)
EAL: Using IOMMU type 8 (No-IOMMU)
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
Warning: NUMA should be configured manually by using --port-numa-config and --ring-numa-config parameters along with --numa.
testpmd: create a new mbuf pool <mb_pool_0>: n=163456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.
Configuring Port 0 (socket 0)
EAL: Error disabling MSI-X interrupts for fd 24
Port 0: 56:48:4F:53:54:01
Checking link statuses...
Done
testpmd> set fwd icmpecho
Set icmpecho packet forwarding mode
testpmd> start
icmpecho packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation mode: native
Logical Core 1 (socket 0) forwards packets on 1 streams:
RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
icmpecho packet forwarding packets/burst=32
nb forwarding cores=1 - nb forwarding ports=1
port 0: RX queue number: 1 Tx queue number: 1
Rx offloads=0x0 Tx offloads=0x0
RX queue: 0
RX desc=0 - RX free threshold=0
RX threshold registers: pthresh=0 hthresh=0 wthresh=0
RX Offloads=0x0
TX queue: 0
TX desc=0 - TX free threshold=0
TX threshold registers: pthresh=0 hthresh=0 wthresh=0
TX offloads=0x0 - TX RS bit threshold=0
testpmd> set verbose 9
Change verbose level from 0 to 9
testpmd> port 0/queue 0: received 1 packets
src=DE:D0:06:AC:DC:3B - dst=56:48:4F:53:54:01 - pool=mb_pool_0 - type=0x0800 - length=98 - nb_segs=1 - sw ptype: L2_ETHER L3_IPV4 - l2_len=14 - l3_len=20 - Receive queue=0x0
ol_flags: RTE_MBUF_F_RX_L4_CKSUM_UNKNOWN RTE_MBUF_F_RX_IP_CKSUM_UNKNOWN RTE_MBUF_F_RX_OUTER_L4_CKSUM_UNKNOWN
Port 0 pkt-len=98 nb-segs=1
ETH: src=DE:D0:06:AC:DC:3B dst=56:48:4F:53:54:01 type=0x0800
IPV4: src=192.168.101.2 dst=192.168.101.3 proto=1 (ICMP)
ICMP: echo request seq id=223
port 0/queue 0: sent 1 packets
src=56:48:4F:53:54:01 - dst=DE:D0:06:AC:DC:3B - pool=mb_pool_0 - type=0x0800 - length=98 - nb_segs=1 - sw ptype: L2_ETHER L3_IPV4 - l2_len=14 - l3_len=20 - Send queue=0x0
ol_flags: RTE_MBUF_F_TX_L4_NO_CKSUM
port 0/queue 0: received 1 packets
src=DE:D0:06:AC:DC:3B - dst=56:48:4F:53:54:01 - pool=mb_pool_0 - type=0x0800 - length=98 - nb_segs=1 - sw ptype: L2_ETHER L3_IPV4 - l2_len=14 - l3_len=20 - Receive queue=0x0
ol_flags: RTE_MBUF_F_RX_L4_CKSUM_UNKNOWN RTE_MBUF_F_RX_IP_CKSUM_UNKNOWN RTE_MBUF_F_RX_OUTER_L4_CKSUM_UNKNOWN
Port 0 pkt-len=98 nb-segs=1
ETH: src=DE:D0:06:AC:DC:3B dst=56:48:4F:53:54:01 type=0x0800
IPV4: src=192.168.101.2 dst=192.168.101.3 proto=1 (ICMP)
ICMP: echo request seq id=224
port 0/queue 0: sent 1 packets
src=56:48:4F:53:54:01 - dst=DE:D0:06:AC:DC:3B - pool=mb_pool_0 - type=0x0800 - length=98 - nb_segs=1 - sw ptype: L2_ETHER L3_IPV4 - l2_len=14 - l3_len=20 - Send queue=0x0
ol_flags: RTE_MBUF_F_TX_L4_NO_CKSUM
port 0/queue 0: received 1 packets
src=DE:D0:06:AC:DC:3B - dst=56:48:4F:53:54:01 - pool=mb_pool_0 - type=0x0800 - length=98 - nb_segs=1 - sw ptype: L2_ETHER L3_IPV4 - l2_len=14 - l3_len=20 - Receive queue=0x0
ol_flags: RTE_MBUF_F_RX_L4_CKSUM_UNKNOWN RTE_MBUF_F_RX_IP_CKSUM_UNKNOWN RTE_MBUF_F_RX_OUTER_L4_CKSUM_UNKNOWN
Port 0 pkt-len=98 nb-segs=1
ETH: src=DE:D0:06:AC:DC:3B dst=56:48:4F:53:54:01 type=0x0800
IPV4: src=192.168.101.2 dst=192.168.101.3 proto=1 (ICMP)
ICMP: echo request seq id=225
port 0/queue 0: sent 1 packets
src=56:48:4F:53:54:01 - dst=DE:D0:06:AC:DC:3B - pool=mb_pool_0 - type=0x0800 - length=98 - nb_segs=1 - sw ptype: L2_ETHER L3_IPV4 - l2_len=14 - l3_len=20 - Send queue=0x0
ol_flags: RTE_MBUF_F_TX_L4_NO_CKSUM
port 0/queue 0: received 1 packets
src=DE:D0:06:AC:DC:3B - dst=56:48:4F:53:54:01 - pool=mb_pool_0 - type=0x0800 - length=98 - nb_segs=1 - sw ptype: L2_ETHER L3_IPV4 - l2_len=14 - l3_len=20 - Receive queue=0x0
ol_flags: RTE_MBUF_F_RX_L4_CKSUM_UNKNOWN RTE_MBUF_F_RX_IP_CKSUM_UNKNOWN RTE_MBUF_F_RX_OUTER_L4_CKSUM_UNKNOWN
Port 0 pkt-len=98 nb-segs=1
ETH: src=DE:D0:06:AC:DC:3B dst=56:48:4F:53:54:01 type=0x0800
IPV4: src=192.168.101.2 dst=192.168.101.3 proto=1 (ICMP)
ICMP: echo request seq id=226
port 0/queue 0: sent 1 packets
src=56:48:4F:53:54:01 - dst=DE:D0:06:AC:DC:3B - pool=mb_pool_0 - type=0x0800 - length=98 - nb_segs=1 - sw ptype: L2_ETHER L3_IPV4 - l2_len=14 - l3_len=20 - Send queue=0x0
ol_flags: RTE_MBUF_F_TX_L4_NO_CKSUM
port 0/queue 0: received 1 packets
src=DE:D0:06:AC:DC:3B - dst=56:48:4F:53:54:01 - pool=mb_pool_0 - type=0x0800 - length=98 - nb_segs=1 - sw ptype: L2_ETHER L3_IPV4 - l2_len=14 - l3_len=20 - Receive queue=0x0
ol_flags: RTE_MBUF_F_RX_L4_CKSUM_UNKNOWN RTE_MBUF_F_RX_IP_CKSUM_UNKNOWN RTE_MBUF_F_RX_OUTER_L4_CKSUM_UNKNOWN
Port 0 pkt-len=98 nb-segs=1
ETH: src=DE:D0:06:AC:DC:3B dst=56:48:4F:53:54:01 type=0x0800
IPV4: src=192.168.101.2 dst=192.168.101.3 proto=1 (ICMP)
ICMP: echo request seq id=227
port 0/queue 0: sent 1 packets
src=56:48:4F:53:54:01 - dst=DE:D0:06:AC:DC:3B - pool=mb_pool_0 - type=0x0800 - length=98 - nb_segs=1 - sw ptype: L2_ETHER L3_IPV4 - l2_len=14 - l3_len=20 - Send queue=0x0
ol_flags: RTE_MBUF_F_TX_L4_NO_CKSUM
port 0/queue 0: received 1 packets
src=DE:D0:06:AC:DC:3B - dst=56:48:4F:53:54:01 - pool=mb_pool_0 - type=0x0800 - length=98 - nb_segs=1 - sw ptype: L2_ETHER L3_IPV4 - l2_len=14 - l3_len=20 - Receive queue=0x0
ol_flags: RTE_MBUF_F_RX_L4_CKSUM_UNKNOWN RTE_MBUF_F_RX_IP_CKSUM_UNKNOWN RTE_MBUF_F_RX_OUTER_L4_CKSUM_UNKNOWN
Port 0 pkt-len=98 nb-segs=1
ETH: src=DE:D0:06:AC:DC:3B dst=56:48:4F:53:54:01 type=0x0800
IPV4: src=192.168.101.2 dst=192.168.101.3 proto=1 (ICMP)
ICMP: echo request seq id=228
port 0/queue 0: sent 1 packets
src=56:48:4F:53:54:01 - dst=DE:D0:06:AC:DC:3B - pool=mb_pool_0 - type=0x0800 - length=98 - nb_segs=1 - sw ptype: L2_ETHER L3_IPV4 - l2_len=14 - l3_len=20 - Send queue=0x0
ol_flags: RTE_MBUF_F_TX_L4_NO_CKSUM
port 0/queue 0: received 1 packets
src=DE:D0:06:AC:DC:3B - dst=56:48:4F:53:54:01 - pool=mb_pool_0 - type=0x0800 - length=98 - nb_segs=1 - sw ptype: L2_ETHER L3_IPV4 - l2_len=14 - l3_len=20 - Receive queue=0x0
ol_flags: RTE_MBUF_F_RX_L4_CKSUM_UNKNOWN RTE_MBUF_F_RX_IP_CKSUM_UNKNOWN RTE_MBUF_F_RX_OUTER_L4_CKSUM_UNKNOWN
Port 0 pkt-len=98 nb-segs=1
ETH: src=DE:D0:06:AC:DC:3B dst=56:48:4F:53:54:01 type=0x0800
IPV4: src=192.168.101.2 dst=192.168.101.3 proto=1 (ICMP)
ICMP: echo request seq id=229
port 0/queue 0: sent 1 packets
src=56:48:4F:53:54:01 - dst=DE:D0:06:AC:DC:3B - pool=mb_pool_0 - type=0x0800 - length=98 - nb_segs=1 - sw ptype: L2_ETHER L3_IPV4 - l2_len=14 - l3_len=20 - Send queue=0x0
ol_flags: RTE_MBUF_F_TX_L4_NO_CKSUM
stopport 0/queue 0: received 1 packets
src=DE:D0:06:AC:DC:3B - dst=56:48:4F:53:54:01 - pool=mb_pool_0 - type=0x0800 - length=98 - nb_segs=1 - sw ptype: L2_ETHER L3_IPV4 - l2_len=14 - l3_len=20 - Receive queue=0x0
ol_flags: RTE_MBUF_F_RX_L4_CKSUM_UNKNOWN RTE_MBUF_F_RX_IP_CKSUM_UNKNOWN RTE_MBUF_F_RX_OUTER_L4_CKSUM_UNKNOWN
Port 0 pkt-len=98 nb-segs=1
ETH: src=DE:D0:06:AC:DC:3B dst=56:48:4F:53:54:01 type=0x0800
IPV4: src=192.168.101.2 dst=192.168.101.3 proto=1 (ICMP)
ICMP: echo request seq id=230
port 0/queue 0: sent 1 packets
src=56:48:4F:53:54:01 - dst=DE:D0:06:AC:DC:3B - pool=mb_pool_0 - type=0x0800 - length=98 - nb_segs=1 - sw ptype: L2_ETHER L3_IPV4 - l2_len=14 - l3_len=20 - Send queue=0x0
ol_flags: RTE_MBUF_F_TX_L4_NO_CKSUM
Telling cores to stop...
Waiting for lcores to finish...
---------------------- Forward statistics for port 0 ----------------------
RX-packets: 15 RX-dropped: 0 RX-total: 15
TX-packets: 15 TX-dropped: 0 TX-total: 15
----------------------------------------------------------------------------
+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
RX-packets: 15 RX-dropped: 0 RX-total: 15
TX-packets: 15 TX-dropped: 0 TX-total: 15
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Done.
testpmd>
