Hi Elias, Thanks for the detailed report. I suspect you are correct, it seems to be related to the doorbell update to notify the NIC there are some work to do. Could you check https://gerrit.fd.io/r/c/vpp/+/27708 and report whether it fixes the issue?
Best ben > -----Original Message----- > From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Elias Rudberg > Sent: vendredi 26 juin 2020 11:13 > To: vpp-dev@lists.fd.io > Subject: [vpp-dev] RDMA problem in master and stable/2005, started with > commit introducing direct verb for Cx4/5 tx #mellanox #rdma > > Hello VPP experts, > > There seems to be a problem with the RDMA driver in VPP when using > Mellanox ConnectX5 network interfaces. This problem appears for the > master branch and for the stable/2005 branch, while stable/2001 does > not have this problem. > > The problem is that when a frame with 2 packets is to be sent, only the > first packets is sent directly while the second packet gets delayed. It > seems like the second packet is only sent later, when some other frame > with other packets is to be sent then the delayed earlier packet is > also sent. > > Perhaps this can go undetected if there is lots of traffic all the > time, if there is always new traffic to flush out any delayed packets > from earlier. So to reproduce it, it seems best to have a testing setup > with very little traffic such that there are several seconds without > any traffic, then it seems like packets can get delayed for several > seconds. Note that the delay is not seen inside VPP where packet traces > look like the packets are sent directly, VPP thinks they are sent but > it seems some packets are held in the NIC and only sent later on. > Monitoring traffic arriving at the other end shows that there was a > delay. > > The behavior seems reproducible, except when there is other traffic > being sent soon after since that causes the delayed packets to be sent. > > The specific case when this came up for us was when using VPP for NAT > with ipfix logging turned on, and doing some ping tests. Then when a > single ping echo request packet is to be NATed, that usually works fine > but sometimes there is also a ipfix logging packet to be sent, that > ends up in the same frame so that the frame has 2 packets. Then the > ipfix logging packet gets sent directly while the ICMP packet is > delayed, sometimes so much that the ping failed, it timed out. I don't > think the problem has anything to do with NAT or ipfix logging, it > seems like a more general problem with the rdma plugin. > > Testing previous commits indicates that the problem started with this > commit: > > dc812d9a7 (rdma: introduce direct verb for Cx4/5 tx, 2019-12-16) > > That commit exists in master and in stable/2005 but not in stable/2001 > which fits with that this problem is seen for master and stable/2005 > but not for stable/2001. > > Tried updating to the latest Mellanox driver (v5.0-2.1.8) but that did > not help. > > In the code in src/plugins/rdma/output.c it seems like the function > rdma_device_output_tx_mlx5() is handling the packets, but I was not > able to fully understand how it works. There is a concept of a > "doorbell" function call there, apparently the idea is that when > packets are to be sent, info about the packets is prepared and then the > "doorbell" is used to alert the NIC that there are things to send. From > my limited understanding, it seems like the doorbell currently results > in only the first packet is really being physically sent by the NIC > directly, while remaining packets are somehow stored and sent later. So > far I don't understand exactly why that happens or how to fix it. > > As a workaround, it seems to work to simply revert the entire rdma > plugin to the way it looks in the stable/2001 branch, then the problem > seems to disappear. But that probably means we lose performance gains > and other improvements in the newer code. > > Can someone with insight in the rdma plugin please help try to fix > this? > > Best regards, > Elias
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#16828): https://lists.fd.io/g/vpp-dev/message/16828 Mute This Topic: https://lists.fd.io/mt/75120690/21656 Mute #mellanox: https://lists.fd.io/g/fdio+vpp-dev/mutehashtag/mellanox Mute #rdma: https://lists.fd.io/g/fdio+vpp-dev/mutehashtag/rdma Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
