On 4/26/24 22:05, Gavin McKee wrote: > Thanks again for coming back on this Ilya, > > Another option I am looking at here is to switch the kernal path (Open > vSwitch kernel module) with OVS-DOCA as we are using the CX6/7 card > https://docs.nvidia.com/doca/archive/doca-v2.0.2/ovs-doca/index.html > > I'm trying to wrangle the documented Known Limitations > > - Only one insertion thread is supported (n-offload-threads=1). > - Only a single PF is currently supported. > - Geneve options are not supported. > - Only 250K connections are offloaded by CT offload. > - Only 8 CT zones are supported by CT offload. > - OVS restart on non-tunnel configuration is not supported. Remove all > ports before restarting. > > The one that concerns me is the 8 CT zones supported by CT offload, as > potentially with OVN I may have many CT zones if we have many customer > colocated on the same compute node. > > Seems any way I turn I'm getting kicked :D
You'll also need a special binary build from NVIDIA. The docs you linked are closed-source fork and not an upstream OVS. OVS doesn't support offloading of connection tracking in userspace datapath. > > Gav > > On Fri, 26 Apr 2024 at 12:42, Ilya Maximets <i.maxim...@ovn.org> wrote: >> >> On 4/26/24 20:12, Gavin McKee wrote: >>> Thanks for coming back to me on this. >>> >>> Moving kernal versions around is not a straightforward option here - >>> especially when you are using hardware offload . The OFED driver >>> version is coupled to the kernal so if we move from that we are out of >>> support coverage . >>> >>> Doing an ovn-appctl -t ovn-controller recompute does not resolve the >>> problem , again just taking a big hammer like restarting openvswitch >>> does. >>> >>> How would we proceed here ? Are there any Open vSwitch kernal module >>> patches we could try to get a resolution ? >> >> You can try the commit I linked below. That will mean that you'll >> need to re-build your kernel. There is no other way. >> >> In the past we had out-of-tree module, but it is deprecated for a >> long time, contains multiple issues and is unlikely to work with >> new kernels, especially heavily modified ones, like RHEL kernels. >> >> Note that the issue is not localized to OVS, but affects TC as well >> as they now share the NAT implementation. So, even if just swapping >> the openvswitch kernel module was possible, it wouldn't help much. >> >>> >>> One option we are looking at is regressing the entire stack back to Rocky >>> 9.1 . >> >> This may be an option. The bug I mentioned in a previous email exists >> in RHEL 9.3, so it exists in Rocky 9.3 as well, at least it should >> since they claim "bug-for-bug compatibility". So, 3 options to fix this >> particular bug (I don't know if it is causing your issue, but it is >> a severe bug that can potentially be a cause): >> >> 1. Re-build the kernel to include the fix. >> 2. Downgrade from 9.3 to an earlier release. >> 3. Wait for 9.4. >> >> Best regards, Ilya Maximets. >> >>> >>> Gav >>> >>> On Wed, 24 Apr 2024 at 04:44, Ilya Maximets <i.maxim...@ovn.org> wrote: >>>> >>>> On 4/23/24 17:39, Gavin McKee wrote: >>>>> If you look at both traces (non working and working) the thing that >>>>> stands out to me is this >>>>> >>>>> At line 10 in the working file the following entry exists >>>>> ct_state NEW tcp (SYN_SENT) orig [172.27.16.11.38793 > >>>>> 172.27.31.189.9100] reply [172.27.31.189.9100 > 172.27.16.11.38793] >>>>> zone 195 >>>>> >>>>> his doesn't happen in the non working file - I just see the following >>>>> >>>>> 3904992932904126 [swapper/125] 0 [kr] queue_userspace_packet >>>>> #ddf92049d47aeff1d0b6625620000 (skb 18382861792850905088) n 4 >>>>> if 38 (enp148s0f0_1) rxif 38 172.27.16.11.42303 > >>>>> 172.27.31.189.9100 ttl 64 tos 0x0 id 36932 off 0 [DF] len 52 proto TCP >>>>> (6) flags [S] seq 2266263186 win 42340 >>>>> upcall_enqueue (miss) (125/3904992932890052) q 3682874017 ret 0 >>>>> + 3904992932907247 [swapper/125] 0 [kr] ovs_dp_upcall >>>>> #ddf92049d47aeff1d0b6625620000 (skb 18382861792850905088) n 5 >>>>> if 38 (enp148s0f0_1) rxif 38 172.27.16.11.42303 > >>>>> 172.27.31.189.9100 ttl 64 tos 0x0 id 36932 off 0 [DF] len 52 proto TCP >>>>> (6) flags [S] seq 2266263186 win 42340 >>>>> upcall_ret (125/3904992932890052) ret 0 >>>>> >>>>> I am wondering if a failure to track the ct_state SYN is causing the >>>>> returning ACK to drop ? >>>>> >>>>> + 3904992936344421 [swapper/125] 0 [tp] skb:kfree_skb >>>>> #ddf9204d21c48ff1d0b676c330c00 (skb 18382861792850913792) n 3 drop >>>>> (TC_INGRESS) >>>>> if 33 (genev_sys_6081) rxif 33 172.27.31.189.9100 > >>>>> 172.27.16.11.42303 ttl 64 tos 0x0 id 0 off 0 [DF] len 52 proto TCP (6) >>>>> flags [S.] seq 605271182 ack 2266263187 win 42340 >>>>> >>>>> On Mon, 22 Apr 2024 at 18:54, Gavin McKee <gavmcke...@googlemail.com> >>>>> wrote: >>>>>> >>>>>> Ok @Adrian Moreno @Flavio Leitner >>>>>> >>>>>> Two more detailed Retis traces attached. One is not working - the >>>>>> same session that I can't establish a TCP session to on port 9010 >>>>>> 172.27.16.11.42303 > 172.27.31.189.9100 >>>>>> >>>>>> Then I restart Open vSwtich and try again >>>>>> 172.27.16.11.38793 > 172.27.31.189.9100 (this works post restart) >>>>>> >>>>>> It looks to me in the non working example that we - >>>>>> SEND SYN -> exits the tunnel interface genev_sys_6081 via >>>>>> enp148s0f0np0 - exactly as expected >>>>>> RECV ACK -> tcp_gro_receive then -> net:netif_receive_skb where we hit >>>>>> drop (TC_INGRESS) >>>>>> >>>>>> After a restart things seem to be very different >>>>>> >>>>>> Any ideas where to look next ? >>>> >>>> You mentioned that you're using 5.14.0-362.8.1.el9_3.x86 kernel. >>>> RHEL 9.3 contains a large refactor for OVS connection tracking, >>>> but it doesn't contain at least one fix for this refactor: >>>> >>>> https://github.com/torvalds/linux/commit/e6345d2824a3f58aab82428d11645e0da861ac13 >>>> >>>> This may cause all sorts of incorrect packet processing in the >>>> kernel. I'd suggest trying the latest upstream v6.8.7 kernel >>>> that has all the known fixes or try the 9.4 beta kernel that >>>> should have the fix mentioned above. Downgrading to 9.2 may also >>>> be an option since 9.2 doesn't contain the refactoring, IIRC. >>>> >>>> I'd not recommend running with this bug present anyway. >>>> >>>> Updating your OVN 23.09.1 to 23.09.3 may also be worth trying. >>>> The fact that OVS restart fixes the issue may also indicate a >>>> problem with incremental processing in ovn-controller. >>>> Next time the issue happens try to force flow recompute with >>>> ovn-appctl -t ovn-controller recompute >>>> And see if that fixes the issue. If it does, it would be great >>>> to have OpenFlow dumps before and after the recompute for >>>> comparison. >>>> >>>> Best regards, Ilya Maximets. >> _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
