I think this is the actual flow change in ovs side. When br-ex(br-ex is external bridge, openstack is bond interface) is added as output interface it breaks offloading. Still do not know why it changes :) Do you have any idea why ovs decide to redirect traffic to external bridge?
*Offloaded flow:* ufid:10d312f5-7c7a-4bab-8594-090bbcd9af53, recirc_id(0x1),dp_hash(0/0),skb_priority(0/0),in_port(ens37f0npf0vf7),skb_mark(0/0),ct_state(0x22/0x2e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),packet_type(ns=0/0,id=0/0),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(src= 0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:1325324668, bytes:4434013761329, used:3.390s, offloaded:yes, dp:tc, actions:push_vlan(vid=111,pcp=0),openstack *Not offloaded flow:* ufid:10d312f5-7c7a-4bab-8594-090bbcd9af53, recirc_id(0x1),dp_hash(0/0),skb_priority(0/0),in_port(ens37f0npf0vf7),skb_mark(0/0),ct_state(0x22/0x2e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),packet_type(ns=0/0,id=0/0),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(src= 0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:1387549198, bytes:4643405901604, used:0.000s, dp:tc, actions:push_vlan(vid=111,pcp=0),br-ex,openstack Thanks, Serhat On Wed, May 21, 2025 at 11:51 AM Serhat Rıfat Demircan < demircan.ser...@gmail.com> wrote: > After little more digging, I see miss upcalls and flow modify logs like > below while tc flow switching to not_in_hw. I also see similar upcall while > switching back to in_hw. > > > 2025-05-21T08:14:27.189Z|19748|dpif(handler3)|DBG|system@ovs-system: miss > upcall: > > recirc_id(0),dp_hash(0),skb_priority(0),in_port(6),skb_mark(0),ct_state(0),ct_zone(0),ct_mark(0),ct_label(0),eth(src=fa:16:7e:75:94:76,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(src=IP_A,dst=IP_B,proto=6,tos=0,ttl=64,frag=no),tcp(src=52774,dst=6835),tcp_flags(ack) > tcp,vlan_tci=0x0000,dl_src=fa:16:7e:75:94:76,dl_dst=00:1c:73:aa:bb:cc,nw_src=IP_A,nw_dst=IP_B,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,tp_src=52774,tp_dst=6835,tcp_flags=ack > tcp_csum:5abe > > 2025-05-21T08:14:27.189Z|104292|dpif(revalidator10)|DBG|system@ovs-system: > put[modify] ufid:52588bba-04e5-43fa-b677-5a96dbd11f22 > recirc_id(0x5),dp_hash(0/0),skb_priority(0/0),in_port(4),skb_mark(0/0),ct_state(0x22/0xf),ct_zone(0x1/0),ct_mark(0/0),ct_label(0/0),ct_tuple4(src=IP_C/ > 0.0.0.0,dst=IP_D/0.0.0.0 > ,proto=6/0,tp_src=58586/0,tp_dst=6821/0),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(src=IP_C/ > 0.0.0.0,dst=IP_D/0.0.0.0,proto=6/0,tos=0/0,ttl=64/0,frag=no),tcp(src=58586/0,dst=6821/0),tcp_flags(0/0), > actions:push_vlan(vid=111,pcp=0),1,5 > > On Tue, May 20, 2025 at 7:49 PM Ilya Maximets <i.maxim...@ovn.org> wrote: > >> On 5/20/25 4:32 PM, Serhat Rıfat Demircan wrote: >> > Yes, statistics are updated. >> > >> > # ovs-appctl dpctl/dump-flows type=offloaded | grep >> dst=00:1c:73:aa:bb:cc >> > >> recirc_id(0x5),in_port(4),ct_state(+est-rel+rpl-inv+trk),ct_zone(0x1),ct_mark(0),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(frag=no), >> packets:222, bytes:30340, used:0.170s, actions:push_vlan(vid=111,pcp=0),5 >> > >> recirc_id(0x5),in_port(4),ct_state(+est-rel-rpl+trk),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(frag=no), >> packets:54220862, bytes:182391451911, used:0.170s, >> actions:push_vlan(vid=111,pcp=0),5 >> >> Then one of the options would be to turn on debug logging and try >> and catch the moment the flow actions change, and then correlate the >> change with some other events in the log. I'd suggest to start with: >> ovs-appctl vlog/set ofproto_dpif_xlate:file:dbg dpif:file:dbg >> >> Best regards, Ilya Maximets. >> >> > >> > On Tue, May 20, 2025 at 3:39 PM Ilya Maximets <i.maxim...@ovn.org >> <mailto:i.maxim...@ovn.org>> wrote: >> > >> > On 5/20/25 2:20 PM, Serhat Rıfat Demircan wrote: >> > > Redirecting to br-ex happens even if there is a single hardware >> offloaded port on hypervisor. Still can't find the actual reason. >> > > Increasing mac-aging-time and mac-table-size did not help also. >> > > >> > > By the way, it is bidirectional storage(ceph) traffic. >> > >> > Is statistics on the datapath flows properly updated? >> > i.e. if you watch 'ovs-appctl dpctl/dump-flows', do you see stats >> > properly incremented when the flow is offloaded? >> > >> > > >> > > >> > > >> > > On Wed, May 14, 2025 at 1:39 PM Ilya Maximets <i.maxim...@ovn.org >> <mailto:i.maxim...@ovn.org> <mailto:i.maxim...@ovn.org <mailto: >> i.maxim...@ovn.org>>> wrote: >> > > >> > > On 5/8/25 4:15 PM, Serhat Rıfat Demircan via discuss wrote: >> > > > Hello Everyone, >> > > > >> > > > I've been testing OVS hardware offloaded ports with >> Openstack for a while. >> > > > I have ConnectX-5 and ConnectX-6 cards in my lab. Iperf >> tests are looking >> > > > promising, but when I try to use hardware offloaded ports >> for krbd traffic, >> > > > after some time, traffic falls back to the software path. >> Another interesting >> > > > thing for me is without changing anything, it can also >> switch back to the >> > > > hardware path. I could not find the real reason. >> > > > >> > > > I'm not sure if this is a configuration issue or if I'm >> hitting to cards limit. >> > > > If any of you guys are experienced with this all ideas are >> more than welcome. >> > > > >> > > > Openstack version: Caracal >> > > > Operation System: Ubuntu 22.04 >> > > > OVS version: 3.3.0 >> > > > >> > > > I see tc flow change: >> > > > >> > > > *From Offloaded:* >> > > > $ tc filter show dev ens37f0npf0vf6 ingress >> > > > filter protocol ip pref 2 flower chain 0 >> > > > filter protocol ip pref 2 flower chain 0 handle 0x1 >> > > > dst_mac 00:1c:73:aa:bb:cc >> > > > src_mac fa:16:7e:2e:8c:f0 >> > > > eth_type ipv4 >> > > > ip_flags nofrag >> > > > skip_sw >> > > > in_hw in_hw_count 1 >> > > > action order 1: vlan push id 111 protocol 802.1Q priority >> 0 pipe >> > > > index 4 ref 1 bind 1 >> > > > no_percpu >> > > > used_hw_stats delayed >> > > > >> > > > action order 2: mirred (Egress Redirect to device >> openstack) stolen >> > > > index 4 ref 1 bind 1 >> > > > cookie 605b82e5b94ef3fa63c3e78349851a27 >> > > > no_percpu >> > > > used_hw_stats delayed >> > > > >> > > > *To software path:* >> > > > $ tc filter show dev ens37f0npf0vf6 ingress >> > > > filter protocol ip pref 2 flower chain 0 >> > > > filter protocol ip pref 2 flower chain 0 handle 0x1 >> > > > dst_mac 00:1c:73:aa:bb:cc >> > > > src_mac fa:16:7e:2e:8c:f0 >> > > > eth_type ipv4 >> > > > ip_flags nofrag >> > > > not_in_hw >> > > > action order 1: vlan push id 111 protocol 802.1Q priority >> 0 pipe >> > > > index 4 ref 1 bind 1 >> > > > no_percpu >> > > > >> > > > action order 2: skbedit ptype host pipe >> > > > index 11 ref 1 bind 1 >> > > > >> > > > action order 3: mirred (Ingress Mirror to device br-ex) pipe >> > > > index 13 ref 1 bind 1 >> > > > cookie 5037716e444e83b81c4ef9844df2377d >> > > > no_percpu >> > > > >> > > > action order 4: mirred (Egress Redirect to device >> openstack) stolen >> > > > index 15 ref 1 bind 1 >> > > > cookie 5037716e444e83b81c4ef9844df2377d >> > > > no_percpu >> > > >> > > Here you can see that traffic is getting broadcasted on all >> ports in the >> > > br-ex bridge, and since some of these ports are virtual, this >> operation >> > > cannot be offloaded to hardware. >> > > >> > > I would guess, after some time the FDB entry expires and OVS >> no longer >> > > knows on which port the traffic supposed to be sent out. Is >> that a >> > > unidirection traffic? i.e. are you also sending traffic in >> the opposite >> > > direction? If not, are you sending some periodic ARP >> requests for the >> > > bridge to learn where the destination is? >> > > >> > > If the traffic is bidirectional, then it might be a NIC >> driver/firmware >> > > issue not providing updated stats. >> > > >> > > Best regards, Ilya Maximets. >> > > >> > >> >>
_______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
