Finally I can confirm and reproduce when gateway mac ages out from mac
address table(or flush) ovs starts to flood all ports and flows switch to
software path. When It learn gateway mac address by any arp requests
hardware offload works again.
It seems increasing mac-aging on br-ex can be an option. What do you
suggest permanent solution for this issue?
Thanks,
Serhat
On Thu, May 22, 2025 at 19:50 Serhat Rıfat Demircan <
demircan.ser...@gmail.com> wrote:
> No OVN, it is only OVS. oftrace says it flooding because "no learned MAC
> for destination". Is this expected while there is existing traffic? And
> another question is, can I prevent flooding? :)
>
> # ovs-appctl ofproto/trace br-int
>> "in_port=4,ip,eth_src=fa:16:7e:e1:50:af,eth_dst=00:1c:73:aa:bb:cc,nw_src=X.X.X.X,nw_dst=Y.Y.Y.Y"
>> Flow:
>> ip,in_port=4,vlan_tci=0x0000,dl_src=fa:16:7e:e1:50:af,dl_dst=00:1c:73:aa:bb:cc,nw_src=X.X.X.X,nw_dst=Y.Y.Y.Y,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0,nw_frag=no
>
>
> bridge("br-int")
>> ----------------
>> 0. priority 0, cookie 0x64ecae89c6f9a081
>> goto_table:58
>> 58. priority 0, cookie 0x64ecae89c6f9a081
>> goto_table:60
>> 60. in_port=4, priority 100, cookie 0x64ecae89c6f9a081
>> set_field:0x4->reg5
>> set_field:0x1->reg6
>> resubmit(,71)
>> 71. ip,reg5=0x4,in_port=4,dl_src=fa:16:7e:e1:50:af,nw_src=X.X.X.X,
>> priority 65, cookie 0x64ecae89c6f9a081
>> ct(table=72,zone=NXM_NX_REG6[0..15])
>> drop
>> -> A clone of the packet is forked to recirculate. The forked
>> pipeline will be resumed at table 72.
>> -> Sets the packet to an untracked state, and clears all the
>> conntrack fields.
>
>
> Final flow:
>> ip,reg5=0x4,reg6=0x1,in_port=4,vlan_tci=0x0000,dl_src=fa:16:7e:e1:50:af,dl_dst=00:1c:73:aa:bb:cc,nw_src=X.X.X.X,nw_dst=Y.Y.Y.Y,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0,nw_frag=no
>> Megaflow:
>> recirc_id=0,ct_state=-trk,eth,ip,in_port=4,dl_src=fa:16:7e:e1:50:af,nw_src=X.X.X.X,nw_proto=0,nw_frag=no
>> Datapath actions: ct(zone=1),recirc(0x1)
>
>
>
>> ===============================================================================
>> recirc(0x1) - resume conntrack with default ct_state=trk|new (use
>> --ct-next to customize)
>>
>> ===============================================================================
>
>
> Flow:
>> recirc_id=0x1,ct_state=new|trk,ct_zone=1,eth,ip,reg5=0x4,reg6=0x1,in_port=4,vlan_tci=0x0000,dl_src=fa:16:7e:e1:50:af,dl_dst=00:1c:73:aa:bb:cc,nw_src=X.X.X.X,nw_dst=Y.Y.Y.Y,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0,nw_frag=no
>
>
> bridge("br-int")
>> ----------------
>> thaw
>> Resuming from table 72
>> 72. ct_state=+new-est,ip,reg5=0x4, priority 74, cookie 0x64ecae89c6f9a081
>> resubmit(,73)
>> 73. ct_state=+new-est,ip,reg5=0x4, priority 90, cookie 0x64ecae89c6f9a081
>> ct(commit,zone=NXM_NX_REG6[0..15])
>> drop
>> -> Sets the packet to an untracked state, and clears all the
>> conntrack fields.
>> resubmit(,91)
>> 91. priority 1, cookie 0x64ecae89c6f9a081
>> resubmit(,94)
>> 94.
>> reg6=0x1,dl_src=fa:16:7e:e1:50:af,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00,
>> priority 10, cookie 0x64ecae89c6f9a081
>> push_vlan:0x8100
>> set_field:4097->vlan_vid
>> output:1
>
>
> bridge("br-ex")
>> ---------------
>> 0. in_port=2,dl_vlan=1, priority 4, cookie 0xd9b277145604416
>> set_field:4207->vlan_vid
>> NORMAL
>> -> no learned MAC for destination, flooding
>
>
> Final flow:
>> recirc_id=0x1,eth,ip,reg5=0x4,reg6=0x1,in_port=4,dl_vlan=1,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=fa:16:7e:e1:50:af,dl_dst=00:1c:73:aa:bb:cc,nw_src=X.X.X.X,nw_dst=Y.Y.Y.Y,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0,nw_frag=no
>> Megaflow:
>> recirc_id=0x1,ct_state=+new-est-rel-rpl,eth,ip,in_port=4,dl_src=fa:16:7e:e1:50:af,dl_dst=00:1c:73:aa:bb:cc,nw_frag=no
>> Datapath actions: ct(commit,zone=1),push_vlan(vid=111,pcp=0),1,5
>
>
> Thanks,
> Serhat
>
> On Thu, May 22, 2025 at 6:41 PM Numan Siddique <num...@ovn.org> wrote:
>
>> On Thu, May 22, 2025 at 11:33 AM Serhat Rıfat Demircan via discuss
>> <ovs-discuss@openvswitch.org> wrote:
>> >
>> > I think this is the actual flow change in ovs side. When br-ex(br-ex is
>> external bridge, openstack is bond interface) is added as output interface
>> it breaks offloading. Still do not know why it changes :) Do you have any
>> idea why ovs decide to redirect traffic to external bridge?
>> >
>> > Offloaded flow:
>> > ufid:10d312f5-7c7a-4bab-8594-090bbcd9af53,
>> recirc_id(0x1),dp_hash(0/0),skb_priority(0/0),in_port(ens37f0npf0vf7),skb_mark(0/0),ct_state(0x22/0x2e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),packet_type(ns=0/0,id=0/0),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(src=
>> 0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no),
>> packets:1325324668, bytes:4434013761329, used:3.390s, offloaded:yes, dp:tc,
>> actions:push_vlan(vid=111,pcp=0),openstack
>> >
>> > Not offloaded flow:
>> > ufid:10d312f5-7c7a-4bab-8594-090bbcd9af53,
>> recirc_id(0x1),dp_hash(0/0),skb_priority(0/0),in_port(ens37f0npf0vf7),skb_mark(0/0),ct_state(0x22/0x2e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),packet_type(ns=0/0,id=0/0),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(src=
>> 0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no),
>> packets:1387549198, bytes:4643405901604, used:0.000s, dp:tc,
>> actions:push_vlan(vid=111,pcp=0),br-ex,openstack
>>
>> Maybe you run ofproto/trace on the sample traffic and see why it is
>> outputting to br-ex ?
>>
>> Is it with OVN ?
>>
>> Thanks
>> Numan
>>
>> >
>> > Thanks,
>> > Serhat
>> >
>> > On Wed, May 21, 2025 at 11:51 AM Serhat Rıfat Demircan <
>> demircan.ser...@gmail.com> wrote:
>> >>
>> >> After little more digging, I see miss upcalls and flow modify logs
>> like below while tc flow switching to not_in_hw. I also see similar upcall
>> while switching back to in_hw.
>> >>
>> >>
>> >> 2025-05-21T08:14:27.189Z|19748|dpif(handler3)|DBG|system@ovs-system:
>> miss upcall:
>> >>
>> recirc_id(0),dp_hash(0),skb_priority(0),in_port(6),skb_mark(0),ct_state(0),ct_zone(0),ct_mark(0),ct_label(0),eth(src=fa:16:7e:75:94:76,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(src=IP_A,dst=IP_B,proto=6,tos=0,ttl=64,frag=no),tcp(src=52774,dst=6835),tcp_flags(ack)
>> >>
>> tcp,vlan_tci=0x0000,dl_src=fa:16:7e:75:94:76,dl_dst=00:1c:73:aa:bb:cc,nw_src=IP_A,nw_dst=IP_B,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,tp_src=52774,tp_dst=6835,tcp_flags=ack
>> tcp_csum:5abe
>> >>
>> >>
>> 2025-05-21T08:14:27.189Z|104292|dpif(revalidator10)|DBG|system@ovs-system:
>> put[modify] ufid:52588bba-04e5-43fa-b677-5a96dbd11f22
>> recirc_id(0x5),dp_hash(0/0),skb_priority(0/0),in_port(4),skb_mark(0/0),ct_state(0x22/0xf),ct_zone(0x1/0),ct_mark(0/0),ct_label(0/0),ct_tuple4(src=IP_C/
>> 0.0.0.0,dst=IP_D/0.0.0.0
>> ,proto=6/0,tp_src=58586/0,tp_dst=6821/0),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(src=IP_C/
>> 0.0.0.0,dst=IP_D/0.0.0.0,proto=6/0,tos=0/0,ttl=64/0,frag=no),tcp(src=58586/0,dst=6821/0),tcp_flags(0/0),
>> actions:push_vlan(vid=111,pcp=0),1,5
>> >>
>> >> On Tue, May 20, 2025 at 7:49 PM Ilya Maximets <i.maxim...@ovn.org>
>> wrote:
>> >>>
>> >>> On 5/20/25 4:32 PM, Serhat Rıfat Demircan wrote:
>> >>> > Yes, statistics are updated.
>> >>> >
>> >>> > # ovs-appctl dpctl/dump-flows type=offloaded | grep
>> dst=00:1c:73:aa:bb:cc
>> >>> >
>> recirc_id(0x5),in_port(4),ct_state(+est-rel+rpl-inv+trk),ct_zone(0x1),ct_mark(0),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(frag=no),
>> packets:222, bytes:30340, used:0.170s, actions:push_vlan(vid=111,pcp=0),5
>> >>> >
>> recirc_id(0x5),in_port(4),ct_state(+est-rel-rpl+trk),eth(src=fa:16:7e:e1:50:af,dst=00:1c:73:aa:bb:cc),eth_type(0x0800),ipv4(frag=no),
>> packets:54220862, bytes:182391451911, used:0.170s,
>> actions:push_vlan(vid=111,pcp=0),5
>> >>>
>> >>> Then one of the options would be to turn on debug logging and try
>> >>> and catch the moment the flow actions change, and then correlate the
>> >>> change with some other events in the log. I'd suggest to start with:
>> >>> ovs-appctl vlog/set ofproto_dpif_xlate:file:dbg dpif:file:dbg
>> >>>
>> >>> Best regards, Ilya Maximets.
>> >>>
>> >>> >
>> >>> > On Tue, May 20, 2025 at 3:39 PM Ilya Maximets <i.maxim...@ovn.org
>> <mailto:i.maxim...@ovn.org>> wrote:
>> >>> >
>> >>> > On 5/20/25 2:20 PM, Serhat Rıfat Demircan wrote:
>> >>> > > Redirecting to br-ex happens even if there is a single
>> hardware offloaded port on hypervisor. Still can't find the actual reason.
>> >>> > > Increasing mac-aging-time and mac-table-size did not help
>> also.
>> >>> > >
>> >>> > > By the way, it is bidirectional storage(ceph) traffic.
>> >>> >
>> >>> > Is statistics on the datapath flows properly updated?
>> >>> > i.e. if you watch 'ovs-appctl dpctl/dump-flows', do you see
>> stats
>> >>> > properly incremented when the flow is offloaded?
>> >>> >
>> >>> > >
>> >>> > >
>> >>> > >
>> >>> > > On Wed, May 14, 2025 at 1:39 PM Ilya Maximets <
>> i.maxim...@ovn.org <mailto:i.maxim...@ovn.org> <mailto:i.maxim...@ovn.org
>> <mailto:i.maxim...@ovn.org>>> wrote:
>> >>> > >
>> >>> > > On 5/8/25 4:15 PM, Serhat Rıfat Demircan via discuss
>> wrote:
>> >>> > > > Hello Everyone,
>> >>> > > >
>> >>> > > > I've been testing OVS hardware offloaded ports with
>> Openstack for a while.
>> >>> > > > I have ConnectX-5 and ConnectX-6 cards in my lab. Iperf
>> tests are looking
>> >>> > > > promising, but when I try to use hardware offloaded
>> ports for krbd traffic,
>> >>> > > > after some time, traffic falls back to the software
>> path. Another interesting
>> >>> > > > thing for me is without changing anything, it can also
>> switch back to the
>> >>> > > > hardware path. I could not find the real reason.
>> >>> > > >
>> >>> > > > I'm not sure if this is a configuration issue or if I'm
>> hitting to cards limit.
>> >>> > > > If any of you guys are experienced with this all ideas
>> are more than welcome.
>> >>> > > >
>> >>> > > > Openstack version: Caracal
>> >>> > > > Operation System: Ubuntu 22.04
>> >>> > > > OVS version: 3.3.0
>> >>> > > >
>> >>> > > > I see tc flow change:
>> >>> > > >
>> >>> > > > *From Offloaded:*
>> >>> > > > $ tc filter show dev ens37f0npf0vf6 ingress
>> >>> > > > filter protocol ip pref 2 flower chain 0
>> >>> > > > filter protocol ip pref 2 flower chain 0 handle 0x1
>> >>> > > > dst_mac 00:1c:73:aa:bb:cc
>> >>> > > > src_mac fa:16:7e:2e:8c:f0
>> >>> > > > eth_type ipv4
>> >>> > > > ip_flags nofrag
>> >>> > > > skip_sw
>> >>> > > > in_hw in_hw_count 1
>> >>> > > > action order 1: vlan push id 111 protocol 802.1Q
>> priority 0 pipe
>> >>> > > > index 4 ref 1 bind 1
>> >>> > > > no_percpu
>> >>> > > > used_hw_stats delayed
>> >>> > > >
>> >>> > > > action order 2: mirred (Egress Redirect to device
>> openstack) stolen
>> >>> > > > index 4 ref 1 bind 1
>> >>> > > > cookie 605b82e5b94ef3fa63c3e78349851a27
>> >>> > > > no_percpu
>> >>> > > > used_hw_stats delayed
>> >>> > > >
>> >>> > > > *To software path:*
>> >>> > > > $ tc filter show dev ens37f0npf0vf6 ingress
>> >>> > > > filter protocol ip pref 2 flower chain 0
>> >>> > > > filter protocol ip pref 2 flower chain 0 handle 0x1
>> >>> > > > dst_mac 00:1c:73:aa:bb:cc
>> >>> > > > src_mac fa:16:7e:2e:8c:f0
>> >>> > > > eth_type ipv4
>> >>> > > > ip_flags nofrag
>> >>> > > > not_in_hw
>> >>> > > > action order 1: vlan push id 111 protocol 802.1Q
>> priority 0 pipe
>> >>> > > > index 4 ref 1 bind 1
>> >>> > > > no_percpu
>> >>> > > >
>> >>> > > > action order 2: skbedit ptype host pipe
>> >>> > > > index 11 ref 1 bind 1
>> >>> > > >
>> >>> > > > action order 3: mirred (Ingress Mirror to device br-ex)
>> pipe
>> >>> > > > index 13 ref 1 bind 1
>> >>> > > > cookie 5037716e444e83b81c4ef9844df2377d
>> >>> > > > no_percpu
>> >>> > > >
>> >>> > > > action order 4: mirred (Egress Redirect to device
>> openstack) stolen
>> >>> > > > index 15 ref 1 bind 1
>> >>> > > > cookie 5037716e444e83b81c4ef9844df2377d
>> >>> > > > no_percpu
>> >>> > >
>> >>> > > Here you can see that traffic is getting broadcasted on
>> all ports in the
>> >>> > > br-ex bridge, and since some of these ports are virtual,
>> this operation
>> >>> > > cannot be offloaded to hardware.
>> >>> > >
>> >>> > > I would guess, after some time the FDB entry expires and
>> OVS no longer
>> >>> > > knows on which port the traffic supposed to be sent out.
>> Is that a
>> >>> > > unidirection traffic? i.e. are you also sending traffic
>> in the opposite
>> >>> > > direction? If not, are you sending some periodic ARP
>> requests for the
>> >>> > > bridge to learn where the destination is?
>> >>> > >
>> >>> > > If the traffic is bidirectional, then it might be a NIC
>> driver/firmware
>> >>> > > issue not providing updated stats.
>> >>> > >
>> >>> > > Best regards, Ilya Maximets.
>> >>> > >
>> >>> >
>> >>>
>> > _______________________________________________
>> > discuss mailing list
>> > disc...@openvswitch.org
>> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>>
>
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
