Hi all,
I was checking the datapath flows before and after the upgrade and the
below second flow from each output (before/after the upgrade) has a
difference about the ct_mark:
##before the upgrade
#
recirc_id(0x1e6e85),tunnel(tun_id=0x1c,src=10.XX.XX.X3X,dst=10.XX.XX.4,tos=0x20,geneve({}{}),flags(-df+csum+key)),in_port(137),ct_state(-new+est-rel-rpl-inv+trk),ct_mark(0/0x1),eth(src=fa:16:3e:9b:b3:c6,dst=fa:16:3e:d7:c9:46),eth_type(0x86dd),ipv6(src=2000::/ffc0::,dst=2001:db8:2:a::301,proto=58,hlimit=62,frag=no),
packets:9, bytes:1062, used:0.074s, actions:1706
#
recirc_id(0),tunnel(tun_id=0x1c,src=10.XX.XX.X3X,dst=10.XX.XX.4,tos=0x20,geneve({class=0x102,type=0x80,len=4,0x60008/0x7fffffff}),flags(-df+csum+key)),in_port(137),ct_mark(0/0x2),eth(src=fa:16:3e:9b:b3:c6,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x86dd),ipv6(src=2000::/ffc0::,dst=2001:db8:2:a::301,proto=58,hlimit=62,frag=no),
packets:8, bytes:944, used:0.114s, actions:ct(zone=7185),recirc(0x1e6e85)
#
recirc_id(0),in_port(1706),eth(src=fa:16:3e:d7:c9:46,dst=fa:16:3e:9b:b3:c6),eth_type(0x86dd),ipv6(src=2001:db8:2:a::301,dst=2001:db8:1:2::10,proto=58,hlimit=255,frag=no),icmpv6(type=128/0xfc),
packets:8, bytes:944, used:0.122s, actions:ct(zone=7185),recirc(0x1e6e86)
##after the upgrade
#
recirc_id(0xae310f),tunnel(tun_id=0x1c,src=10.XX.XX.X3X,dst=10.XX.XX.4,tos=0x20,geneve({}{}),flags(-df+csum+key)),in_port(137),ct_state(-new+est-rel-rpl-inv+trk),ct_mark(0/0x1),eth(src=fa:16:3e:9b:b3:c6,dst=fa:16:3e:d7:c9:46),eth_type(0x86dd),ipv6(src=2000::/ffc0::,dst=2001:db8:2:a::301,proto=58,hlimit=62,frag=no),
packets:1569, bytes:185142, used:0.238s, actions:1706
#
recirc_id(0),tunnel(tun_id=0x1c,src=0.XX.XX.X3X,dst=10.XX.XX.4,tos=0x20,geneve({class=0x102,type=0x80,len=4,0x60008/0x7fffffff}),flags(-df+csum+key)),in_port(137),eth(src=fa:16:3e:9b:b3:c6,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x86dd),ipv6(src=2000::/ffc0::,dst=2001:db8:2:a::301,proto=58,hlimit=62,frag=no),icmpv6(type=128/0x80),
packets:1569, bytes:185142, used:0.362s,
actions:ct(zone=7185),recirc(0xae310f)
#
recirc_id(0),in_port(1706),eth(src=fa:16:3e:d7:c9:46,dst=fa:16:3e:9b:b3:c6),eth_type(0x86dd),ipv6(src=2001:db8:2:a::301,dst=2001:db8:1:2::10,proto=58,hlimit=255,frag=no),icmpv6(type=128/0xfc),
packets:0, bytes:0, used:never, actions:ct(zone=7185),recirc(0xae46f4)
After the upgrade, the second above flow is missing the ct_mark(0/0x2), so
that is interesting.
Regards,
Tiago Pires
On Fri, Apr 26, 2024 at 6:32 PM Tiago Pires <tiago.pi...@luizalabs.com>
wrote:
> Hi all,
>
> Enabling the debugging, we can see the following:
>
> 2024-04-26T20:54:25.880Z|00004|dpif(handler75)|DBG|system@ovs-system:
> miss upcall:
>
> recirc_id(0),dp_hash(0),skb_priority(0),in_port(1706),skb_mark(0),ct_state(0),ct_zone(0),ct_mark(0),ct_label(0),eth(src=fa:16:3e:d7:c9:46,dst=fa:16:3e:9b:b3:c6),eth_type(0x86dd),ipv6(src=2001:db8:2:a::301,dst=2001:db8:1:2::10,label=0xbde5f,proto=58,tclass=0,hlimit=255,frag=no),icmpv6(type=129,code=0)
> icmp6,vlan_tci=0x0000,dl_src=fa:16:3e:d7:c9:46,dl_dst=fa:16:3e:9b:b3:c6,ipv6_src=2001:db8:2:a::301,ipv6_dst=2001:db8:1:2::10,ipv6_label=0xbde5f,nw_tos=0,nw_ecn=0,nw_ttl=255,nw_frag=no,icmp_type=129,icmp_code=0
> icmp6_csum:e5d6
> #
> 2024-04-26T20:54:29.933Z|00042|dpif(revalidator131)|DBG|system@ovs-system:
> flow_del ufid:82c1cdb8-d56e-4d0b-b65c-96edecef76e1
> recirc_id(0),dp_hash(0),skb_priority(0),in_port(1706),skb_mark(0),ct_state(0),ct_zone(0),ct_mark(0),ct_label(0),eth(src=fa:16:3e:d7:c9:46,dst=fa:16:3e:9b:b3:c6),eth_type(0x86dd),ipv6(src=2001:db8:2:a::301,dst=2001:db8:1:2::10,label=0xbde5f,proto=58,tclass=0,hlimit=255,frag=no),icmpv6(type=129,code=0),
> packets:0, bytes:0, used:never
> #
> 2024-04-26T20:54:37.632Z|00173|dpif(handler23)|DBG|system@ovs-system:
> put[create] ufid:82c1cdb8-d56e-4d0b-b65c-96edecef76e1
> recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(1706),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=fa:16:3e:d7:c9:46,dst=fa:16:3e:9b:b3:c6),eth_type(0x86dd),ipv6(src=2001:db8:2:a::301,dst=2001:db8:1:2::10,label=0xbde5f/0,proto=58,tclass=0/0,hlimit=255,frag=no),icmpv6(type=129/0xfc,code=0/0),
> actions:ct(zone=7185),recirc(0x5ef91)
>
> There is this miss upcall and then the revalidator removes the flow from
> the datapath and it is added again. So, probably this is creating the
> jitter behavior.
>
> Regards,
>
> Tiago Pires
>
> On Fri, Apr 26, 2024 at 3:33 PM Tiago Pires <tiago.pi...@luizalabs.com>
> wrote:
>
>> Hi all,
>>
>> While testing the upgrade path from OVN 22.03.1/OVS 2.17.2 to OVN
>> 23.03.1/OVS 3.1.3 on Ubuntu 22.04/kernel 5.15 and 6.5 we are seeing a
>> strange behavior for icmpv6 traffic.
>> Before the upgrade a simple north-south or west-east ping between IPv6
>> hosts would have a low jitter like below:
>>
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2712 ttl=62 time=0.676 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2713 ttl=62 time=0.829 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2714 ttl=62 time=0.568 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2715 ttl=62 time=0.700 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2716 ttl=62 time=0.768 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2717 ttl=62 time=0.599 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2718 ttl=62 time=0.656 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2719 ttl=62 time=0.689 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2720 ttl=62 time=0.724 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2721 ttl=62 time=0.419 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2722 ttl=62 time=0.732 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2723 ttl=62 time=0.717 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2724 ttl=62 time=0.755 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2725 ttl=62 time=0.765 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2726 ttl=62 time=0.535 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2727 ttl=62 time=0.865 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2728 ttl=62 time=0.692 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2729 ttl=62 time=0.597 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2730 ttl=62 time=0.661 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=2731 ttl=62 time=0.558 ms
>>
>> But after the upgrade, the same ping started to have a higher jitter:
>>
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=37 ttl=253 time=2.14 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=38 ttl=253 time=50.2 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=39 ttl=253 time=57.0 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=40 ttl=253 time=61.5 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=41 ttl=253 time=2.16 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=42 ttl=253 time=1.68 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=43 ttl=253 time=1.63 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=44 ttl=253 time=3.32 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=45 ttl=253 time=1.87 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=46 ttl=253 time=39.6 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=47 ttl=253 time=2.87 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=48 ttl=253 time=60.0 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=49 ttl=253 time=1.79 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=50 ttl=253 time=2.06 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=51 ttl=253 time=2.45 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=52 ttl=253 time=2.10 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=53 ttl=253 time=4.39 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=54 ttl=253 time=2.91 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=55 ttl=253 time=1.79 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=56 ttl=253 time=1.80 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=57 ttl=253 time=2.26 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=58 ttl=253 time=55.1 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=59 ttl=253 time=57.2 ms
>> 64 bytes from 2001:db8:2:a::301: icmp_seq=60 ttl=253 time=3.34 ms
>>
>> --- 2001:db8:2:a::301 ping statistics ---
>> 60 packets transmitted, 60 received, 0% packet loss, time 59120ms
>> rtt min/avg/max/mdev = 0.531/16.329/61.464/23.395 ms
>>
>> The icmp v4 is not affected and we have the same jitter before and after
>> the upgrade. Regarding throughput, I ran a TCP/UDP (v4/v6) throughput test
>> before and after the upgrade and the numbers are similar, so it seems it
>> happens only in special with icmpv6 traffic.
>>
>> Checking the datapath, I can see the flow related with the in_port(1706)
>> where the VM is connected being removed and installed again:
>>
>> # ovs-dpctl dump-flows | grep 2001:db8:2:a::301
>> recirc_id(0x1ad3d),tunnel(tun_id=0x1c,src=10.26.73.135,dst=10.26.72.4,tos=0x20,geneve({}{}),flags(-df+csum+key)),in_port(137),ct_state(-new+est-rel-rpl-inv+trk),ct_mark(0/0x1),eth(src=fa:16:3e:9b:b3:c6,dst=fa:16:3e:d7:c9:46),eth_type(0x86dd),ipv6(src=2000::/ffc0::,dst=2001:db8:2:a::301,proto=58,hlimit=62,frag=no),
>> packets:7, bytes:826, used:0.674s, actions:1706
>> recirc_id(0),tunnel(tun_id=0x1c,src=10.26.73.135,dst=10.26.72.4,tos=0x20,geneve({class=0x102,type=0x80,len=4,0x60008/0x7fffffff}),flags(-df+csum+key)),in_port(137),eth(src=fa:16:3e:9b:b3:c6,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x86dd),ipv6(src=2000::/ffc0::,dst=2001:db8:2:a::301,proto=58,hlimit=62,frag=no),
>> packets:7, bytes:826, used:0.709s, actions:ct(zone=7185),recirc(0x1ad3d)
>>
>> # ovs-dpctl dump-flows | grep 2001:db8:2:a::301
>> recirc_id(0x1ad3d),tunnel(tun_id=0x1c,src=10.26.73.135,dst=10.26.72.4,tos=0x20,geneve({}{}),flags(-df+csum+key)),in_port(137),ct_state(-new+est-rel-rpl-inv+trk),ct_mark(0/0x1),eth(src=fa:16:3e:9b:b3:c6,dst=fa:16:3e:d7:c9:46),eth_type(0x86dd),ipv6(src=2000::/ffc0::,dst=2001:db8:2:a::301,proto=58,hlimit=62,frag=no),
>> packets:11, bytes:1298, used:0.190s, actions:1706
>> recirc_id(0),in_port(1706),eth(src=fa:16:3e:d7:c9:46,dst=fa:16:3e:9b:b3:c6),eth_type(0x86dd),ipv6(src=2001:db8:2:a::301,dst=2001:db8:1:2::10,proto=58,hlimit=255,frag=no),icmpv6(type=128/0xfc),
>> packets:0, bytes:0, used:never, actions:ct(zone=7185),recirc(0x1b13c)
>> recirc_id(0),tunnel(tun_id=0x1c,src=10.26.73.135,dst=10.26.72.4,tos=0x20,geneve({class=0x102,type=0x80,len=4,0x60008/0x7fffffff}),flags(-df+csum+key)),in_port(137),eth(src=fa:16:3e:9b:b3:c6,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x86dd),ipv6(src=2000::/ffc0::,dst=2001:db8:2:a::301,proto=58,hlimit=62,frag=no),
>> packets:11, bytes:1298, used:0.237s, actions:ct(zone=7185),recirc(0x1ad3d)
>>
>> (Note: no OVS HW Offloading)
>>
>> So, it seems there is a flow missing, the flow goes to userspace and it
>> is installed again on the datapath. Maybe it can explain the higher jitter.
>>
>> From what I explained, does anyone remember any recent changes that could
>> introduce such behavior? If anyone who has an IPv6 deployment and is using
>> the last OVN/OVS could test if there is similar behavior, I would
>> appreciate it.
>>
>> Regards,
>>
>> Tiago Pires
>>
>
--
_‘Esta mensagem é direcionada apenas para os endereços constantes no
cabeçalho inicial. Se você não está listado nos endereços constantes no
cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa
mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão
imediatamente anuladas e proibidas’._
* **‘Apesar do Magazine Luiza tomar
todas as precauções razoáveis para assegurar que nenhum vírus esteja
presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por
quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
