On 3/3/25 3:53 PM, support wrote: > Sorry for mispelling Dumitru! Y is damn close to U, didn't get use to > new keyboard yet.
No worries! :) > Little clarification: > We making instances public through L2 network. Meanwhile we also have 1 > router (OpenStack entity) for private network where kubernetes works > with IP attached from public network to let kubernetes nodes have access > to internet. > >> Hi Dumitry, >> Thank you for you reply. Just tried to change broadcast-arps-to-all- >> routers to false. It really helped for about some seconds (i was able >> to request IP address using dhclient immediately) then it returned >> back to the same behaviour and ovs-vswitchd having CPU overload. >> node1: >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >> COMMAND >> 139416 root 20 0 8390536 521684 8448 R 98.7 0.1 2353:23 >> ovs-vswitchd >> 139511 root 20 0 8390536 521684 8448 R 11.3 0.1 143:54.62 >> revalidator92 >> 139523 root 20 0 8390536 521684 8448 R 11.3 0.1 126:49.06 >> revalidator104 >> 139528 root 20 0 8390536 521684 8448 R 10.7 0.1 133:27.08 >> revalidator109 >> 139510 root 20 0 8390536 521684 8448 R 10.3 0.1 126:13.46 >> revalidator91 >> 139519 root 20 0 8390536 521684 8448 R 10.0 0.1 143:28.01 >> revalidator100 >> 139522 root 20 0 8390536 521684 8448 R 10.0 0.1 91:53.11 >> revalidator103 >> 139526 root 20 0 8390536 521684 8448 R 10.0 0.1 134:08.83 >> revalidator107 >> 139531 root 20 0 8390536 521684 8448 S 10.0 0.1 99:15.78 >> revalidator112 >> 139527 root 20 0 8390536 521684 8448 R 9.7 0.1 40:55.66 >> revalidator108 >> 139516 root 20 0 8390536 521684 8448 R 9.3 0.1 94:20.53 >> revalidator97 >> 139509 root 20 0 8390536 521684 8448 S 8.0 0.1 194:47.10 >> revalidator90 >> 139517 root 20 0 8390536 521684 8448 S 5.3 0.1 48:20.00 >> revalidator98 >> 139520 root 20 0 8390536 521684 8448 R 5.3 0.1 126:20.74 >> revalidator101 >> 139497 root 20 0 8390536 521684 8448 S 5.0 0.1 143:10.67 >> handler78 >> 139513 root 20 0 8390536 521684 8448 S 5.0 0.1 93:14.85 >> revalidator94 >> 139514 root 20 0 8390536 521684 8448 R 5.0 0.1 42:13.77 >> revalidator95 >> 139521 root 20 0 8390536 521684 8448 S 5.0 0.1 92:28.66 >> revalidator102 >> 139518 root 20 0 8390536 521684 8448 R 3.7 0.1 14:07.25 >> revalidator99 >> >> node2: >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >> COMMAND >> 3255815 root 20 0 8383004 342144 8448 R 99.9 0.0 996:44.31 >> ovs-vswitchd >> 3255922 root 20 0 8383004 342144 8448 S 9.6 0.0 47:23.99 >> revalidator102 >> 3255927 root 20 0 8383004 342144 8448 S 9.3 0.0 64:20.30 >> revalidator107 >> 3255929 root 20 0 8383004 342144 8448 S 9.3 0.0 64:32.72 >> revalidator109 >> 3255920 root 20 0 8383004 342144 8448 S 8.3 0.0 75:16.18 >> revalidator100 >> 3255921 root 20 0 8383004 342144 8448 S 8.3 0.0 65:45.10 >> revalidator101 >> 3255923 root 20 0 8383004 342144 8448 S 8.3 0.0 34:15.98 >> revalidator103 >> 3255924 root 20 0 8383004 342144 8448 S 8.3 0.0 65:31.02 >> revalidator104 >> 3255910 root 20 0 8383004 342144 8448 S 8.0 0.0 93:24.25 >> revalidator90 >> 3255912 root 20 0 8383004 342144 8448 S 8.0 0.0 74:40.90 >> revalidator92 >> 3255926 root 20 0 8383004 342144 8448 S 8.0 0.0 46:46.51 >> revalidator106 >> 3255914 root 20 0 8383004 342144 8448 S 7.6 0.0 47:15.57 >> revalidator94 >> 3255917 root 20 0 8383004 342144 8448 S 5.6 0.0 33:24.94 >> revalidator97 >> 3255918 root 20 0 8383004 342144 8448 S 4.3 0.0 24:15.37 >> revalidator98 >> 3255911 root 20 0 8383004 342144 8448 S 4.0 0.0 65:40.29 >> revalidator91 >> 3255858 root 20 0 8383004 342144 8448 S 3.3 0.0 48:44.56 >> handler38 >> 3255897 root 20 0 8383004 342144 8448 S 1.7 0.0 0:37.98 >> handler77 >> 3255828 root 20 0 8383004 342144 8448 S 1.3 0.0 11:48.57 >> urcu8 >> 3255884 root 20 0 8383004 342144 8448 S 0.7 0.0 2:17.81 >> handler64 >> 3255913 root 20 0 8383004 342144 8448 S 0.7 0.0 65:57.06 >> revalidator93 >> 3255916 root 20 0 8383004 342144 8448 S 0.7 0.0 24:13.42 >> revalidator96 >> >> Yep, there is a lot of logs of pinctrl received packet-in. >> 2025-03-03T14:23:46.919Z|49223|pinctrl(ovn_pinctrl0)|DBG|pinctrl received >> packet-in | opcode=unrecognized(27)| OF_Table_ID=43| >> OF_Cookie_ID=0x435b65d6| in-port=4| src-mac=30:b6:4f:5f:db:a0, >> dst-mac=ff:ff:ff:ff:ff:ff| src-ip=83.219.243.1, dst-ip=83.219.243.143 >> 2025-03-03T14:23:46.919Z|49224|pinctrl(ovn_pinctrl0)|DBG|pinctrl received >> packet-in | opcode=unrecognized(27)| OF_Table_ID=43| >> OF_Cookie_ID=0x435b65d6| in-port=4| src-mac=30:b6:4f:5f:db:a0, >> dst-mac=ff:ff:ff:ff:ff:ff| src-ip=83.219.243.1, dst-ip=83.219.243.57 >> 2025-03-03T14:23:46.919Z|49225|pinctrl(ovn_pinctrl0)|DBG|pinctrl received >> packet-in | opcode=unrecognized(27)| OF_Table_ID=43| >> OF_Cookie_ID=0x435b65d6| in-port=4| src-mac=30:b6:4f:5f:db:a0, >> dst-mac=ff:ff:ff:ff:ff:ff| src-ip=109.206.241.1, dst-ip=109.206.241.187 >> 2025-03-03T14:23:46.919Z|49226|pinctrl(ovn_pinctrl0)|DBG|pinctrl received >> packet-in | opcode=unrecognized(27)| OF_Table_ID=43| >> OF_Cookie_ID=0x435b65d6| in-port=4| src-mac=30:b6:4f:5f:db:a0, >> dst-mac=ff:ff:ff:ff:ff:ff| src-ip=83.219.243.1, dst-ip=83.219.243.193 >> 2025-03-03T14:23:46.919Z|49227|pinctrl(ovn_pinctrl0)|DBG|pinctrl received >> packet-in | opcode=unrecognized(27)| OF_Table_ID=43| >> OF_Cookie_ID=0x435b65d6| in-port=4| src-mac=30:b6:4f:5f:db:a0, >> dst-mac=ff:ff:ff:ff:ff:ff| src-ip=83.219.243.1, dst-ip=83.219.243.154 >> 2025-03-03T14:23:46.919Z|49228|pinctrl(ovn_pinctrl0)|DBG|pinctrl received >> packet-in | opcode=unrecognized(27)| OF_Table_ID=43| >> OF_Cookie_ID=0x435b65d6| in-port=65533| src-mac=30:b6:4f:5f:db:a0, >> dst-mac=ff:ff:ff:ff:ff:ff| src-ip=83.219.243.1, dst-ip=83.219.243.126 >> 2025-03-03T14:23:46.919Z|49229|pinctrl(ovn_pinctrl0)|DBG|pinctrl received >> packet-in | opcode=unrecognized(27)| OF_Table_ID=43| >> OF_Cookie_ID=0x435b65d6| in-port=4| src-mac=30:b6:4f:5f:db:a0, >> dst-mac=ff:ff:ff:ff:ff:ff| src-ip=89.169.15.1, dst-ip=89.169.15.238 >> 2025-03-03T14:23:46.919Z|49230|pinctrl(ovn_pinctrl0)|DBG|pinctrl received >> packet-in | opcode=unrecognized(27)| OF_Table_ID=43| >> OF_Cookie_ID=0x435b65d6| in-port=65533| src-mac=30:b6:4f:5f:db:a0, >> dst-mac=ff:ff:ff:ff:ff:ff| src-ip=83.217.210.1, dst-ip=83.217.210.148 >> 2025-03-03T14:23:46.919Z|49231|pinctrl(ovn_pinctrl0)|DBG|pinctrl received >> packet-in | opcode=unrecognized(27)| OF_Table_ID=43| >> OF_Cookie_ID=0x435b65d6| in-port=4| src-mac=30:b6:4f:5f:db:a0, >> dst-mac=ff:ff:ff:ff:ff:ff| src-ip=83.219.243.1, dst-ip=83.219.243.214 >> 2025-03-03T14:23:46.919Z|49232|pinctrl(ovn_pinctrl0)|DBG|pinctrl received >> packet-in | opcode=unrecognized(27)| OF_Table_ID=43| >> OF_Cookie_ID=0x435b65d6| in-port=65533| src-mac=30:b6:4f:5f:db:a0, >> dst-mac=ff:ff:ff:ff:ff:ff| src-ip=83.219.243.1, dst-ip=83.219.243.146 >> 2025-03-03T14:23:46.919Z|49233|pinctrl(ovn_pinctrl0)|DBG|pinctrl received >> packet-in | opcode=unrecognized(27)| OF_Table_ID=43| >> OF_Cookie_ID=0x435b65d6| in-port=4| src-mac=30:b6:4f:5f:db:a0, >> dst-mac=ff:ff:ff:ff:ff:ff| src-ip=89.169.15.1, dst-ip=89.169.15.28 >> Right, that explains why setting the option to not broadcast arp requests doesn't do much in your case. These are IP packets with L2 broadcast destination IPs. But being L2 broadcast they seem to hit the same mc_group split rules as I suspected in my previous email. >> These are just few lines, more is available in gist to not dump it all there. >> https://gist.githubusercontent.com/frct1/ >> b6baebfb12a6132d8c5597024049f88f/raw/ >> dabd26c2e2a7bf467717d6ee034926ce5f71d273/pinctrl%2520logs.txt >> What surprises me though is that they actually need to be sent to reg15=0x8000 (MC_FLOOD). In your gist src-mac 30:b6:4f:5f:db:a0 is by far the most common, would you happen to know who that corresponds to? Is it legit IP traffic with L2 broadcast destination? >> > I'm guessing your topology might be similar (a "large" public logical >> > switch with quite a few connected logical routers). >> Exactly! There is vlan network with about 1100 ports in total using >> just L2 connections with router that acts as gateway for kubernetes >> cluster. >>> Hi Ilia, >>> >>> Thanks for all the additional information! >>> >>> On 3/2/25 8:55 PM, support via discuss wrote: >>> > Probably i found something interesting here. >>> > top -H -p $(pidof ovs-vswitchd) >>> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>> > COMMAND >>> > 673931 root 20 0 8392700 462408 7040 R 99.9 0.1 1216:19 >>> > ovs-vswitchd >>> > 673941 root 20 0 8392700 462408 7040 S 6.7 0.1 4:18.00 >>> > handler10 >>> > 673982 root 20 0 8392700 462408 7040 S 6.7 0.1 0:49.26 >>> > handler51 >>> > 674009 root 20 0 8392700 462408 7040 S 6.7 0.1 73:17.17 >>> > handler78 >>> > 673932 root 20 0 8392700 462408 7040 S 0.0 0.1 2:07.06 >>> > handler1 >>> > 673933 root 20 0 8392700 462408 7040 S 0.0 0.1 0:34.96 >>> > handler2 >>> > 673934 root 20 0 8392700 462408 7040 S 0.0 0.1 3:57.85 >>> > handler3 >>> > 673935 root 20 0 8392700 462408 7040 S 0.0 0.1 0:41.66 >>> > handler4 >>> > 673936 root 20 0 8392700 462408 7040 S 0.0 0.1 18:46.97 >>> > urcu5 >>> > 673937 root 20 0 8392700 462408 7040 S 0.0 0.1 4:04.14 >>> > handler6 >>> > 673938 root 20 0 8392700 462408 7040 S 0.0 0.1 0:44.03 >>> > handler7 >>> > 673939 root 20 0 8392700 462408 7040 S 0.0 0.1 4:20.97 >>> > handler8 >>> > >>> > I guess this could be a real issue of lost DHCP packets. >>> > Next are top lines of perf record >>> > 1.19% ovs-vswitchd libc.so.6 [.] realloc >>> > >>> > 0.64% revalidator90 ovs-vswitchd [.] >>> 0x00000000000797de >>> > >>> > 0.59% ovs-vswitchd ovs-vswitchd [.] >>> 0x00000000000487de >>> > >>> > 0.58% ovs-vswitchd ovs-vswitchd [.] >>> 0x00000000000797de >>> > >>> > 0.58% ovs-vswitchd [kernel.kallsyms] [k] >>> available_idle_cpu >>> > >>> > 0.57% ovs-vswitchd [kernel.kallsyms] [k] tun_net_xmit >>> >>> > >>> > 0.55% ovs-vswitchd ovs-vswitchd [.] >>> 0x00000000000a561a >>> > >>> > 0.51% ovs-vswitchd libc.so.6 [.] >>> 0x0000000000191747 >>> > >>> > 0.50% ovs-vswitchd [kernel.kallsyms] [k] task_h_load >>> > >>> > 0.48% ovs-vswitchd [kernel.kallsyms] [k] llist_add_batch >>> > >>> > 0.48% revalidator90 ovs-vswitchd [.] >>> 0x00000000000797ca >>> > >>> > 0.47% revalidator96 ovs-vswitchd [.] >>> 0x00000000000797de >>> > >>> > 0.45% revalidator90 ovs-vswitchd [.] >>> 0x00000000000487de >>> > >>> > 0.44% revalidator93 ovs-vswitchd [.] >>> 0x00000000000487de >>> > >>> > 0.43% revalidator93 ovs-vswitchd [.] >>> 0x00000000000797de >>> > >>> > 0.40% ovs-vswitchd ovs-vswitchd [.] >>> 0x00000000000797ca >>> > >>> > 0.40% revalidator112 ovs-vswitchd [.] >>> 0x00000000000797de >>> > >>> > 0.39% ovs-vswitchd [kernel.kallsyms] [k] __dev_queue_xmit >>> > >>> > I've definitely saw bug reports regarding libc and openvswitch. >>> > >>> > >>> >> I'm back to digging into this more deep. >>> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>> >> COMMAND >>> >> 4446 root 20 0 395500 88276 6336 S 1.0 0.0 13:07.66 >>> >> ovn_pinctrl0 >>> >> 3773 root 20 0 395500 88276 6336 S 0.0 0.0 5:19.43 >>> >> ovn-controller >>> >> 4449 root 20 0 395500 88276 6336 S 0.0 0.0 0:00.00 >>> >> urcu1 >>> >> 4452 root 20 0 395500 88276 6336 S 0.0 0.0 0:00.07 >>> >> ovn_statctrl2 >>> >> 5832 root 20 0 395500 88276 6336 S 0.0 0.0 0:04.07 >>> >> stopwatch3 >>> >> >>> >> Interesting thing that ovn-controller as well as pinctrl doesn't seems >>> >> starvating for CPU as it is usage is likely about 1%. >>> >> Host CPU is also feeling a way good >>> >> %Cpu(s): 29.5 us, 20.0 sy, 0.0 ni, 50.0 id, 0.0 wa, 0.0 hi, 0.6 >>> >> si, 0.0 st >>> >> >>> >> Despite that, ovs-vswitchd seems consuming way more CPU >>> >> >>> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>> >> COMMAND >>> >> 139416 root 20 0 8378828 452288 8448 R 98.7 0.1 990:55.85 >>> >> ovs-vswitchd >>> >> 139509 root 20 0 8378828 452288 8448 S 10.7 0.1 82:49.98 >>> >> revalidator90 >>> >> 139522 root 20 0 8378828 452288 8448 R 10.7 0.1 32:01.96 >>> >> revalidator103 >>> >> 139519 root 20 0 8378828 452288 8448 R 10.3 0.1 54:28.10 >>> >> revalidator100 >>> >> 139516 root 20 0 8378828 452288 8448 S 10.0 0.1 32:34.74 >>> >> revalidator97 >>> >> 139526 root 20 0 8378828 452288 8448 R 10.0 0.1 49:16.24 >>> >> revalidator107 >>> >> 139531 root 20 0 8378828 452288 8448 R 10.0 0.1 40:45.87 >>> >> revalidator112 >>> >> 139523 root 20 0 8378828 452288 8448 S 9.7 0.1 49:39.00 >>> >> revalidator104 >>> >> 139512 root 20 0 8378828 452288 8448 S 7.7 0.1 49:37.77 >>> >> revalidator93 >>> >> 139497 root 20 0 8378828 452288 8448 S 7.3 0.1 67:31.93 >>> >> handler78 >>> >> 139513 root 20 0 8378828 452288 8448 S 7.3 0.1 36:44.18 >>> >> revalidator94 >>> >> 139525 root 20 0 8378828 452288 8448 S 6.3 0.1 36:08.10 >>> >> revalidator106 >>> >> 139520 root 20 0 8378828 452288 8448 S 5.7 0.1 49:48.41 >>> >> revalidator101 >>> >> 139521 root 20 0 8378828 452288 8448 R 5.7 0.1 36:22.60 >>> >> revalidator102 >>> >> 139510 root 20 0 8378828 452288 8448 R 5.0 0.1 48:53.38 >>> >> revalidator91 >>> >> 139511 root 20 0 8378828 452288 8448 R 5.0 0.1 54:09.25 >>> >> revalidator92 >>> >> >>> >> More logs: >>> >> 15:44:45.190923 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, >>> >> Request from fa:16:3e:50:22:c4, length 300 >>> >> 15:44:46.446755 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, >>> >> Request from fa:16:3e:50:22:c4, length 300 >>> >> 15:44:47.491209 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, >>> >> Request from fa:16:3e:50:22:c4, length 300 >>> >> 15:44:47.869273 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, >>> >> Request from fa:16:3e:50:22:c4, length 300 >>> >> 15:44:48.297140 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, >>> >> Request from fa:16:3e:50:22:c4, length 300 >>> >> 15:44:51.190709 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, >>> >> Request from fa:16:3e:50:22:c4, length 300 >>> >> >>> >> Manually tried to use dhclient to obtain IP but no success. There is >>> >> logs that ovn-controller sent DHCPOFFER but no packet arrived to VM >>> >> interface. >>> >>> Switched back to configuration with no neutron-dhcp-agent. I have >>> synced >>> >>> ovn db with neutron-ovn-db-sync-util. >>> >>> >>> >>> There is coverage/show result of node where i'm seeing this issue >>> right >>> >>> now (current network bandwidth is about 300Mbps). >>> >>> # ovs-appctl coverage/show >>> >>> Event coverage, avg rate over last: 5 seconds, last minute, last >>> hour, >>> >>> hash=982b7b40: >>> >>> nln_changed 0.0/sec 0.000/sec 0.0197/sec >>> total: 8568 >>> >>> route_table_dump 0.0/sec 0.000/sec 0.0072/sec >>> total: 3953 >>> >>> netlink_sent 1158.4/sec 1109.700/sec 1018.6758/sec >>> total: >>> >>> 840427925 >>> >>> netlink_recv_jumbo 144.2/sec 140.883/sec 212.7489/sec >>> total: >>> >>> 127752548 >>> >>> netlink_received 1258.4/sec 1229.533/sec 1192.0311/sec >>> total: >>> >>> 1072129295 >>> >>> netdev_set_ethtool 0.0/sec 0.000/sec 0.0006/sec >>> total: 361 >>> >>> netdev_get_ethtool 0.0/sec 0.000/sec 0.0039/sec >>> total: 2808 >>> >>> netdev_get_ifindex 0.0/sec 0.000/sec 0.3358/sec total: >>> >>> 14218 >>> >>> netdev_set_policing 0.0/sec 0.000/sec 0.0028/sec >>> total: 1685 >>> >>> vconn_sent 596.2/sec 576.967/sec 585.4778/sec >>> total: >>> >>> 643013987 >>> >>> vconn_received 320.0/sec 310.883/sec 307.5108/sec >>> total: >>> >>> 360905481 >>> >>> util_xalloc 937354.8/sec 912615.150/sec 873946.4494/ >>> sec >>> >>> total: 1068314545738 >>> >>> unixctl_replied 0.0/sec 0.033/sec 0.0331/sec total: >>> >>> 33146 >>> >>> unixctl_received 0.0/sec 0.033/sec 0.0331/sec total: >>> >>> 33146 >>> >>> long_poll_interval 0.0/sec 0.000/sec 0.0000/sec >>> total: 1 >>> >>> stream_open 0.0/sec 0.000/sec 0.0000/sec >>> total: 2 >>> >>> pstream_open 0.0/sec 0.000/sec 0.0000/sec >>> total: 5 >>> >>> seq_change 3283.6/sec 3255.167/sec 3039.7792/sec >>> total: >>> >>> 3088392125 >>> >>> rconn_sent 596.2/sec 576.967/sec 585.4767/sec >>> total: >>> >>> 643013979 >>> >>> rconn_queued 596.2/sec 576.967/sec 585.5044/sec >>> total: >>> >>> 643014079 >>> >>> rconn_overflow 0.0/sec 0.000/sec 2.0272/sec total: >>> >>> 16543 >>> >>> This means we're tail dropping messages that should be sent to the >>> controller because we're reaching the queue size limit (100 by default >>> IIUC). >>> >>> >>> rconn_discarded 0.0/sec 0.000/sec 0.0278/sec >>> total: 100 >>> >>> This indicates that a controller connection (ovn-controller has a few) >>> has disconnected while there weere messages still queued on it to be >>> sent to the controller. >>> >>> >>> poll_zero_timeout 275.2/sec 247.600/sec 161.9242/sec >>> total: >>> >>> 136657138 >>> >>> poll_create_node 4749.6/sec 4417.217/sec 4062.3500/sec >>> total: >>> >>> 4007153255 >>> >>> txn_success 0.2/sec 0.183/sec 0.1828/sec total: >>> >>> 184805 >>> >>> txn_incomplete 0.4/sec 0.367/sec 0.4856/sec total: >>> >>> 372203 >>> >>> txn_unchanged 0.0/sec 0.000/sec 0.1169/sec >>> total: 7146 >>> >>> netdev_get_stats 77.6/sec 71.133/sec 70.5939/sec total: >>> >>> 73986502 >>> >>> mac_learning_moved 0.0/sec 0.000/sec 0.0000/sec >>> total: 425 >>> >>> mac_learning_expired 0.6/sec 0.217/sec 0.4439/sec total: >>> >>> 409542 >>> >>> mac_learning_learned 0.2/sec 0.233/sec 0.4403/sec total: >>> >>> 410444 >>> >>> hmap_expand 298.0/sec 265.833/sec 329.0514/sec >>> total: >>> >>> 291663955 >>> >>> hmap_pathological 10.4/sec 6.050/sec 9.1406/sec total: >>> >>> 8189560 >>> >>> hindex_expand 0.0/sec 0.000/sec 0.0000/sec >>> total: 12 >>> >>> hindex_pathological 0.0/sec 0.000/sec 0.0000/sec >>> total: 148 >>> >>> miniflow_malloc 0.0/sec 0.000/sec 21.7272/sec total: >>> >>> 3395676 >>> >>> flow_extract 843.4/sec 801.333/sec 727.9828/sec >>> total: >>> >>> 755385765 >>> >>> dpif_port_del 0.0/sec 0.000/sec 0.0006/sec >>> total: 379 >>> >>> dpif_port_add 0.0/sec 0.000/sec 0.0006/sec >>> total: 367 >>> >>> dpif_flow_put_error 6.0/sec 7.633/sec 5.1642/sec total: >>> >>> 4420389 >>> >>> dpif_flow_put 232.2/sec 208.867/sec 126.8286/sec >>> total: >>> >>> 98257942 >>> >>> dpif_flow_get 0.0/sec 0.000/sec 0.0000/sec >>> total: 386 >>> >>> dpif_flow_flush 0.0/sec 0.000/sec 0.0000/sec >>> total: 1 >>> >>> dpif_flow_del 182.2/sec 203.100/sec 118.5047/sec >>> total: >>> >>> 93717525 >>> >>> dpif_execute_with_help 0.0/sec 0.000/sec 0.0017/sec >>> total: 1649 >>> >>> dpif_execute_error 0.0/sec 0.000/sec 0.0006/sec >>> total: 398 >>> >>> dpif_execute 571.8/sec 537.350/sec 439.8069/sec >>> total: >>> >>> 472799472 >>> >>> cmap_shrink 0.8/sec 0.733/sec 1.4025/sec total: >>> >>> 1180907 >>> >>> cmap_expand 1.0/sec 0.583/sec 1.5000/sec total: >>> >>> 1182882 >>> >>> ccmap_shrink 0.0/sec 0.000/sec 0.1786/sec >>> total: 747 >>> >>> ccmap_expand 0.0/sec 0.000/sec 0.1200/sec >>> total: 774 >>> >>> connmgr_async_unsent 0.0/sec 0.000/sec 0.0372/sec >>> total: 139 >>> >>> xlate_actions 4171.0/sec 3071.683/sec 3133.7436/sec >>> total: >>> >>> 3107954003 >>> >>> upcall_ukey_replace 0.0/sec 0.117/sec 0.1739/sec total: >>> >>> 155715 >>> >>> upcall_ukey_contention 0.0/sec 0.000/sec 0.0003/sec >>> total: 157 >>> >>> upcall_flow_limit_grew 0.0/sec 0.550/sec 0.2328/sec total: >>> >>> 56359 >>> >>> ukey_replace_contention 0.0/sec 0.017/sec 0.0008/sec >>> total: 847 >>> >>> revalidate_missed_dp_flow 0.0/sec 0.117/sec 0.0472/sec total: >>> >>> 106876 >>> >>> handler_duplicate_upcall 14.0/sec 12.817/sec 11.5303/sec total: >>> >>> 9317449 >>> >>> dumped_new_flow 0.0/sec 0.000/sec 0.0000/sec >>> total: 842 >>> >>> dumped_duplicate_flow 0.0/sec 0.000/sec 0.0003/sec total: >>> >>> 37029 >>> >>> rev_mac_learning 0.8/sec 0.417/sec 0.6419/sec total: >>> >>> 708403 >>> >>> rev_flow_table 0.0/sec 0.000/sec 0.0033/sec >>> total: 4003 >>> >>> rev_port_toggled 0.0/sec 0.000/sec 0.0000/sec >>> total: 119 >>> >>> rev_reconfigure 0.0/sec 0.000/sec 0.0008/sec >>> total: 691 >>> >>> ofproto_update_port 0.0/sec 0.000/sec 0.0039/sec >>> total: 2430 >>> >>> ofproto_recv_openflow 320.0/sec 310.883/sec 307.5097/sec >>> total: >>> >>> 360905473 >>> >>> ofproto_packet_out 319.6/sec 310.633/sec 299.3358/sec >>> total: >>> >>> 360555564 >>> >>> ofproto_flush 0.0/sec 0.000/sec 0.0000/sec >>> total: 1 >>> >>> bridge_reconfigure 0.0/sec 0.000/sec 0.1125/sec >>> total: 5108 >>> >>> 100 events never hit >>> >>> >>> >>> > Do you see any warning/error logs in ovs-vswitchd.log? >>> >>> I see no noticable errors except usual warn logs like invalid packet. >>> >>> >>> >>> > Just double checking, you're saying that with low numbers of >>> instances >>> >>> > you don't have this issue? >>> >>> Exactly. With relatively low instances count there is no visible >>> issue. >>> >>> >>> >>> > I wonder if you're not potentially hitting the maximum resubmit >>> limit >>> >>> > (4096 resubmits) due to DHCPOFFERS being sent as broadcast. There >>> >>> > should be warning logs indicating that in ovs-vswitchd. >>> >>> >>> >>> There is no error or warn logs mentioning resubmit, but a lot of such >>> >>> logs: >>> >>> >>> >>> 2025-03-01T11:27:16.473Z|19884389|vconn|DBG|unix#33063: received: >>> >>> OFPT_PACKET_OUT (OF1.5) (xid=0xe3465): in_port=CONTROLLER >>> >>> actions=set_field:0x300->reg0,set_field:0x9->reg11,set_field:0xa- >>> >>> >reg12,set_field:0xe0->reg13,set_field:0x1->reg14,set_field:0x291- >>> >>> >reg15,set_field:0x2->metadata,set_field:0- >>> >>> >pkt_mark,set_field:0x9002->reg6,set_field:0x8000- >>> >>> >reg15,resubmit(CONTROLLER,43) >>> >>> data_len=60 >>> >>> 2025-03-01T11:27:16.972Z|19884399|vconn|DBG|unix#33063: received: >>> >>> OFPT_PACKET_OUT (OF1.5) (xid=0xe358a): in_port=CONTROLLER >>> >>> actions=set_field:0x300->reg0,set_field:0x9->reg11,set_field:0xa- >>> >>> >reg12,set_field:0x93->reg13,set_field:0x1->reg14,set_field:0xdd- >>> >>> >reg15,set_field:0x2->metadata,set_field:0- >>> >>> >pkt_mark,set_field:0x9001->reg6,set_field:0x8000- >>> >>> >reg15,resubmit(CONTROLLER,43) >>> >>> data_len=60 >>> >>> 2025-03-01T11:27:17.873Z|19884417|vconn|DBG|unix#33063: received: >>> >>> OFPT_PACKET_OUT (OF1.5) (xid=0xe3765): in_port=CONTROLLER >>> >>> actions=set_field:0x300->reg0,set_field:0x9->reg11,set_field:0xa- >>> >>> >reg12,set_field:0x93->reg13,set_field:0x1->reg14,set_field:0xdd- >>> >>> >reg15,set_field:0x2->metadata,set_field:0- >>> >>> >pkt_mark,set_field:0x9001->reg6,set_field:0x8000- >>> >>> >reg15,resubmit(CONTROLLER,43) >>> >>> data_len=60 >>> >>> 2025-03-01T11:27:17.972Z|19884419|vconn|DBG|unix#33063: received: >>> >>> OFPT_PACKET_OUT (OF1.5) (xid=0xe37ba): in_port=CONTROLLER >>> >>> actions=set_field:0x300->reg0,set_field:0x9->reg11,set_field:0xa- >>> >>> >reg12,set_field:0xe0->reg13,set_field:0x1->reg14,set_field:0x291- >>> >>> >reg15,set_field:0x2->metadata,set_field:0- >>> >>> >pkt_mark,set_field:0x9002->reg6,set_field:0x8000- >>> >>> >reg15,resubmit(CONTROLLER,43) >>> >>> data_len=60 >>> >>> These are due to the fact that OVN's multicast groups have been split >>> since commit >>> https://github.com/ovn-org/ovn/ >>> commit/325c7b203d8bfd12bc1285ad11390c1a55cd6717 >>> >>> A similar problem has been reported/discussed in: >>> https://mail.openvswitch.org/pipermail/ovs-discuss/2025- >>> February/053455.html >>> >>> To confirm that it's ARP requests (broadcast) that's causing problems in >>> your case, we'd need some more logs from you: >>> >>> ovn-appctl vlog/set pinctrl:dbg >>> >>> We're interested in logs that look like: >>> 2025-02-12T20:35:41.490Z|10791|pinctrl(ovn_pinctrl0)|DBG|pinctrl >>> received packet-in | opcode=... >>> >>> I'm guessing your topology might be similar (a "large" public logical >>> switch with quite a few connected logical routers). If so, a potential >>> solution discussed there was to set >>> "other_config:broadcast-arps-to-all-routers=false" on the public logical >>> switch. Would you be able to check if changing that setting helps in >>> your case too please? >>> >>> Thanks, >>> Dumitru >>> >>> >>> >>> >>> I've checked presence of error/warn logs at ovs-vswitchd using next >>> >>> command >>> >>> # cat /var/log/kolla/openvswitch/ovs-vswitchd.log | grep -i >>> "resubmit" | >>> >>> grep -iE "warn|err >>> >>> >>> >>> Also i have non-deployed region running 2024.2 OpenStack and did some >>> >>> dumb tests like creating 1500 VMs with about 500 instances per >>> >>> hypervisor and didn't see this issue. However, i didn't use L2 >>> network >>> >>> in this case, only virtual network like geneve/vxlan. I will try >>> today >>> >>> to create sample L2 vlan with mac learning enabled at the border >>> router >>> >>> and report back. I think this bug will not occur if there is no a >>> lot of >>> >>> packets coming through ovs, handling DHCP with no issues... >>> >>> >>> >>> 24.02.2025 16:00, Dumitru Ceara пишет: >>> >>> > On 2/20/25 11:13 PM, Ilia Baikov via discuss wrote: >>> >>> >> Hi >>> >>> >> >>> >>> > Hi Ilia, >>> >>> > >>> >>> > + Ilya Maximets for more ovs insights. >>> >>> > >>> >>> >> Over a lot of observes and testing various number of possible >>> >>> fixes came >>> >>> >> with nothing... >>> >>> >> >>> >>> >> Yes, there are messages that contains NXT_RESUME substring. >>> >>> >> >>> >>> >> ovn-controller: >>> >>> >> 2025-02-18T21:05:51.771Z|1041129|vconn(ovn_pinctrl0)|DBG|unix:/ >>> >>> var/run/ >>> >>> >> openvswitch/br-int.mgmt: sent (Success): NXT_RESUME (OF1.5) >>> (xid=0x0): >>> >>> >> cookie=0x404b44bc total_len=348 >>> >>> >> >>> >>> >>> reg0=0x308,reg11=0x9,reg12=0xa,reg13=0x13c,reg14=0x4a8,metadata=0x2,in_port=312 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:08:23.729Z|1044255|vconn(ovn_pinctrl0)|DBG|unix:/ >>> >>> var/run/ >>> >>> >> openvswitch/br-int.mgmt: sent (Success): NXT_RESUME (OF1.5) >>> (xid=0x0): >>> >>> >> cookie=0xffe9d519 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x308,reg11=0x9,reg12=0xa,reg13=0x112,reg14=0x2dd,metadata=0x2,in_port=363 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:10:33.865Z|1046917|vconn(ovn_pinctrl0)|DBG|unix:/ >>> >>> var/run/ >>> >>> >> openvswitch/br-int.mgmt: sent (Success): NXT_RESUME (OF1.5) >>> (xid=0x0): >>> >>> >> cookie=0x5bc96bd6 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x28b,reg11=0x9,reg12=0xa,reg13=0x57,reg14=0x423,metadata=0x2,in_port=83 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:11:06.733Z|1047596|vconn(ovn_pinctrl0)|DBG|unix:/ >>> >>> var/run/ >>> >>> >> openvswitch/br-int.mgmt: sent (Success): NXT_RESUME (OF1.5) >>> (xid=0x0): >>> >>> >> cookie=0x5bc96bd6 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x28b,reg11=0x9,reg12=0xa,reg13=0x5f,reg14=0x413,metadata=0x2,in_port=91 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:12:16.984Z|1048995|vconn(ovn_pinctrl0)|DBG|unix:/ >>> >>> var/run/ >>> >>> >> openvswitch/br-int.mgmt: sent (Success): NXT_RESUME (OF1.5) >>> (xid=0x0): >>> >>> >> cookie=0xfded3de0 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x308,reg11=0x9,reg12=0xa,reg13=0xc5,reg14=0x354,metadata=0x2,in_port=373 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:13:03.396Z|1049934|vconn(ovn_pinctrl0)|DBG|unix:/ >>> >>> var/run/ >>> >>> >> openvswitch/br-int.mgmt: sent (Success): NXT_RESUME (OF1.5) >>> (xid=0x0): >>> >>> >> cookie=0xfded3de0 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x308,reg11=0x9,reg12=0xa,reg13=0xc5,reg14=0x354,metadata=0x2,in_port=373 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:19:48.641Z|1058227|vconn(ovn_pinctrl0)|DBG|unix:/ >>> >>> var/run/ >>> >>> >> openvswitch/br-int.mgmt: sent (Success): NXT_RESUME (OF1.5) >>> (xid=0x0): >>> >>> >> cookie=0xbbfea837 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x308,reg11=0x9,reg12=0xa,reg13=0x132,reg14=0x434,metadata=0x2,in_port=302 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:19:50.069Z|1058255|vconn(ovn_pinctrl0)|DBG|unix:/ >>> >>> var/run/ >>> >>> >> openvswitch/br-int.mgmt: sent (Success): NXT_RESUME (OF1.5) >>> (xid=0x0): >>> >>> >> cookie=0x5bc96bd6 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x28b,reg11=0x9,reg12=0xa,reg13=0x6f,reg14=0x148,metadata=0x2,in_port=107 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:21:38.224Z|1060474|vconn(ovn_pinctrl0)|DBG|unix:/ >>> >>> var/run/ >>> >>> >> openvswitch/br-int.mgmt: sent (Success): NXT_RESUME (OF1.5) >>> (xid=0x0): >>> >>> >> cookie=0x5bc96bd6 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x28b,reg11=0x9,reg12=0xa,reg13=0xdb,reg14=0x336,metadata=0x2,in_port=215 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:21:42.735Z|1060568|vconn(ovn_pinctrl0)|DBG|unix:/ >>> >>> var/run/ >>> >>> >> openvswitch/br-int.mgmt: sent (Success): NXT_RESUME (OF1.5) >>> (xid=0x0): >>> >>> >> cookie=0xffe9d519 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x308,reg11=0x9,reg12=0xa,reg13=0x112,reg14=0x2dd,metadata=0x2,in_port=363 >>> (via action) data_len=348 (unbuffered) >>> >>> >> >>> >>> >> ovs-vswitchd >>> >>> >> 2025-02-18T20:55:45.673Z|1086321|vconn|DBG|unix#0: received: >>> >>> NXT_RESUME >>> >>> >> (OF1.5) (xid=0x0): cookie=0x58482a66 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x308,reg11=0x9,reg12=0xa,reg13=0x151,reg14=0x4b1,metadata=0x2,in_port=375 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:01:05.872Z|1092877|vconn|DBG|unix#0: received: >>> >>> NXT_RESUME >>> >>> >> (OF1.5) (xid=0x0): cookie=0x983e6272 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x308,reg11=0x9,reg12=0xa,reg13=0x15f,reg14=0x4d3,metadata=0x2,in_port=368 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:01:23.672Z|1093239|vconn|DBG|unix#0: received: >>> >>> NXT_RESUME >>> >>> >> (OF1.5) (xid=0x0): cookie=0xfded3de0 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x308,reg11=0x9,reg12=0xa,reg13=0xc5,reg14=0x354,metadata=0x2,in_port=373 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:04:03.272Z|1096513|vconn|DBG|unix#0: received: >>> >>> NXT_RESUME >>> >>> >> (OF1.5) (xid=0x0): cookie=0xfded3de0 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x308,reg11=0x9,reg12=0xa,reg13=0xc5,reg14=0x354,metadata=0x2,in_port=373 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:07:36.773Z|1100887|vconn|DBG|unix#0: received: >>> >>> NXT_RESUME >>> >>> >> (OF1.5) (xid=0x0): cookie=0xfded3de0 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x308,reg11=0x9,reg12=0xa,reg13=0xc5,reg14=0x354,metadata=0x2,in_port=373 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:07:51.472Z|1101187|vconn|DBG|unix#0: received: >>> >>> NXT_RESUME >>> >>> >> (OF1.5) (xid=0x0): cookie=0xa6d16298 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x308,reg11=0x9,reg12=0xa,reg13=0x4b,reg14=0x1bb,metadata=0x2,in_port=349 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:08:08.673Z|1101537|vconn|DBG|unix#0: received: >>> >>> NXT_RESUME >>> >>> >> (OF1.5) (xid=0x0): cookie=0x8b831d17 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x308,reg11=0x9,reg12=0xa,reg13=0x26,reg14=0x123,metadata=0x2,in_port=34 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:09:09.173Z|1102776|vconn|DBG|unix#0: received: >>> >>> NXT_RESUME >>> >>> >> (OF1.5) (xid=0x0): cookie=0xbf424a23 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x308,reg11=0x9,reg12=0xa,reg13=0x85,reg14=0x2d9,metadata=0x2,in_port=371 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:13:03.574Z|1108046|vconn|DBG|unix#0: received: >>> >>> NXT_RESUME >>> >>> >> (OF1.5) (xid=0x0): cookie=0xfded3de0 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x308,reg11=0x9,reg12=0xa,reg13=0xc5,reg14=0x354,metadata=0x2,in_port=373 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:17:18.772Z|1113273|vconn|DBG|unix#0: received: >>> >>> NXT_RESUME >>> >>> >> (OF1.5) (xid=0x0): cookie=0x5bc96bd6 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x28b,reg11=0x9,reg12=0xa,reg13=0x7c,reg14=0x241,metadata=0x2,in_port=120 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:18:51.673Z|1115178|vconn|DBG|unix#0: received: >>> >>> NXT_RESUME >>> >>> >> (OF1.5) (xid=0x0): cookie=0xfded3de0 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x308,reg11=0x9,reg12=0xa,reg13=0xc5,reg14=0x354,metadata=0x2,in_port=373 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:19:36.872Z|1116105|vconn|DBG|unix#0: received: >>> >>> NXT_RESUME >>> >>> >> (OF1.5) (xid=0x0): cookie=0x5bc96bd6 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x28b,reg11=0x9,reg12=0xa,reg13=0x87,reg14=0x4b,metadata=0x2,in_port=131 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:19:52.172Z|1116417|vconn|DBG|unix#0: received: >>> >>> NXT_RESUME >>> >>> >> (OF1.5) (xid=0x0): cookie=0xfded3de0 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x308,reg11=0x9,reg12=0xa,reg13=0xc5,reg14=0x354,metadata=0x2,in_port=373 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:20:24.572Z|1117078|vconn|DBG|unix#0: received: >>> >>> NXT_RESUME >>> >>> >> (OF1.5) (xid=0x0): cookie=0x5bc96bd6 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x28b,reg11=0x9,reg12=0xa,reg13=0x8b,reg14=0x2fb,metadata=0x2,in_port=135 >>> (via action) data_len=348 (unbuffered) >>> >>> >> 2025-02-18T21:21:31.672Z|1118459|vconn|DBG|unix#0: received: >>> >>> NXT_RESUME >>> >>> >> (OF1.5) (xid=0x0): cookie=0x5bc96bd6 total_len=348 >>> >>> >> >>> >>> >>> reg0=0x28b,reg11=0x9,reg12=0xa,reg13=0x93,reg14=0xdd,metadata=0x2,in_port=143 >>> (via action) data_len=348 (unbuffered) >>> >>> >> >>> >>> >> The interesting logs is also at ovn-controller logs >>> >>> >> 2025-02-18T21:21:10.152Z|1059900|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:a6:57:54 13.37.13.68 >>> >>> >> 2025-02-18T21:21:10.560Z|1059909|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:43:d4:9a 13.38.13.15 >>> >>> >> 2025-02-18T21:21:14.178Z|1059983|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:cb:77:8d 13.37.13.72 >>> >>> >> 2025-02-18T21:21:16.785Z|1060037|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:cc:d3:66 13.38.13.62 >>> >>> >> 2025-02-18T21:21:19.661Z|1060097|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:7e:1b:a8 13.37.13.173 >>> >>> >> 2025-02-18T21:21:22.582Z|1060157|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:cc:d3:66 13.38.13.62 >>> >>> >> 2025-02-18T21:21:25.903Z|1060225|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:f6:ae:8b 13.37.13.11 >>> >>> >> 2025-02-18T21:21:28.643Z|1060281|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:b0:98:e0 13.37.13.101 >>> >>> >> 2025-02-18T21:21:32.687Z|1060361|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:82:6e:c3 13.37.13.183 >>> >>> >> 2025-02-18T21:21:34.517Z|1060398|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:cb:08:c7 13.37.13.55 >>> >>> >> 2025-02-18T21:21:38.673Z|1060484|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:f9:5b:25 13.37.13.98 >>> >>> >> 2025-02-18T21:21:40.782Z|1060528|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:cc:d3:66 13.38.13.62 >>> >>> >> 2025-02-18T21:21:44.097Z|1060596|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:95:0c:5e 13.37.13.197 >>> >>> >> 2025-02-18T21:21:46.484Z|1060646|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:cc:d3:66 13.38.13.62 >>> >>> >> 2025-02-18T21:21:50.203Z|1060722|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:92:f8:ba 13.37.13.124 >>> >>> >> 2025-02-18T21:21:52.764Z|1060776|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:ce:b8:1e 13.37.13.126 >>> >>> >> 2025-02-18T21:21:55.503Z|1060832|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:8d:88:f6 13.38.13.63 >>> >>> >> 2025-02-18T21:21:59.520Z|1060914|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:8f:20:53 13.37.13.63 >>> >>> >> 2025-02-18T21:22:01.620Z|1060958|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:8a:7a:47 13.37.13.150 >>> >>> >> 2025-02-18T21:22:04.418Z|1061015|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:70:7e:5f 13.37.13.216 >>> >>> >> 2025-02-18T21:22:07.477Z|1061079|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:2a:7a:ad 13.37.13.50 >>> >>> >> 2025-02-18T21:22:11.719Z|1061165|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:cc:d3:66 13.38.13.62 >>> >>> >> 2025-02-18T21:22:13.798Z|1061208|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:a6:57:54 13.37.13.68 >>> >>> >> 2025-02-18T21:22:16.612Z|1061266|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:81:5f:99 13.37.13.14 >>> >>> >> 2025-02-18T21:22:19.830Z|1061334|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:cc:d3:66 13.38.13.62 >>> >>> >> 2025-02-18T21:22:22.460Z|1061388|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:cc:d3:66 13.38.13.62 >>> >>> >> 2025-02-18T21:22:26.001Z|1061460|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:cc:d3:66 13.38.13.62 >>> >>> >> 2025-02-18T21:22:28.383Z|1061506|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:cc:d3:66 13.38.13.62 >>> >>> >> 2025-02-18T21:22:31.594Z|1061572|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:58:35:35 13.37.13.64 >>> >>> >> 2025-02-18T21:22:35.249Z|1061648|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:cc:d3:66 13.38.13.62 >>> >>> >> 2025-02-18T21:22:37.578Z|1061695|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:82:6e:c3 13.37.13.183 >>> >>> >> 2025-02-18T21:22:40.514Z|1061755|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:cc:d3:66 13.38.13.62 >>> >>> >> 2025-02-18T21:22:43.373Z|1061815|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:58:72:c4 13.37.13.65 >>> >>> >> 2025-02-18T21:22:46.557Z|1061881|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:02:5e:ac 13.37.13.111 >>> >>> >> 2025-02-18T21:22:50.020Z|1061951|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:cc:d3:66 13.38.13.62 >>> >>> >> 2025-02-18T21:22:53.511Z|1062023|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:5f:fd:6f 13.37.13.15 >>> >>> >> 2025-02-18T21:22:55.717Z|1062069|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:cc:d3:66 13.38.13.62 >>> >>> >> 2025-02-18T21:22:58.812Z|1062133|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:8d:88:f6 13.38.13.63 >>> >>> >> 2025-02-18T21:23:01.450Z|1062189|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:9a:da:47 13.37.13.225 >>> >>> >> 2025-02-18T21:23:05.545Z|1062273|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:8a:7a:47 13.37.13.150 >>> >>> >> 2025-02-18T21:23:08.512Z|1062332|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:70:7e:5f 13.37.13.216 >>> >>> >> 2025-02-18T21:23:10.980Z|1062384|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:02:ff:8a 13.37.13.233 >>> >>> >> 2025-02-18T21:23:14.405Z|1062454|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:cc:d3:66 13.38.13.62 >>> >>> >> 2025-02-18T21:23:17.172Z|1062511|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:2e:73:8f 13.37.13.186 >>> >>> >> 2025-02-18T21:23:19.435Z|1062559|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:39:8d:2f 13.37.13.153 >>> >>> >> 2025-02-18T21:23:23.049Z|1062633|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:cc:d3:66 13.38.13.62 >>> >>> >> 2025-02-18T21:23:25.653Z|1062687|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:d4:36:c6 13.37.13.60 >>> >>> >> 2025-02-18T21:23:28.768Z|1062749|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:f1:64:3a 13.37.13.125 >>> >>> >> 2025-02-18T21:23:33.340Z|1062843|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:56:c4:04 13.37.13.219 >>> >>> >> 2025-02-18T21:23:34.400Z|1062865|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:6a:7b:a4 13.37.13.121 >>> >>> >> 2025-02-18T21:23:39.485Z|1062968|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:cc:d3:66 13.38.13.62 >>> >>> >> 2025-02-18T21:23:41.393Z|1063007|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:82:6e:c3 13.37.13.183 >>> >>> >> 2025-02-18T21:23:44.410Z|1063069|pinctrl(ovn_pinctrl0)|INFO| >>> DHCPOFFER >>> >>> >> fa:16:3e:52:ef:94 13.37.13.182 >>> >>> >> >>> >>> >> IPs are sanitized, anyway 13.37.13 belongs to subnet #1 and >>> >>> 13.38.13 to >>> >>> >> subnet #2. Observing large regression across all infra, like >>> >>> almost 50% >>> >>> >> overall network bandwith loss over time. >>> >>> >> >>> >>> >> Any thoughts guys? >>> >>> >> >>> >>> > Do you see any warning/error logs in ovs-vswitchd.log? >>> >>> > >>> >>> >> Context: we run a lot of small instances with 1 x fixed IP >>> with little >>> >>> >> amount of external traffic, about 300 per hypervisor, >>> sometimes more, >>> >>> >> sometimes less anyway didn't see anything similar with plain >>> ovs and >>> >>> >> neutron-openvswitch-agent (openstack). >>> >>> >> UPD: Once i manually cleared dhcpv4_options column in all >>> ports seems >>> >>> >> like neutron-dhcp-agent (dnsmasq based) started to send >>> dhcpoffers and >>> >>> >> most of instances back to normal. However, didn't manage what >>> is the >>> >>> >> reason to ovn-controller not send DHCPOFFER to VIF. Seems like >>> a big >>> >>> >> point of regression starting from N number of instances or >>> >>> something else. >>> >>> >> >>> >>> > Just double checking, you're saying that with low numbers of >>> instances >>> >>> > you don't have this issue? >>> >>> > >>> >>> > I wonder if you're not potentially hitting the maximum resubmit >>> limit >>> >>> > (4096 resubmits) due to DHCPOFFERS being sent as broadcast. There >>> >>> > should be warning logs indicating that in ovs-vswitchd. >>> >>> > >>> >>> > It might also help if you could share the output of: >>> >>> > >>> >>> > ovs-appctl coverage/show >>> >>> > >>> >>> >>> 10.02.2025 17:36, Numan Siddique: >>> >>> >>>> >>> >>> >>>> On Sat, Feb 8, 2025 at 1:27 PM support via discuss <ovs- >>> >>> >>>> disc...@openvswitch.org> wrote: >>> >>> >>>> >>> >>> >>>> I'm observing for a long time next bug: ovn-controller >>> doesn't >>> >>> >>>> send DHCPOFFER but it logs that it does. However this VIF >>> >>> doesn't >>> >>> >>>> get it at all. >>> >>> >>>> But once ovn-controller restarted it successfully seems >>> like it >>> >>> >>>> actually send DHCPOFFER and immediately receives DHCPACK >>> >>> from VIF. >>> >>> >>>> >>> >>> >>>> >>> >>> >>>> When you see this issue, can you enable vconn and rcon debug >>> >>> logs in >>> >>> >>>> both ovn-controller and ovs-vswitchd ? >>> >>> >>>> >>> >>> >>>> ovn-appctl vlog/set rconn:dbg >>> >>> >>>> ovn-appctl vlog/set vconn:dbg >>> >>> >>>> >>> >>> >>>> ovs-appctl vlog/set rconn:dbg >>> >>> >>>> ovs-appctl vlog/set rconn:dbg >>> >>> >>>> >>> >>> >>>> If ovn-controller replies with a DHCPOFFER, you should see the >>> >>> >>>> NXT_RESUME related logs in both ovn-controller and ovs-vswitchd. >>> >>> >>>> >>> >>> >>>> This can help you narrow down the issue. >>> >>> >>>> >>> >>> >>>> Thanks >>> >>> >>>> Numan >>> >>> >>>> >>> >>> >>>> |2025-02-08T14:27:57.462Z|00053|pinctrl(ovn_pinctrl0)|INFO| >>> >>> DHCPOFFER >>> >>> >>>> fa:16:3e:80:67:62 13.37.0.5 >>> >>> >>>> 2025-02-08T14:28:01.746Z|00054|pinctrl(ovn_pinctrl0)|INFO| >>> >>> DHCPOFFER >>> >>> >>>> fa:16:3e:80:67:62 13.37.0.5 >>> >>> >>>> 2025-02-08T14:28:05.582Z|00055|pinctrl(ovn_pinctrl0)|INFO| >>> >>> DHCPOFFER >>> >>> >>>> fa:16:3e:80:67:62 13.37.0.5 >>> >>> >>>> 2025-02-08T14:28:13.053Z|00056|pinctrl(ovn_pinctrl0)|INFO| >>> >>> DHCPOFFER >>> >>> >>>> fa:16:3e:80:67:62 13.37.0.5 >>> >>> >>>> 2025-02-08T14:28:21.972Z|00057|pinctrl(ovn_pinctrl0)|INFO| >>> >>> DHCPOFFER >>> >>> >>>> fa:16:3e:84:8e:7f 13.37.0.37 >>> >>> >>>> 2025-02-08T14:28:25.109Z|00058|pinctrl(ovn_pinctrl0)|INFO| >>> >>> DHCPOFFER >>> >>> >>>> fa:16:3e:84:8e:7f 13.37.0.37 >>> >>> >>>> 2025-02-08T14:28:29.362Z|00059|pinctrl(ovn_pinctrl0)|INFO| >>> >>> DHCPOFFER >>> >>> >>>> fa:16:3e:84:8e:7f 13.37.0.37 >>> >>> >>>> 2025-02-08T14:28:30.003Z|00060|pinctrl(ovn_pinctrl0)|INFO| >>> >>> DHCPOFFER >>> >>> >>>> fa:16:3e:80:67:62 13.37.0.5 >>> >>> >>>> 2025-02-08T14:28:37.820Z|00061|pinctrl(ovn_pinctrl0)|INFO| >>> >>> DHCPOFFER >>> >>> >>>> fa:16:3e:84:8e:7f 13.37.0.37 >>> >>> >>>> 2025-02-08T14:28:38.619Z|00002| >>> fatal_signal(ovn_statctrl2)|WARN| >>> >>> >>>> terminating >>> >>> >>>> with signal 15 (Terminated) # ovn_controller is restarted by >>> >>> >>>> docker restart ovn_controller >>> >>> >>>> 2025-02-08T14:28:50.639Z|00001|vlog|INFO|opened log file >>> >>> >>>> /var/log/kolla/openvswitch/ovn-controller.log >>> >>> >>>> 2025-02-08T14:28:50.640Z|00002|vif_plug_representor| >>> INFO|unable >>> >>> >>>> to set udev receive buffer size: Operation not permitted >>> >>> >>>> 2025-02-08T14:28:50.641Z|00003|reconnect|INFO| >>> tcp:127.0.0.1:6640 >>> >>> >>>> <http://127.0.0.1:6640>: connecting... >>> >>> >>>> 2025-02-08T14:28:50.641Z|00004|reconnect|INFO| >>> tcp:127.0.0.1:6640 >>> >>> >>>> <http://127.0.0.1:6640>: connected >>> >>> >>>> 2025-02-08T14:28:50.738Z|00005|main|INFO|OVN internal >>> version is >>> >>> >>>> : [24.09.0-20.37.0-73.6] >>> >>> >>>> 2025-02-08T14:28:50.738Z|00006|main|INFO|OVS IDL >>> reconnected, >>> >>> >>>> force recompute. >>> >>> >>>> 2025-02-08T14:28:50.738Z|00007|reconnect|INFO| >>> tcp:10.10.9.5:6642 >>> >>> >>>> <http://10.10.9.5:6642>: connecting... >>> >>> >>>> 2025-02-08T14:28:50.738Z|00008|main|INFO|OVNSB IDL >>> reconnected, >>> >>> >>>> force recompute. >>> >>> >>>> 2025-02-08T14:28:50.739Z|00009|ovn_util|INFO|statctrl: >>> >>> connecting >>> >>> >>>> to switch: "unix:/var/run/openvswitch/br-int.mgmt" >>> >>> >>>> 2025-02-08T14:28:50.739Z|00010|rconn|INFO|unix:/var/run/ >>> >>> >>>> openvswitch/br-int.mgmt: >>> >>> >>>> connecting... >>> >>> >>>> 2025-02-08T14:28:50.739Z|00011|ovn_util|INFO|pinctrl: >>> connecting >>> >>> >>>> to switch: "unix:/var/run/openvswitch/br-int.mgmt" >>> >>> >>>> 2025-02-08T14:28:50.739Z|00012|rconn|INFO|unix:/var/run/ >>> >>> >>>> openvswitch/br-int.mgmt: >>> >>> >>>> connecting... >>> >>> >>>> 2025-02-08T14:28:50.739Z|00013|reconnect|INFO| >>> tcp:10.10.9.5:6642 >>> >>> >>>> <http://10.10.9.5:6642>: connected >>> >>> >>>> 2025-02-08T14:28:50.739Z|00001|rconn(ovn_statctrl2)| >>> INFO|unix:/ >>> >>> >>>> var/run/openvswitch/br-int.mgmt: >>> >>> >>>> connected >>> >>> >>>> 2025-02-08T14:28:50.743Z|00001|rconn(ovn_pinctrl0)|INFO| >>> unix:/ >>> >>> >>>> var/run/openvswitch/br-int.mgmt: >>> >>> >>>> connected 2025-02-08T14:28:50.756Z|00014|ovn_util|INFO| >>> features: >>> >>> >>>> connecting to switch: "unix:/var/run/openvswitch/br- >>> int.mgmt" >>> >>> >>>> 2025-02-08T14:28:50.757Z|00015|rconn|INFO|unix:/var/run/ >>> >>> >>>> openvswitch/br-int.mgmt: >>> >>> >>>> connecting... 2025-02-08T14:28:50.757Z|00016|features| >>> INFO|OVS >>> >>> >>>> Feature: ct_zero_snat, state: supported >>> >>> >>>> 2025-02-08T14:28:50.757Z|00017|features|INFO|OVS Feature: >>> >>> >>>> ct_flush, state: supported >>> >>> >>>> 2025-02-08T14:28:50.757Z|00018|features|INFO|OVS Feature: >>> >>> >>>> dp_hash_l4_sym_support, state: supported >>> >>> >>>> 2025-02-08T14:28:50.757Z|00019|main|INFO|OVS feature set >>> >>> changed, >>> >>> >>>> force recompute. >>> >>> >>>> 2025-02-08T14:28:50.757Z|00020|ovn_util|INFO|ofctrl: >>> connecting >>> >>> >>>> to switch: "unix:/var/run/openvswitch/br-int.mgmt" >>> >>> >>>> 2025-02-08T14:28:50.757Z|00021|rconn|INFO|unix:/var/run/ >>> >>> >>>> openvswitch/br-int.mgmt: >>> >>> >>>> connecting... >>> >>> >>>> 2025-02-08T14:28:50.758Z|00022|rconn|INFO|unix:/var/run/ >>> >>> >>>> openvswitch/br-int.mgmt: >>> >>> >>>> connected 2025-02-08T14:28:50.758Z|00023|main|INFO|OVS >>> OpenFlow >>> >>> >>>> connection reconnected,force recompute. >>> >>> >>>> 2025-02-08T14:28:50.759Z|00024|rconn|INFO|unix:/var/run/ >>> >>> >>>> openvswitch/br-int.mgmt: >>> >>> >>>> connected 2025-02-08T14:28:50.760Z|00025|features|INFO|OVS >>> >>> >>>> Feature: meter_support, state: supported >>> >>> >>>> 2025-02-08T14:28:50.760Z|00026|features|INFO|OVS Feature: >>> >>> >>>> group_support, state: supported >>> >>> >>>> 2025-02-08T14:28:50.760Z|00027|features|INFO|OVS Feature: >>> >>> >>>> sample_action_with_registers, state: supported >>> >>> >>>> 2025-02-08T14:28:50.760Z|00028|main|INFO|OVS feature set >>> >>> changed, >>> >>> >>>> force recompute. >>> >>> >>>> 2025-02-08T14:28:51.483Z|00029|binding|INFO|Releasing lport >>> >>> >>>> 978a923d-1076-484b-a1d5-ff5827ae69ec from this chassis >>> >>> >>>> (sb_readonly=0) >>> >>> >>>> 2025-02-08T14:28:53.658Z|00030|poll_loop|INFO|wakeup due to >>> >>> >>>> [POLLIN] on fd 30 (10.10.9.3:60902 >>> >>> >>>> <http://10.10.9.3:60902><->10.10.9.5:6642 >>> >>> >>>> <http://10.10.9.5:6642>) at lib/stream-fd.c:157 (78% CPU >>> usage) >>> >>> >>>> 2025-02-08T14:28:54.726Z|00002|pinctrl(ovn_pinctrl0)|INFO| >>> >>> DHCPOFFER >>> >>> >>>> fa:16:3e:84:8e:7f 13.37.0.37 >>> >>> >>>> 2025-02-08T14:28:59.134Z|00003|pinctrl(ovn_pinctrl0)|INFO| >>> >>> DHCPACK >>> >>> >>>> fa:16:3e:84:8e:7f 13.37.0.37 >>> >>> >>>> 2025-02-08T14:29:01.499Z|00004|pinctrl(ovn_pinctrl0)|INFO| >>> >>> DHCPOFFER >>> >>> >>>> fa:16:3e:80:67:62 13.37.0.5 >>> >>> >>>> 2025-02-08T14:29:03.245Z|00005|pinctrl(ovn_pinctrl0)|INFO| >>> >>> DHCPACK >>> >>> >>>> fa:16:3e:84:8e:7f 13.37.0.37 >>> >>> >>>> 2025-02-08T14:29:03.455Z|00031|memory|INFO|98744 kB peak >>> >>> resident >>> >>> >>>> set size after 12.8 seconds >>> >>> >>>> 2025-02-08T14:29:03.455Z|00032|memory|INFO|idl-cells- >>> >>> >>>> OVN_Southbound:168023 >>> >>> >>>> idl-cells-Open_vSwitch:36874 lflow-cache-entries-cache- >>> expr:6494 >>> >>> >>>> lflow-cache-entries-cache-matches:1413 lflow-cache-size- >>> KB:20726 >>> >>> >>>> local_datapath_usage-KB:1 ofctrl_desired_flow_usage-KB:6342 >>> >>> >>>> ofctrl_installed_flow_usage-KB:4695 >>> ofctrl_sb_flow_ref_usage- >>> >>> >>>> KB:2355 | >>> >>> >>>> >>> >>> >>>> Any suggestions? >>> >>> >>>> >>> >>> >>>> Environment: >>> >>> >>>> OpenStack 2024.2 >>> >>> >>>> Kernel 6.8.0-50-generic #51 >>> >>> >>>> <https://github.com/ovn-org/ovn/issues/51>~22.04.1- >>> Ubuntu SMP >>> >>> >>>> PREEMPT_DYNAMIC >>> >>> >>>> >>> >>> >>>> |ovs-vsctl (Open vSwitch) 3.4.0 DB Schema 8.7.0 | >>> >>> >>>> >>> >>> >>>> |ovs-ofctl (Open vSwitch) 3.4.0 OpenFlow versions 0x1:0x6 | >>> >>> >>>> >>> >>> >>>> |# ovn-nbctl --version ovn-nbctl 24.09.0 Open vSwitch >>> Library >>> >>> >>>> 3.4.0 DB Schema 7.6.0 | >>> >>> >>>> >>> >>> >>>> |# ovn-sbctl --version ovn-sbctl 24.09.0 Open vSwitch >>> Library >>> >>> >>>> 3.4.0 DB Schema 20.37.0| >>> >>> >>>> >>> >>> > Thanks, >>> >>> > Dumitru >>> >>> > Thanks, Dumitru _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
