On Wed, May 8, 2024 at 3:20 PM Gavin McKee via discuss <ovs-discuss@openvswitch.org> wrote: > > Hi, > > Can someone help me understand why this issue occurs > > > ovn-controller 23.09.1 > Open vSwitch Library 3.2.2 > > We have an issue with some machines intermittently unable to resolve > DNS for external domains (example dig +noall +answer > harmonic-openai-canada.openai.azure.com) > > In the OVN controller log I see the following > > 2024-05-08T12:12:35.596Z|30138|inc_proc_eng|INFO|node: > logical_flow_output, recompute (missing handler for input SB_dns) took > 7950ms > 2024-05-08T14:50:29.747Z|30312|inc_proc_eng|INFO|node: > logical_flow_output, recompute (missing handler for input SB_dns) took > 8634ms > 2024-05-08T14:50:46.673Z|30329|inc_proc_eng|INFO|node: > logical_flow_output, recompute (missing handler for input SB_dns) took > 8774ms > 2024-05-08T14:54:40.781Z|30353|inc_proc_eng|INFO|node: > logical_flow_output, recompute (missing handler for input SB_dns) took > 8535ms > 2024-05-08T14:58:43.381Z|30433|inc_proc_eng|INFO|node: > logical_flow_output, recompute (missing handler for input SB_dns) took > 8541ms > 2024-05-08T14:58:56.802Z|30488|inc_proc_eng|INFO|node: > logical_flow_output, recompute (missing handler for input SB_dns) took > 8820ms > 2024-05-08T15:02:50.704Z|30512|inc_proc_eng|INFO|node: > logical_flow_output, recompute (missing handler for input SB_dns) took > 8739ms > 2024-05-08T15:03:05.206Z|30529|inc_proc_eng|INFO|node: > logical_flow_output, recompute (missing handler for input SB_dns) took > 8686ms > 2024-05-08T15:08:39.441Z|30569|inc_proc_eng|INFO|node: > logical_flow_output, recompute (missing handler for input SB_dns) took > 9167ms > 2024-05-08T15:09:09.152Z|30603|inc_proc_eng|INFO|node: > logical_flow_output, recompute (missing handler for input SB_dns) took > 8985ms > 2024-05-08T15:12:14.361Z|30632|inc_proc_eng|INFO|node: > logical_flow_output, recompute (missing handler for input SB_dns) took > 8569ms > 2024-05-08T15:13:52.535Z|30705|inc_proc_eng|INFO|node: > logical_flow_output, recompute (missing handler for input SB_dns) took > 8764ms > 2024-05-08T15:14:53.989Z|30732|inc_proc_eng|INFO|node: > logical_flow_output, recompute (missing handler for input SB_dns) took > 8802ms > 2024-05-08T15:16:30.911Z|30757|inc_proc_eng|INFO|node: > logical_flow_output, recompute (missing handler for input SB_dns) took > 8776ms > 2024-05-08T15:17:09.371Z|30784|inc_proc_eng|INFO|node: > logical_flow_output, recompute (missing handler for input SB_dns) took > 9062ms > > Why would this happen and is there something I can do about it ? Are > there more logs needed ?
This indicates that your deployment is creating, updating or deleting a DNS row in the Northbound database and in turn ovn-northd is updating the SB DNS rows. When ovn-controller receives the Southbound DNS updates, it falls back to a full recompute because we are not handling these changes incrementally. Since OVN native DNS is configured in your deployment, each DNS packet is sent to ovn-controller for lookup. Even though a separate pinctrl thread handles packet-ins, dns handling is blocked until the main ovn-controller thread releases a mutex [1]. There are a few ways to resolve this 1. Disable native OVN DNS if you're not using this feature. To disable, don't create any DNS records in the OVN Northbound db. 2. Investigate why your deployment is updating the NB DBS table and avoid it if its not required. 3. Implement a handler for SB DNS so that ovn-controller do not fall back to a full recompute 4. Avoid locking on the mutex for DNS handling in pinctrl thread [1]. (3) or (4) requires code changes. Thanks Numan [1] - https://github.com/ovn-org/ovn/blob/main/controller/pinctrl.c#L3807 > > Gav > _______________________________________________ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
