Ok so 1. Customers depend on the internal DNS reccords, so this is needed for production operations 2. I can take a look at the updates - would using conditional monitoring work here? We have ovn-monitor-all=true , would this help at all ? 3 & 4 . Is that something the community can help with? Is that a viable long term fix we could maybe get a patch for ?
Gav On Wed, 8 May 2024 at 14:30, Numan Siddique <num...@ovn.org> wrote: > > On Wed, May 8, 2024 at 3:20 PM Gavin McKee via discuss > <ovs-discuss@openvswitch.org> wrote: > > > > Hi, > > > > Can someone help me understand why this issue occurs > > > > > > ovn-controller 23.09.1 > > Open vSwitch Library 3.2.2 > > > > We have an issue with some machines intermittently unable to resolve > > DNS for external domains (example dig +noall +answer > > harmonic-openai-canada.openai.azure.com) > > > > In the OVN controller log I see the following > > > > 2024-05-08T12:12:35.596Z|30138|inc_proc_eng|INFO|node: > > logical_flow_output, recompute (missing handler for input SB_dns) took > > 7950ms > > 2024-05-08T14:50:29.747Z|30312|inc_proc_eng|INFO|node: > > logical_flow_output, recompute (missing handler for input SB_dns) took > > 8634ms > > 2024-05-08T14:50:46.673Z|30329|inc_proc_eng|INFO|node: > > logical_flow_output, recompute (missing handler for input SB_dns) took > > 8774ms > > 2024-05-08T14:54:40.781Z|30353|inc_proc_eng|INFO|node: > > logical_flow_output, recompute (missing handler for input SB_dns) took > > 8535ms > > 2024-05-08T14:58:43.381Z|30433|inc_proc_eng|INFO|node: > > logical_flow_output, recompute (missing handler for input SB_dns) took > > 8541ms > > 2024-05-08T14:58:56.802Z|30488|inc_proc_eng|INFO|node: > > logical_flow_output, recompute (missing handler for input SB_dns) took > > 8820ms > > 2024-05-08T15:02:50.704Z|30512|inc_proc_eng|INFO|node: > > logical_flow_output, recompute (missing handler for input SB_dns) took > > 8739ms > > 2024-05-08T15:03:05.206Z|30529|inc_proc_eng|INFO|node: > > logical_flow_output, recompute (missing handler for input SB_dns) took > > 8686ms > > 2024-05-08T15:08:39.441Z|30569|inc_proc_eng|INFO|node: > > logical_flow_output, recompute (missing handler for input SB_dns) took > > 9167ms > > 2024-05-08T15:09:09.152Z|30603|inc_proc_eng|INFO|node: > > logical_flow_output, recompute (missing handler for input SB_dns) took > > 8985ms > > 2024-05-08T15:12:14.361Z|30632|inc_proc_eng|INFO|node: > > logical_flow_output, recompute (missing handler for input SB_dns) took > > 8569ms > > 2024-05-08T15:13:52.535Z|30705|inc_proc_eng|INFO|node: > > logical_flow_output, recompute (missing handler for input SB_dns) took > > 8764ms > > 2024-05-08T15:14:53.989Z|30732|inc_proc_eng|INFO|node: > > logical_flow_output, recompute (missing handler for input SB_dns) took > > 8802ms > > 2024-05-08T15:16:30.911Z|30757|inc_proc_eng|INFO|node: > > logical_flow_output, recompute (missing handler for input SB_dns) took > > 8776ms > > 2024-05-08T15:17:09.371Z|30784|inc_proc_eng|INFO|node: > > logical_flow_output, recompute (missing handler for input SB_dns) took > > 9062ms > > > > Why would this happen and is there something I can do about it ? Are > > there more logs needed ? > > This indicates that your deployment is creating, updating or deleting a DNS > row > in the Northbound database and in turn ovn-northd is updating the SB DNS rows. > When ovn-controller receives the Southbound DNS updates, it falls back > to a full recompute > because we are not handling these changes incrementally. Since OVN > native DNS is configured > in your deployment, each DNS packet is sent to ovn-controller for lookup. > Even though a separate pinctrl thread handles packet-ins, dns > handling is blocked until > the main ovn-controller thread releases a mutex [1]. > > There are a few ways to resolve this > > 1. Disable native OVN DNS if you're not using this feature. To > disable, don't create any DNS records in the OVN Northbound db. > 2. Investigate why your deployment is updating the NB DBS table and > avoid it if its not required. > 3. Implement a handler for SB DNS so that ovn-controller do not fall > back to a full recompute > 4. Avoid locking on the mutex for DNS handling in pinctrl thread [1]. > > (3) or (4) requires code changes. > > Thanks > Numan > > [1] - https://github.com/ovn-org/ovn/blob/main/controller/pinctrl.c#L3807 > > > > > > Gav > > _______________________________________________ > > discuss mailing list > > disc...@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
