On 5/17/24 14:20, Шагов Георгий wrote: > At the first, Numan, Dumitru, Han, I really appreciate your replies, will do > my best to describe the case in details >
No problem! > NS> Does your deployment create/delete logical switches/routers frequently > > It is a separate service, i.e.: Open Stack Nova that is creating a Port every > time when user creates new VM. It does not happen too frequently, yet, if it > happens, we have a huge transaction flow at NorthD then this transaction > going from SBDB to NBDB could be delayed, i.e. this means not every > transaction takes 52 secs in delay, I think it is 25%, that is considerable, > of cause. > > HZ> Yes, it is better to understand why in this deployment the recompute took > so long (52s). Is it simply too large scale, or is it because of some > uncommon configuration that we don't handle efficiently and can be optimized > to improve recompute performance > > A Very good question. > Yes, there is a peculiarity with our custom installation, i.e.: we have > implemented L3 connectivity to External Provider Network, this customization > was done at ovn-controller level. That was a biz demand and it is working > fine, but as a drawback we have got a problem with huge Logical_DP_Groups I > described earlier: Link1 > > DC> But the best way to tell is to have a way to reproduce this, e.g., NB/SB > databases and the NB/SB jsonrpc update that caused the recompute > > Yes, we are working on this and at this moment we have an ovn-heater > installation, that is using Vanilla (non-custom) OVN installation, simulating > the case we have at prod Kind of related, if you have time it would be awesome if you could share your ovn-heater configuration file. Maybe I can use it internally in our scale test lab and report results upstream. > Could you please tell me the bast way in submitting the dumps for both > (NB|SB)DBs > I can do an Issue at ovn github: https://github.com/ovn-org/ovn/issues > That works for me but please link the issue here. Most development happens on-list and github issues don't get as much attention. > Thank you > Yours truly, George > Best regards, Dumitru > > From: Numan Siddique <num...@ovn.org> > Date: Wednesday, 8 May 2024, 19:01 > To: Шагов Георгий <gmsha...@cloud.ru>, Dumitru Ceara <dce...@redhat.com>, Han > Zhou <hz...@ovn.org> > Cc: "ovs-discuss@openvswitch.org" <ovs-discuss@openvswitch.org> > Subject: Re: [ovs-discuss] NorthD inc-engine Handlers; OVN 24.03 > > ВНИМАНИЕ! ВНЕШНИЙ ОТПРАВИТЕЛЬ > Если отправитель почты неизвестен, не переходите по ссылкам, не сообщайте > пароль, > не запускайте вложения и сообщите коллегам из ЦКЗ на > secur...@cloud.ru<mailto:secur...@cloud.ru> > > > On Wed, May 8, 2024 at 8:42 AM Шагов Георгий via discuss > <ovs-discuss@openvswitch.org<mailto:ovs-discuss@openvswitch.org>> wrote: > Hello everyone > > In some aspect it might be considered as a continuation of this thread: > (link1), yet it is different > After we have upgrade from OVN 22.03 to OVN 24.03, we have indeed found > increase in performance in 3-4 times > And yet still we do observe high CPU load for NorthD process; taking deeper > into the logs we have found: > > > Thanks for reporting this issue. > > > 2024-05-07T08:36:46.505Z|18503|poll_loop|INFO|wakeup due to [POLLIN] on fd 15 > (10.34.22.66:60716<http://10.34.22.66:60716><->10.34.22.66:6642<http://10.34.22.66:6642>) > at lib/stream-fd.c:157 (94% CPU usage) > 2024-05-07T08:37:38.857Z|18504|inc_proc_eng|INFO|node: northd, recompute > (missing handler for input SB_datapath_binding) took 52313ms > 2024-05-07T08:37:48.335Z|18505|inc_proc_eng|INFO|node: lflow, recompute > (failed handler for input northd) took 7759ms > 2024-05-07T08:37:48.718Z|18506|timeval|WARN|Unreasonably long 62213ms poll > interval (56201ms user, 2900ms system) > > As you can see there is a significant delay in 52 secs > Correct me please, if I am in the wrong, but IMU: ‘missing handler for’ – > practically means absence of the inc-engine handler from some node (in this > sample: SB_datapath_binding) > > That's correct. > > Before plunging into Development it would be great to clarify/adjust with > Community’s position > > * Why there is not handler for this node? > > Our approach has been to add a handler for any input change only if it is > frequent or if it can be easily handled. > We also have skipped adding handlers if it increases the code complexity. > Having said that I think we are open > to adding more handlers if it makes sense or if it results in scale > improvements. > > Right now we fall back to a full recompute of northd engine for any changes > to a logical switch or logical router. > Does your deployment create/delete logical switches/routers frequently ? Is > it possible to enable ovn debug logs > and share them ? I'm curious to know what are the changes to SB datapath > binding. > > Feel free to share your OVN NB and SB DBs if you're ok with it. I can deploy > those DBs and see why recompute is so expensive. > > > > * Any particular reason for this or just the peculiarity of our > installation highlighted this issue? > > My guess is that your installation is frequently creating , deleting or > modifying logical switches or routers. > > > * > * Do you think there is a reason in implementing that handler? > (SB_datapath_binding) > > I'm fine adding a handler if it helps in the scale. In our use cases, we > don't frequently create/delete the logical switches and routers > and hence it is ok to fall back to full recomputes for such changes. > > > * > > Any ideas are highly appreciated. > > You're welcome to work on it and submit patches to add a handler for > SB_datapath_binding. > > @Dumitru Ceara<mailto:dce...@redhat.com> @Han Zhou<mailto:hz...@ovn.org> if > you've any reservations on adding more handlers please do comment here. > > Thanks > Numan > > > > > Yours truly, George > > > > * Link1: > https://mail.openvswitch.org/pipermail/ovs-discuss/2024-March/053035.html > УВЕДОМЛЕНИЕ О КОНФИДЕНЦИАЛЬНОСТИ: Это электронное сообщение и любые > документы, приложенные к нему, содержат конфиденциальную информацию. > Настоящим уведомляем Вас о том, что если это сообщение не предназначено Вам, > использование, копирование, распространение информации, содержащейся в > настоящем сообщении, а также осуществление любых действий на основе этой > информации, строго запрещено. Если Вы получили это сообщение по ошибке, > пожалуйста, сообщите об этом отправителю по электронной почте и удалите это > сообщение. > CONFIDENTIALITY NOTICE: This email and any files attached to it are > confidential. If you are not the intended recipient you are notified that > using, copying, distributing or taking any action in reliance on the contents > of this information is strictly prohibited. If you have received this email > in error please notify the sender and delete this email. > > _______________________________________________ > discuss mailing list > disc...@openvswitch.org<mailto:disc...@openvswitch.org> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > УВЕДОМЛЕНИЕ О КОНФИДЕНЦИАЛЬНОСТИ: Это электронное сообщение и любые > документы, приложенные к нему, содержат конфиденциальную информацию. > Настоящим уведомляем Вас о том, что если это сообщение не предназначено Вам, > использование, копирование, распространение информации, содержащейся в > настоящем сообщении, а также осуществление любых действий на основе этой > информации, строго запрещено. Если Вы получили это сообщение по ошибке, > пожалуйста, сообщите об этом отправителю по электронной почте и удалите это > сообщение. > CONFIDENTIALITY NOTICE: This email and any files attached to it are > confidential. If you are not the intended recipient you are notified that > using, copying, distributing or taking any action in reliance on the contents > of this information is strictly prohibited. If you have received this email > in error please notify the sender and delete this email. _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
