On 8/4/23 14:14, Vsevolod Vayner wrote: > Hello Ilya -- > > Thank you for your message and clarification. > > Could you please clarify which method/function called ovs-vswitchd and > ovs-dpctlto open the kernel datapath and set flags?
Both are using the same lib/dpif-netlink.c:dpif_netlink_open(). In your particular case, the OVS_DP_F_DISPATCH_UPCALL_PER_CPU breaks upcall delivery, because old OVS 2.12 doesn't support it. > > Thank you. > > > > GCORE <https://gcorelabs.com/> > *Vsevolod Vayner* > > Product Director > > vsevolod.vay...@gcore.com <mailto:vsevolod.vay...@gcore.com> > gcore.com <https://gcore.com/> > <https://www.facebook.com/gcorelabscom/> > <https://www.linkedin.com/company/g-core> > <https://www.youtube.com/channel/UCFhtQE7HB097a1_Nfv-Ma_Q> > <https://www.weibo.com/p/1005056986184457> > > > This message is intended solely for the addressee and may contain > confidential information. If you have received this message in error, please > send it back to us, and immediately and permanently delete it. Do not use, > copy or disclose the information contained in this message or in any > attachment. > > *From: *Ilya Maximets <i.maxim...@ovn.org> > *Date: *Monday, 31 July 2023 at 20:27 > *To: *Vsevolod Vayner <vsevolod.vay...@gcore.com>, b...@openvswitch.org > <b...@openvswitch.org> > *Cc: *[Partner] Fedor Tarasenko <prn.fedor.tarase...@gcore.lu>, [Partner] > Ivan Kramarev <prn.ivan.krama...@gcore.lu>, Dmitri Polyakov > <dmitri.polya...@gcore.com>, [Partner] Ivan Suchkov <ivan.such...@gcore.lu>, > Aleksandr Fedotov <aleksandr.fedo...@gcore.com>, i.maxim...@ovn.org > <i.maxim...@ovn.org> > *Subject: *Re: [ovs-discuss] OVS Stopped forwarding > > On 7/28/23 18:14, Vsevolod Vayner via discuss wrote: >> 1. What you did that made the problem appear. >> >> In our env we use openvswitch exporter to collect dump-flows and data from >> production ovs. >> It runs only ovs-dpctl and ovs-ofctl to collect, parse and convert data to >> prometheus format. >> We run vswitchd, db & exporter in separate containers. >> >> In our production env we have: >> >> Open vSwitch: >> >> ovs-vswitchd (Open vSwitch) 2.12.0 >> DPDK 18.11.2 >> ovs-vsctl (Open vSwitch) 2.12.0 >> DB Schema 8.0.0 >> >> and we also have an installed OpenvSwitch on our exporter to collect >> ovs-ofctl and ovs-dpctl data: >> >> ovs-vswitchd (Open vSwitch) 3.1.3 >> DPDK 22.11.1 >> ovs-vsctl (Open vSwitch) 3.1.3 >> DB Schema 8.3.1 >> >> 2. What you expected to happen. >> >> We expected the exporter to collect statistics using the ovs-ofctl utility >> with the dump-flows [br-int, br-tun, br-ex] and ovs-dpctl show to collect >> lookups. >> >> 3. What actually happened. >> >> openvswitch stopped to forward traffic. I see only ingress traffic on ports >> of ovs, no traffic outs. >> >> 4. The kernel version on which Open vSwitch is running (from /proc/version) >> and the distribution and version number of your OS (e.g. "Centos 5.0"). >> >> Linux version 4.18.0-408.el8.x86_64 (mockbu...@kbuilder.bsys.centos.org >> <mailto:mockbu...@kbuilder.bsys.centos.org >> <mailto:mockbu...@kbuilder.bsys.centos.org>>) (gcc version 8.5.0 20210514 >> (Red Hat 8.5.0-14) (GCC)) #1 SMP Mon Jul 18 17:42:52 UTC 2022 >> >> >> In some cases, only a host reboot helps. >> >> If this is not well-known bug, I'll try to find resource in production to >> reproduce and provide more debug information. > > Hi, Vsevolod. > > The problem is that you're using drastically different versions > of ovs-vswitchd and ovs-dpctl. Every time you run ovs-dpctl, > it will open the kernel datapath and set flags to it in the process. > In your particular case, these flags change the way upcalls are > delivered to ovs-vswitchd, so no traffic can go to userspace > afterwards and no new flows will be installed to a datapath as > a result, because ovs-vswitchd is too old to understand a new > datapath configuration. The issue should be resolved by re-starting > ovs-vswitchd. > > In general, use of ovs-dpctl while ovs-vswitchd is running is not > safe. Use ovs-appctl dpctl/* commands to get the same information > via ovs-vswitchd instead. Or at least make sure that the version > of ovs-dpctl is exactly the same as version of ovs-vswitchd. > > Best regards, Ilya Maximets. > _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
