Here is my explanation: 1. According to ovn docs https://docs.ovn.org/en/latest/intro/install/ovn-upgrades.html "The only step required after upgrading the packages is to restart ovn-northd, which automatically restarts the databases and upgrades the database schema, as well. if you’re using a Linux distribution with systemd: $ sudo systemctl restart ovn-northd
which is not true in Ubuntu case ``` ExecStart=/usr/share/ovn/scripts/ovn-ctl start_northd --ovn-manage-ovsdb=no --no-monitor $OVN_CTL_OPTS ``` ovn-manage-ovsdb=no, so it will not restart the DBs. 2. DB restarts by ovn-ovsdb-server-nb.service and ovn-ovsdb-server-sb.service run_nb_ovsdb and run_sb_ovsdb options in ovn-ctl ``` ExecStart=/usr/share/ovn/scripts/ovn-ctl run_nb_ovsdb $OVN_CTL_OPTS ``` 3. This commit fixed Cluster OVN database upgrade https://github.com/ovn-org/ovn/commit/67e2f386cc838d0b0f9b4b5da7fe611e1113b70c in this bug https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1907081 This is the code that is responsible for the upgrade https://github.com/ovn-org/ovn/blob/5c2d311b8b7b4d5c3a619de72be6a433aa4c44db/utilities/ovn-ctl#L293 ``` if test X$detach = Xno && test $mode = cluster && test -z "$cluster_remote_addr" ; then ``` First two checks are true. Third check means upgrade_cluster (L313) will run on leader node, i.e --db-nb-cluster-remote-addr= and --db-sb-cluster-remote-addr= is not set. 4. During rolling upgrades of controller nodes, the leader will likely move. OVS_CTL_OPTS generated from this template https://github.com/openstack/openstack-ansible-os_neutron/blob/stable/2023.1/templates/ovn-northd-opts.j2 but only when new node joins the cluster https://github.com/openstack/openstack-ansible-os_neutron/blob/stable/2023.1/tasks/providers/ovn_cluster_setup.yml#L105 so after upgrade all 3 nodes will have --db-nb-cluster-remote-addr and --db-sb-cluster-remote-addr set to the some value that was leader node at that time. What we need is --db-nb-cluster-remote-addr= not set on leader node, so during ovn-central restart on leader node upgrade_cluster will run. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2059721 Title: ovn db upgrade To manage notifications about this bug go to: https://bugs.launchpad.net/openstack-ansible/+bug/2059721/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
