Thank you Mathieu for the insights!
To add details to what happened: * Upgrade was never made a #1 priority. It was a one man show for far too long. (myself)
I suppose that confirms that upgrades is very nice to have in production deployments, eventually, maybe... (please read below to continue)
* I also happen to manage and work on other priorities. * Lot of work made to prepare for multiple versions support in our deployment tools. (we use Puppet) * Lot of work in the packaging area to speedup packaging. (we are still using deb packages but with virtualenv to stay Puppet compatible) * We need to forward-port private patches which upstream won't accept and/or are private business logic.
... yet long time maintaining and landing fixes is the ops' *reality* and pain #1. And upgrades are only pain #2. LTS can not directly help with #2, but only indirectly, if the vendors' downstream teams could better cooperate with #1 and have more time and resources to dedicate for #2, upgrades stories for shipped products and distros.
Let's please to not lower the real value of LTS branches and not substitute #1 with #2. This topic is not about bureaucracy and policies, it is about how could the community help vendors to cooperate over maintaining of commodity things, with as less bureaucracy as possible, to ease the operators pains in the end.
* Our developer teams didn't have enough free cycles to work right away on the upgrade. (this means delays) * We need to test compatibility with 3rd party systems which takes some time. (and make them compatible)
This confirms perhaps why it is vital to only run 3rd party CI jobs for LTS branches?
* We need to update systems ever which we don't have full control. This means serious delays when it comes to deployment. * We need to test features/stability during some time in our dev environment. * We need to test features/stability during some time in our staging/pre-prod environment. * We need to announce and inform our users at least 2 weeks in advance before performing an upgrade. * We choose to upgrade one service at a time (in all regions) to avoid a huge big bang upgrade. (this means more maintenance windows to plan and you can't stack them too much) * We need to swiftly respond to bug discovered by our users. This means change of priorities and delay in other service upgrades. * We will soon need to upgrade operating systems to support latest OpenStack versions. (this means we have to stop OpenStack upgrades until all nodes are upgraded)
It seems that the answer to the question sounded, "Why upgrades are so painful and take so much time for ops?" is "as upgrades are not the priority. Long Time Support and maintenance are".
-- Best regards, Bogdan Dobrelya, Irc #bogdando __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
