On Tue, Nov 14, 2017 at 6:44 PM, John Dickinson <m...@not.mn> wrote: > > > On 14 Nov 2017, at 15:18, Mathieu Gagné wrote: > >> On Tue, Nov 14, 2017 at 6:00 PM, Fox, Kevin M <kevin....@pnnl.gov> wrote: >>> The pressure for #2 comes from the inability to skip upgrades and the fact >>> that upgrades are hugely time consuming still. >>> >>> If you want to reduce the push for number #2 and help developers get their >>> wish of getting features into users hands sooner, the path to upgrade >>> really needs to be much less painful. >>> >> >> +1000 >> >> We are upgrading from Kilo to Mitaka. It took 1 year to plan and >> execute the upgrade. (and we skipped a version) >> Scheduling all the relevant internal teams is a monumental task >> because we don't have dedicated teams for those projects and they have >> other priorities. >> Upgrading affects a LOT of our systems, some we don't fully have >> control over. And it can takes months to get new deployment on those >> systems. (and after, we have to test compatibility, of course) >> >> So I guess you can understand my frustration when I'm told to upgrade >> more often and that skipping versions is discouraged/unsupported. >> At the current pace, I'm just falling behind. I *need* to skip >> versions to keep up. >> >> So for our next upgrades, we plan on skipping even more versions if >> the database migration allows it. (except for Nova which is a huge >> PITA to be honest due to CellsV1) >> I just don't see any other ways to keep up otherwise. > > ?!?! > > What does it take for this to never happen again? No operator should need to > plan and execute an upgrade for a whole year to upgrade one year's worth of > code development. > > We don't need new policies, new teams, more releases, fewer releases, or > anything like that. The goal is NOT "let's have an LTS release". The goal > should be "How do we make sure Mattieu and everyone else in the world can > actually deploy and use the software we are writing?" > > Can we drop the entire LTS discussion for now and focus on "make upgrades > take less than a year" instead? After we solve that, let's come back around > to LTS versions, if needed. I know there's already some work around that. > Let's focus there and not be distracted about the best bureaucracy for not > deleting two-year-old branches.
To add details to what happened: * Upgrade was never made a #1 priority. It was a one man show for far too long. (myself) * I also happen to manage and work on other priorities. * Lot of work made to prepare for multiple versions support in our deployment tools. (we use Puppet) * Lot of work in the packaging area to speedup packaging. (we are still using deb packages but with virtualenv to stay Puppet compatible) * We need to forward-port private patches which upstream won't accept and/or are private business logic. * Our developer teams didn't have enough free cycles to work right away on the upgrade. (this means delays) * We need to test compatibility with 3rd party systems which takes some time. (and make them compatible) * We need to update systems ever which we don't have full control. This means serious delays when it comes to deployment. * We need to test features/stability during some time in our dev environment. * We need to test features/stability during some time in our staging/pre-prod environment. * We need to announce and inform our users at least 2 weeks in advance before performing an upgrade. * We choose to upgrade one service at a time (in all regions) to avoid a huge big bang upgrade. (this means more maintenance windows to plan and you can't stack them too much) * We need to swiftly respond to bug discovered by our users. This means change of priorities and delay in other service upgrades. * We will soon need to upgrade operating systems to support latest OpenStack versions. (this means we have to stop OpenStack upgrades until all nodes are upgraded) All those details rapidly add up. We are far far away from a git pull && ./stack.sh I don't want to sound like too harsh but I feel some people live in a vacuum or an ideal world far from the reality of some operators. The above details are just a very small glimpse into my reality. I hope people will understand and have a different perspectives when it comes to upgrades. -- Mathieu _______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
