Thanks to everyone for your feedback in this thread. Also, thanks to Ash and Randy with whom I talked at OSLS last week. It's clear from the feedback that the community, including our end users, feel strongly about having additional releases, beyond the standard, 6-month major release cadence.
I think that Frank's description of monthly micro-releases is compelling and I believe that the work that Fatih, Jack, and Uli are doing with CI evolution and cross-community CI may enable us to consider this type of release process in the future. In the mean time, I'm withdrawing my proposal to drop the 2.0 and 3.0 follow-on releases for the upcoming "E" - major release. I have uploaded a revised schedule <https://wiki.opnfv.org/display/SWREL/E-River> to the wiki. We will discuss the schedule again at the next TSC call. Feel free to continue to respond to this thread with your comments and questions. David On Thu, Feb 23, 2017 at 10:43 AM, Randy Levensalor < [email protected]> wrote: > Thanks for the feedback and ideas of solving our challenges with running > OPNFV as a VNF development and evaluation platform. > > I was not trying to prescribe a specific solution, but wanted to call out > the challenges that we are encountering when running OPFNV in our NFV labs. > > The direction of Tim and Frank’s feedback could help address these > challenges. Enabling the installer teams to focus on maintenance post > release. > > Enabling scenario teams to rollout new scenarios in maintenance releases > that do NOT negatively impact the stability of the currently supported > scenarios could be a reasonable compromise. Or just following the strict > model as outlined by Ulrich would help with the stability. > > Please don’t break backwards compatibility of the configuration files > during maintenance releases. Changing configuration files with major > releases would be manageable. > > > Many Thanks, > > Randy Levensalor | [email protected] > CableLabs® | o 303.661.3455 | c 970.214.1316 > > > On 2/22/17, 8:18 AM, "Frank Brockners (fbrockne)" <[email protected]> > wrote: > > That makes sense. The whole point of micro-releases would be to > release scenarios once they are ready - and it falls into the > responsibility of a scenario owner to shepherd the process. Micro-releases > should *not* be a vehicle to push other teams to go out of their way. > > Frank > > -----Original Message----- > From: Tim Rozet [mailto:[email protected]] > Sent: Mittwoch, 22. Februar 2017 15:53 > To: Ulrich Kleber <[email protected]> > Cc: Randy Levensalor <[email protected]>; Frank Brockners > (fbrockne) <[email protected]>; David McBride < > [email protected]>; TECH-DISCUSS OPNFV < > [email protected]>; [email protected] > Subject: Re: [opnfv-tech-discuss] [release] E-release schedule > > I think having maintenance releases makes sense. From an installer > perspective, if folks want to add a new scenario for a maintenance release > then they need to carry most of the weight of integrating it. It is hard > for a small team like Apex to juggle adding features for a previous release > while rushing to get the next version of OS to work for the following > release. Bug fixing and documentation updates I think we are fine with for > maintenance releases. > > Tim Rozet > Red Hat SDN Team > > ----- Original Message ----- > From: "Ulrich Kleber" <[email protected]> > To: "Randy Levensalor" <[email protected]>, "Frank Brockners > (fbrockne)" <[email protected]>, "David McBride" < > [email protected]>, "TECH-DISCUSS OPNFV" < > [email protected]>, [email protected] > Sent: Wednesday, February 22, 2017 8:00:09 AM > Subject: Re: [opnfv-tech-discuss] [release] E-release schedule > > > > Hi, > > what Randy explains would mean that we should move to a very strict > concept of maintenance releases. > > That would mean we carry through our milestones properly to get the > 1.0 release out. After that we should only do bug fixing. Not a single > feature. > > If we do that, 2.0 and 3.0 will not take big efforts, but each provide > better stability. It shouldn’t even be a problem to do 4.0 if there are > urgent fixes. > > But we should be strict then: Every feature that misses the 1.0, has > to wait for the next major release. > > Just my 2 cents. > > Cheers, > > Uli > > > > > > > From: [email protected] [mailto: > [email protected]] On Behalf Of Randy Levensalor > Sent: Friday, 17 February, 2017 00:16 > To: Frank Brockners (fbrockne); David McBride; TECH-DISCUSS OPNFV; > [email protected] > Subject: Re: [opnfv-tech-discuss] [release] E-release schedule > > > > > Frank, +1 on your suggestions. > > David, I applaud your effort to reduce the overhead for the community. > > > > TL;DR Today I spend > 80% of my time getting OPNFV to run and I’d > prefer to spend > 80% of my time running VNFs on OPNFV. > > > > As a frequent user, use case 1 that Frank defined aligns with my > primary goal for OPNFV. “(1) User/consumer of a readily integrated NFV > stack.” The need for a working platform currently outweigh the need for > additional features. I suspect that as the platform and our use cases > mature, some of the incremental features will become a higher priority. > > > > With the one release every 6 months, I fear that I would have to carry > a large patch set or something to have a functioning platform. > > > > To add a little context to my reservations about the reduced number of > releases, I’d like to share my experience below as an example of how the > release cycle can impact users. Please don’t read this as a complaint or > detracting from the project’s progress and achievement. The fact that we > can run a 100% open source VIM and NFVI with a very limited staff is a > minor miracle. There may some other clever solutions. Fatih and Jack have > mentions some options that could help. > > > > With the Arno, Brahmaputra and Colorado releases we would start > downloading release candidates and kick the tires. We try and look at > multiple installers, but most of our time has been spent on Apex. > > > > The RC installs almost never work. About 1/3 rd of these failures are > due to hardware or user error that are not easy to debug. For instance, a > bad NIC on a compute node can cause the controller install to fail. The > rest are defects or documentation issues. > > > > The1.0 release goes better. I can usually get a few patches into the > first release to resolve critical issue and the community is working day > and night to get the release out the door. > > > > By the 2.0 release, most of the defects are fixed. There are > inevitably a few issues that take more than a month to root cause. Some of > these more challenging defects, which are often related to platform > stability, cannot be fixed until the next release. Resources are already > focused on adding new features and do not have the time nor desire to > backport these fixes. > > > > The longest that I have kept an OPNFV Colorado instance running was 32 > days. That required the Colorado 3.0 release, manually applying a few > patches that will be in Danube and limiting the type activities. > > > > Now we are preparing to kick the tires on Danube continue with > difficulties keeping our lab running long enough to deploy an interesting > use case. In the meantime, we will continue to reinstall at least once a > month and look forward to the Danube release. > > > > > > > Randy Levensalor | [email protected] Cable Labs® | o > 303.661.3455 | c 970.214.1316 > > > > > > > > From: < [email protected] > on behalf of > "Frank Brockners (fbrockne)" < [email protected] > > Date: Thursday, February 16, 2017 at 1:36 AM > To: David McBride < [email protected] >, OPNFV Tech > Discussion < [email protected] >, " > [email protected] " < opnfv-project-leads@lists. > opnfv.org > > Subject: Re: [opnfv-tech-discuss] [release] E-release schedule > > > > > > David, > > > > thanks for the summary. Let’s remind ourselves, that in OPNFV we’re > really trying to meet the needs of two different audiences: (1) > User/consumer of a readily integrated NFV stack – as well as marketing > operations (2) Developer/tester of an NFV stack. Audience #1 is mostly > interested in stability, even if that means that things are released a > little later (i.e. you build on long released components). Audience #2 is > pushing the envelope and requires the ability to evolve/develop and > integrate the latest set of components; once working they desire to release > things to allow others to build on top; and move on/start over. > > > > The current 1.0/2.0/3.0 was an effort to meet the needs of both > audiences, i.e. > > · Have a “major” release. > > · Allow developers to release scenarios when they are ready and > evolve, without too much of a maintenance burden. > > This is also why we typically did not fix component versions for a > release, but said: Based or ODL Boron or later. > > > > I agree that releases are not free – especially the “major” release, > because it comes with significant documentation and coordination needs. > That said, it is mostly the “major” release with a lot of central > coordination which creates efforts. Labeling and pushing an updated version > of test results and documentation is relatively low effort – and can even > be done by a scenario team. It does not even require central coordination. > And our pipeline is now mature enough to do these things with low/moderate > overhead. > > > > So rather than move back in history and go back to a single release > every 6 months, which will make OPNFV a very inflexible organization for > developers, I would strongly suggest that we rather consider evolving the > current release process. IMHO we should be ready to have monthly > micro-releases (scenario owners publish those scenarios which are “ready”, > i.e. have docs ready and pass testing), and every 6 months we do a > macro-release (with marketing/press announcement) which includes the set of > scenarios which are “ready” by then. Macro-releases can be coupled to > certain upstream component versions (as selection criteria for what is > in/out of a macro release) – whereas micro-release would enjoy complete > freedom. > > > > Thoughts? > > > > Thanks, Frank > > > > > > > > From: David McBride [ mailto:[email protected] ] > Sent: Mittwoch, 15. Februar 2017 20:26 > To: TECH-DISCUSS OPNFV < [email protected] >; > [email protected] > Cc: Frank Brockners (fbrockne) < [email protected] >; Tapio Tallgren > < [email protected] > > Subject: [release] E-release schedule > > > > > Greetings, > > > > > > During the TSC call, yesterday, I took an action to start an email > discussion about the schedule for the E-release . > > > > > > Specifically, I suggested that we just plan for a single release, > rather than three releases, as we've done in the past. Then, when the > release date approaches, we evaluate whether we need a point release, then > schedule it at that time. > > > > > > Why? > > > * Scheduling three releases has created a lot of confusion with > the project teams The purpose of the three releases is to give project > teams time to debug and fix scenarios that are not ready for 1.0. They are > not separate development timelines with separate release milestones. > However, many believe that it isn't necessary to meet release milestones, > because they will simply shift to the 2.0 or 3.0 release. > * In the past two releases, the new content released in 2.0 has > been minimal. For example, for Colorado 2.0, just two new scenarios were > released. Human nature is such that, given the opportunity for a later > deadline, many will take it. > * Releases are not free. In addition to the overhead required for > labeling, creating ISOs, and updating documentation, projects that released > in previous releases, are required to update their code for subsequent > releases to resolve any issues, even if they weren't intending to do any > additional work on that major release. For example, let's say that a > project releases in Danube 1.0, they're satisfied with their effort, so > they shift their focus to the E-release. However, changes after 1.0 break > their scenario. So, suddenly, they find themselves working on Danube 2.0, > even though they aren't releasing any new scenarios. This process repeats > for Danube 3.0. > > > During the TSC call, it was suggested that a 2.0 or 3.0 release > provides an opportunity to integrate a late release of a major upstream > component (e.g. ODL). However, this is counter to our previous agreement > not to change major upstream components after the 1.0 release. > Unfortunately, this happened in Colorado and created significant > disruption, including a slip in the 2.0 release. > > > > > > Per our discussion on Tuesday, I've created a wiki page to capture > pros and cons of various schedule options. Feel free to edit it and add > your thoughts. > > > > > > David > > > > > > -- > > > David McBride > > > Release Manager, OPNFV > > > Mobile: +1.805.276.8018 > > > Email/Google Talk: [email protected] > > > Skype: davidjmcbride1 > > > IRC: dmcbride > > _______________________________________________ > opnfv-tech-discuss mailing list > [email protected] > https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss > > > -- *David McBride* Release Manager, OPNFV Mobile: +1.805.276.8018 Email/Google Talk: [email protected] Skype: davidjmcbride1 IRC: dmcbride
_______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
