On 02/13/2014 02:51 AM, Thierry Carrez wrote:
John Griffith wrote:
So we've talked about this a bit and had a number of ideas regarding
how to test and show compatibility for third-party drivers in Cinder.
This has been an eye opening experience (the number of folks that have
NEVER run tempest before, as well as the problems uncovered now that
they're trying it).
I'm even more convinced now that having vendors run these tests is a
good thing and should be required. That being said there's a ton of
push back from my proposal to require that results from a successful
run of the tempest tests to accompany any new drivers submitted to
Cinder.
Could you describe the nature of the pushback ? Is it that the tests are
too deep and reject valid drivers ? Is it that it's deemed unfair to
block new drivers while the existing ones aren't better ? Is it that
it's difficult for them to run those tests and get a report ? Or is it
because they care more about having their name covered in mainline and
not so much about having the code working properly ?
The consensus from the Cinder community for now is that we'll
log a bug for each driver after I3, stating that it hasn't passed
certification tests. We'll then have a public record showing
drivers/vendors that haven't demonstrated functional compatibility,
and in order to close those bugs they'll be required to run the tests
and submit the results to the bug in Launchpad.
So, this seems to be the approach we're taking for Icehouse at least,
it's far from ideal IMO, however I think it's still progress and it's
definitely exposed some issues with how drivers are currently
submitted to Cinder so those are positive things that we can learn
from and improve upon in future releases.
To add some controversy and keep the original intent of having only
known tested and working drivers in the Cinder release, I am going to
propose that any driver that has not submitted successful functional
testing by RC1 that that driver be removed. I'd at least like to see
driver maintainers try... if the test fails a test or two that's
something that can be discussed, but it seems that until now most
drivers just flat out are not even being tested.
I think there are multiple stages here.
Stage 0: noone knows if drivers work
Stage 1: we know the (potentially sad) state of the drivers that are in
the release
Stage 2: only drivers that pass tests are added, drivers that don't pass
tests have a gap analysis and a plan to fix it
Stage 3: drivers that fail tests are removed before release
Stage 4: 3rd-party testing rigs must run tests on every change in order
to stay in tree
At the very minimum you should be at stage 1 for the Icehouse release,
so I agree with your last paragraph. I'd recommend that you start the
Juno cycle at stage 2 (for new drivers), and try to reach stage 3 for
the end of the Juno release.
I have to agree with Thierry here. I think if we can get drivers to
pass the tests
in the Juno timeframe, then it's fine to remove then during Juno.
I think the idea of having drivers run their code through tempest and work
towards passing all of those tests is a great thing for Cinder and
OpenStack in general.
What I would do different for the Icehouse release is this:
If a driver doesn't pass the certification test by IceHouse RC1, then we
have a bug filed
against the driver. I would also put a warning message in the log for
that driver that it
doesn't pass the certification test. I would not remove it from the
codebase.
Also:
if a driver hasn't even run the certification test by RC1, then we
mark the driver as
uncertified and deprecated in the code and throw an error at driver init
time.
We can have a option in cinder.conf that says
ignore_uncertified_drivers=False.
If an admin wants to ignore the error, they set the flag to True, and we
let the driver init at next startup.
The admin then takes full responsibility for running uncertified code.
I think removing the drivers outright is premature for Icehouse,
since the certification process is a new thing.
For Juno, we remove any drivers that are still marked as uncertified and
haven't run the tests.
I think the purpose of the tests is to get vendors to actually run their
code through tempest and
prove to the community that they are willing to show that they are
fixing their code. At the end of the day,
it better serves the community and Cinder if we have many working drivers.
My $0.02,
Walt
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
