On Mon, Mar 30, 2015 at 6:21 PM, Rochelle Grober <rochelle.gro...@huawei.com > wrote:
> Top posting… I believe the main issue was a problem with snapshots that > caused false negatives for most cinder drivers. But, that got fixed. > Unfortunately, we haven’t yet established a good process to notify third > parties when skipped tests are fixed and should be “unskipped”. Maybe > tagging the tests can help on this. But, I really do think this round was > a bit of first run gotchas and rookie mistakes on all sides. A good post > mortem on how to better communicate changes and deadlines may go a long way > to smooth these out in the next round. > > > > --Rocky > > > > John Griffith on Monday, March 30, 2015 15:36 wrote: > > On Mon, Mar 30, 2015 at 4:06 PM, Doug Wiegley < > doug...@parksidesoftware.com> wrote: > > A few reasons, I’m sure there are others: > > > > - Broken tests that hardcode something about the ref implementation. The > test needs to be fixed, of course, but in the meantime, a constantly > failing CI is worthless (hello, lbaas scenario test.) > > Certainly... but that's relatively easy to fix (bug/patch to Tempest). > Although that's not actually the case in this particular context as there > are a handful of third party devices that run the full set of tests that > the ref driver runs with no additional skips or modifications. > > > > > > - Test relies on some “optional” feature, like overlapping IP subnets > that the backend doesn’t support. I’d argue it’s another case of broken > tests if they require an optional feature, but it still needs skipping in > the meantime. > > > > This may be something specific to Neutron perhaps? In Cinder LVM is > pretty much the "lowest common denominator". I'm not aware of any volume > tests in Tempest that rely on optional features that don't pick this up > automatically out of the config (like multi-backend for example). > > > > > > - Some new feature added to an interface, in the presence of > shims/decomposed drivers/plugins (e.g. adding TLS termination support to > lbaas.) Those implementations will lag the feature commit, by definition. > > > > Yeah, certainly I think this highlights some of the differences between > Cinder and Neutron perhaps and the differences in complexity. > > Thanks for the feedback... I don't disagree per say, however Cinder is set > up a bit different here in terms of expectations for base functionality > requirements and compatibility but your points are definitely well taken. > > > > > Thanks, > > doug > > > > > > On Mar 30, 2015, at 2:54 PM, John Griffith <john.griffi...@gmail.com> > wrote: > > > > This may have already been raised/discussed, but I'm kinda confused so > thought I'd ask on the ML here. The whole point of third party CI as I > recall was to run the same tests that we run in the official Gate against > third party drivers. To me that would imply that a CI system/device that > marks itself as "GOOD" doesn't do things like add skips locally that aren't > in the tempest code already? > > > > In other words, seems like cheating to say "My CI passes and all is good, > except for the tests that don't work which I skip... but pay no attention > to those please". > > > > Did I miss something, isn't the whole point of Third Party CI to > demonstrate that a third parties backend is tested and functions to the > same degree that the reference implementations do? So the goal (using > Cinder for example) was to be able to say that any API call that works on > the LVM reference driver will work on the drivers listed in driverlog; and > that we know this because they run the same Tempest API tests? > > > > Don't get me wrong, certainly not saying there's malice or things should > be marked as no good... but if the practice is to skip what you can't do > then maybe that should be documented in the driverlog submission, as > opposed to just stating "Yeah, we run CI successfully". > > > > Thanks, > > John > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > Not top posting... >> I believe the main issue was a problem with snapshots that caused false negatives for most cinder drivers. But, that got fixed Huh? What was the problem, where was the problem, who/what fixed it, was there a bug logged somewhere, what comprises *most* Cinder drivers? Not disputing what you say, but for me it's just raised more questions than anything else.
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
