Sounds like we're actually in agreement, just disagreeing about the topology of the branches!
My preferred topology is actually a series of increasingly stable trunks, each of which has passed each level of the smoke tests (one platform, multi platform, torture tests). The nice thing about this is that then a release can simply be seen as the next level up in the stability hierarchy, with a manual pull. I think Google Chrome has a particularly sophisticated process which we could learn from: http://techcrunch.com/2011/01/11/google-chrome-release-cycle-slideshow/ I still think we should start simple with two trunks - stable and unstable. But I'll be happy if with per-branch testing also - that just seems a lot harder to me! <http://techcrunch.com/2011/01/11/google-chrome-release-cycle-slideshow/> Justin On Thu, Feb 24, 2011 at 10:24 AM, Trey Morris <trey.mor...@rackspace.com>wrote: > Instead of an unstable trunk, i think code should just be better vetted > before it lands in the trunk. If the difference between trunk and your > proposed unstable trunk is a set of automated tests, then those tests can > just as easily be run on a LP branch before it gets into current trunk. We > just need the tests. No matter how many stability levels of trunk we have, > without better tests we can't guarantee stability. > > On Thu, Feb 24, 2011 at 11:15 AM, Justin Santa Barbara < > jus...@fathomdb.com> wrote: > >> Hi Jay, >> >> I couldn't agree more. I had another bug come up yesterday on another of >> my patches (I know - not a good day for me!) where I again broke the >> OpenStack API by requiring the metadata attribute. >> >> In this case, it was missed by the unit tests. I believe I was always >> passing metadata, so simply missed the real world case. Here's the bug >> report: >> https://bugs.launchpad.net/nova/+bug/724143 >> >> This brings up a number of points though: >> >> On testing >> >> >> 1. The bug reporter apparently knows how to program, but instead of us >> getting a test case which we could immediately use, we got a Ruby test >> case. >> I think we should do whatever we can so that people that are moderately >> comfortable with code also feel comfortable submitting a failing test case >> that we can use. I think this means having some version of an API client, >> or even a Reference-Implementation API client, in the source tree, and >> using >> it. >> 2. It's not clear to me how we deal with unit test cases that are >> failing - cases that represent a found bug that is not yet fixed. Maybe >> it >> should be submitted on a bugNNN branch, with failing tests, and then >> whoever >> works on the bug can branch from it? (Of course, we'd be lucky if all >> bugs >> went like this, but it does happen. I often find myself fixing tangential >> issues, and I'd like to know the 'right thing to do' is) >> 3. On this particular metadata bug, it was my fault. I therefore >> submitted a very rapid hotfix which just fixes the issue. I then coded up >> unit tests that use the OpenStack API, and did in fact hit the issue >> naturally, and then included my fix which resolved it. I submitted the >> 'full patch' as a separate branch. >> 4. This 'full patch' branch includes unit tests that bring up the >> OpenStack API and various services in-process, and runs tests just like a >> user. It would have caught the metadata issue. >> 5. The hope is that we can reuse the same tests as smoke tests, by >> simply tweaking them to work against real services instead of bringing up >> in-process stub-services. These could be (some of) the smoke tests in >> Hudson. >> 6. I'd hope that we could have two levels of these smoke tests, one >> that runs on a single configuration (e.g. KVM, OpenISCSI, Glance); another >> that runs a matrix of configurations (and might take an hour or more to >> run) >> 7. Ideally we'd have a torture test that would run overnight and be >> randomized and try to find obscure bugs, even if issues found are not >> necessarily repeatable in the way that non-randomized tests are. >> >> >> We need an unstable trunk: >> >> >> 1. In general, it seems that our end-users are using trunk for >> unreleased functionality and treating it as if it were released. I don't >> think we should be encouraging that, because I know I'll make more >> mistakes >> in future and some of them will make it past the reviewers' defensive line >> into trunk; it's also simply not realistic to require reviewers to review >> every combination - e.g. how can a reviewer really review my HP SAN patch >> without an HP SAN? There will be issues in trunk, and if we have to >> revert >> them rather than just fixing them it will slow us down. The current >> situation is bad for our users and bad for developers. >> 2. One way we could keep everyone happy is by using our test suite to >> auto-merge from an 'unstable trunk' into 'stable trunk', only once code >> passes tests. Commits would initially merge into 'unstable trunk', and we >> would try to keep that branch moving forwards rather than reverting things >> that go wrong. Of course, maintaining a good 'stable trunk' relies on >> having good tests, but I think we're getting there. It's also great >> incentive to write good smoke tests. >> 3. Jay: I believe you've done this to great success on the Drizzle >> project? >> >> >> Justin >> >> >> >> On Thu, Feb 24, 2011 at 6:13 AM, Jay Pipes <jaypi...@gmail.com> wrote: >> >>> Hi all, >>> >>> I'd like to bring up an alternate reason why it was approved and >>> subsequently reverted. >>> >>> The test cases for the OpenStack API (and much of the EC2 API) assume >>> way too many things and mock out too many things. In addition, since >>> there are zero smoketests for the OpenStack API, there were no >>> functional tests that would have *immediately* highlighted this >>> problem (and many other recent EC2 vs OS API problems). >>> >>> In other words, sure, we should revert the patch to "fix things", >>> however the priority should *not* be on refactoring the auth API or >>> the way the auth layer in Nova is handled. The priority should be on >>> writing a smoketest for the OpenStack API so that we can link it into >>> Hudson and these types of issues can be automatically caught. >>> >>> -jay >>> >>> On Wed, Feb 23, 2011 at 10:03 PM, Paul Voccio <paul.voc...@rackspace.com> >>> wrote: >>> > Justin, >>> > I think you hit upon the reason of why I think it was approved and >>> reverted. >>> > Because it hadn't been talked about in a blueprint or a mail sent to >>> the >>> > list (I think I'm up to date on the threads) and a patch landed means >>> other >>> > alternatives weren't considered before pushing it through to begin >>> with. I >>> > think we're all open to talking about how to better the auth system and >>> make >>> > improvements. Dragon has already discussed some alternatives and >>> suggestions >>> > on the BP page below. I think this is the right way to continue the >>> dialog >>> > and we all can agree on a good way forward. >>> > I'm confident we can figure it out. >>> > If I missed a conversation, my apologies. >>> > pvo >>> > From: Vishvananda Ishaya <vishvana...@gmail.com> >>> > Date: Wed, 23 Feb 2011 18:19:41 -0800 >>> > To: Justin Santa Barbara <jus...@fathomdb.com> >>> > Cc: <openstack@lists.launchpad.net> >>> > Subject: Re: [Openstack] Should the OpenStack API re-use the EC2 >>> > credentials? >>> > >>> > Hey Justin, >>> > Does it make any difference that the way the auth is (theoretically) >>> > supposed to work with the os api is that the user gets an auth token >>> from an >>> > external auth server and then uses username / authtoken to actually >>> contact >>> > the api? I think it is just faked out right now to use the access_key >>> > instead of doing external auth, but I think the reason it works like it >>> does >>> > is because the plan was to switch to external auth eventually. >>> > Vish >>> > On Feb 23, 2011, at 5:56 PM, Justin Santa Barbara wrote: >>> > >>> > I previously fixed OpenStack authentication so it would use the same >>> > credentials as EC2. This bugfix was just reverted, because it caused >>> > OpenStack API users to have to enter in different credentials (sorry!), >>> but >>> > primarily because it hadn't been discussed on the mailing list. So >>> here >>> > goes! >>> > Here's a >>> > blueprint: >>> https://blueprints.launchpad.net/nova/+spec/authentication-consistency >>> > Here's an overview of the problem: >>> > EC2 uses an (api_key, api_secret) pair. Post-revert, OpenStack uses >>> the >>> > api_key(!) as the password, but a different value entirely as the >>> username: >>> > (username, api_key). The bugfix made it so that both APIs used the EC2 >>> > credentials (api_key, api_secret) . This did mean that anyone that had >>> > saved the 'bad' OpenStack credentials was unable to continue to use >>> those >>> > credentials. I also overlooked exporting the updated credentials in >>> novarc >>> > (though a merge request was pending). >>> > I actually thought originally that this was a straight-up bug, rather >>> than a >>> > design 'decision', so I should definitely have flagged it better. >>> Again, >>> > sorry to those I impacted. >>> > As things stand now, post-revert, this is probably a security flaw, >>> because >>> > the EC2 API does not treat the api_key as a secret. The EC2 API can >>> > (relatively) safely be run over non-SSL, because it uses signatures >>> instead >>> > of passing the shared secret directly. >>> > This is also not very user-friendly. Post-revert, an end-user must >>> know >>> > whether any particular cloud tool uses the EC2 API or the OpenStack >>> API, so >>> > that they can enter in the correct pair of credentials. That doesn't >>> seem >>> > like a good idea; I think there should be one set of credentials. >>> > >>> > There is some discussion about the idea of having the api_key be >>> > user-friendly. I don't think it buys us anything, because the >>> api_secret is >>> > still going to be un-friendly, but I have no objection as long as it is >>> does >>> > in a way that does not break existing users of the EC2 API. >>> > I propose that: >>> > (1) the OpenStack API and EC2 credentials should be the same as each >>> other >>> > (whatever they are) for the sake of our collective sanity and >>> > (2) we have to change the current configuration anyway for security >>> > reasons. >>> > (3) We should not change the EC2 credentials, because we've shipped >>> the EC2 >>> > API and our users have an expectation that we won't break them without >>> good >>> > reason, so >>> > (4) we must change the credentials for users of the (non-shipped) >>> OpenStack >>> > API. >>> > Estimated user impact: I believe there are two people that will be >>> affected, >>> > and it will take them ~1 minute each, so total impact ~2 minutes. >>> > The longer we delay fixing this, the more people we break and the >>> bigger the >>> > impact. It seems that we have no choice but to do a >>> > non-backwards-compatible authentication change, but I believe this is >>> OK at >>> > the moment because the OpenStack API is not yet stable/released - i.e. >>> we >>> > can still make fixes without worrying about backwards compatibility >>> shims. >>> > We're not in "The Old New Thing" land yet :-) >>> > >>> > >>> > As an aside, I am very unhappy about the way this revert was pushed >>> through >>> > by Rackspace team-members, seemingly without much consideration of >>> > alternatives. Perhaps we should consider changing from needing two >>> > core-approves, to needing one Rackspace core-approve and one >>> non-Rackspace >>> > core-approve. >>> > >>> > Justin >>> > >>> > >>> > _______________________________________________ >>> > Mailing list: https://launchpad.net/~openstack >>> > Post to : openstack@lists.launchpad.net >>> > Unsubscribe : https://launchpad.net/~openstack >>> > More help : https://help.launchpad.net/ListHelp >>> > >>> > _______________________________________________ Mailing list: >>> > https://launchpad.net/~openstack Post to : >>> openstack@lists.launchpad.net >>> > Unsubscribe : https://launchpad.net/~openstack More help : >>> > https://help.launchpad.net/ListHelp >>> > >>> > Confidentiality Notice: This e-mail message (including any attached or >>> > embedded documents) is intended for the exclusive and confidential use >>> of >>> > the >>> > individual or entity to which this message is addressed, and unless >>> > otherwise >>> > expressly indicated, is confidential and privileged information of >>> > Rackspace. >>> > Any dissemination, distribution or copying of the enclosed material is >>> > prohibited. >>> > If you receive this transmission in error, please notify us immediately >>> by >>> > e-mail >>> > at ab...@rackspace.com, and delete the original message. >>> > Your cooperation is appreciated. >>> > >>> > _______________________________________________ >>> > Mailing list: https://launchpad.net/~openstack >>> > Post to : openstack@lists.launchpad.net >>> > Unsubscribe : https://launchpad.net/~openstack >>> > More help : https://help.launchpad.net/ListHelp >>> > >>> > >>> >> >> >> _______________________________________________ >> >> Mailing list: https://launchpad.net/~openstack >> Post to : openstack@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~openstack >> More help : https://help.launchpad.net/ListHelp >> >> > Confidentiality Notice: This e-mail message (including any attached or > embedded documents) is intended for the exclusive and confidential use of the > individual or entity to which this message is addressed, and unless otherwise > expressly indicated, is confidential and privileged information of Rackspace. > Any dissemination, distribution or copying of the enclosed material is > prohibited. > If you receive this transmission in error, please notify us immediately by > e-mail > at ab...@rackspace.com, and delete the original message. > Your cooperation is appreciated. > >
_______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
