Re: [openstack-dev] [OpenStack][InstanceGroup] metadetails and metadata in instance_group.py
> As the person who -2'd the review, I'm thankful you raised this issue on > the ML, Jay. Much appreciated. The "metadetails" term isn't being invented in this patch, of course. I originally complained about the difference when this was being added: https://review.openstack.org/#/c/109505/1/nova/api/openstack/compute/contrib/server_groups.py,cm As best I can tell, the response in that patch set about why it's being translated is wrong (backwards). I expect that the API extension at the time called it "metadetails" and they decided to make the object the same and do the translation there. From what I can tell, the actual server_group API extension that made it into the tree never got the ability to set/change/etc the metadata/metadetails anyway, so there's no reason (AFAICT) to add it in wrongly. If we care to have this functionality, then I propose we change the attribute on the object (we can handle this with versioning) and reflect it as "metadata" in the API. However, I have to ask: do we really need another distinct metadata store attached to server_groups? If not, how about we just remove it from the database and the object, clean up the bit of residue that is still in the API extension and be done with it? --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Retrospective veto revert policy
> Looks reasonable to me. +1 --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][core] Expectations of core reviewers
> I'm not questioning the value of f2f - I'm questioning the idea of > doing f2f meetings sooo many times a year. OpenStack is very much > the outlier here among open source projects - the vast majority of > projects get along very well with much less f2f time and a far > smaller % of their contributors attend those f2f meetings that do > happen. So I really do question what is missing from OpenStack's > community interaction that makes us believe that having 4 f2f > meetings a year is critical to our success. How many is too many? So far, I have found the midcycles to be extremely productive -- productive in a way that we don't see at the summits, and I think other attendees agree. Obviously if budgets start limiting them, then we'll have to deal with it, but I don't want to stop meeting preemptively. IMHO, the reasons to cut back would be: - People leaving with a "well, that was useless..." feeling - Not enough people able to travel to make it worthwhile So far, neither of those have been outcomes of the midcycles we've had, so I think we're doing okay. The design summits are structured differently, where we see a lot more diverse attendance because of the colocation with the user summit. It doesn't lend itself well to long and in-depth discussions about specific things, but it's very useful for what it gives us in the way of exposure. We could try to have less of that at the summit and more midcycle-ish time, but I think it's unlikely to achieve the same level of usefulness in that environment. Specifically, the lack of colocation with too many other projects has been a benefit. This time, Mark and Maru where there from Neutron. Last time, Mark from Neutron and the other Mark from Glance were there. If they were having meetups in other rooms (like at summit) they wouldn't have been there exposed to discussions that didn't seem like they'd have a component for their participation, but did after all (re: nova and glance and who should own flavors). > As pointed out this benefit for core devs has a direct negative > impact on other non-core devs. I'm questioning whether this is > really a net win overall vs other approaches to collaboration. It's a net win, IMHO. > As I explain in the rest of my email below I'm not advocating > getting rid of mid-cycle events entirely. I'm suggesting that > we can attain a reasonable % of the benefits of f2f meetings > by doing more formal virtual meetups and so be more efficient > and inclusive overall. I'd love to see more high-bandwidth mechanisms used to have discussions in between f2f meetings. In fact, one of the outcomes of this last midcycle was that we should have one about APIv3 with the folks that couldn't attend for other reasons. It came up specifically because we made more progress in ninety minutes than we had in the previous eight months (yes, even with a design summit in the middle of that). Expecting cores to be at these sorts of things seems pretty reasonable to me, given the usefulness (and gravity) of the discussions we've been having so far. Companies with more cores will have to send more or make some hard decisions, but I don't want to cut back on the meetings until their value becomes unjustified. --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][core] Expectations of core reviewers
On 8/13/14 11:20 AM, Mike Bayer wrote: > On Aug 13, 2014, at 1:44 PM, Russell Bryant > wrote: >> I disagree. IMO, *expecting* people to travel, potentially across >> the globe, 4 times a year is an unreasonable expectation, and >> quite uncharacteristic of open source projects. If we can't figure >> out a way to have the most important conversations in a way that is >> inclusive of everyone, we're failing with our processes. >> >> By all means, if a subset wants to meet up and make progress on >> some things, I think that's fine. I don't think anyone think it's >> not useful. Well, it doesn't seem at all excessive to me, given the rate and volume at which we do things around here. That said, if a significant number of cores think it's not doable, then I guess that's a data point. From what you said above, it sounds like you're okay with the meetings but not the requirement for cores. I said "expect" above -- is that a reasonable thing? Expect them to be present, unless they have a reason not to be there? Reasons could be personal or preference, but hopefully not "I never come to midcycles because $reason." > It’s difficult to compare OpenStack to other open source projects, in > that it is on such a more massive and high velocity scale than almost > any others (perhaps the Linux kernel is similar). Yeah, I have a hard time justifying anything by comparing us to other projects. I've been involved with plenty and don't think any of them are useful data points for what we should or should not do here in terms of anything related to velocity :) --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit
> You may have noticed that this has merged, along with a further change > that shows the latest results in a table format. (You may need to > force-reload in your browser to see the change.) Friggin. Awesome. > Thanks again to Radoslav Gerganov for writing the original change. Thanks to all involved, as this is a major improvement for everyone! One thing that we discussed at the nova meetup was having a space for each CI we *expect* to vote. I haven't looked at the implementation here, but I assume it just parses the comments to generate the table. How hard would it be to make the table show all the CI systems we expect so that it's very obvious that one has gone MIA (as they often do) before we merge a patch? I think we struggle right now with merging things that a CI system would have NAKed, but only because we didn't notice that it hadn't voted. --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Review priorities as we approach juno-3
> == Move Virt Drivers to use Objects (Juno Work) == > > I couldn't actually find any code out for review for this one apart > from https://review.openstack.org/#/c/94477/, is there more out there? This was an umbrella one to cover a bunch of virt driver objects work done early in the cycle. Much of that is done, I haven't gone looking for anything to see if there are any obvious things to include under this anymore, but I'll try to do that. > == Add a virt driver for Ironic == > > This one is in progress, but we need to keep going at it or we wont > get it merged in time. > > * https://review.openstack.org/#/c/111223/ was approved, but a rebased > ate it. Should be quick to re-approve. > * https://review.openstack.org/#/c/111423/ > * https://review.openstack.org/#/c/111425/ > * ...there are more reviews in this series, but I'd be super happy to > see even a few reviewed I've been reviewing this pretty heavy and I think that it's just taking a while to make changes given the roundabout way they're getting done first in Ironic. I'm pretty confident that this one will be okay. > == VMware: spawn refactor == > > * https://review.openstack.org/#/c/104145/ > * https://review.openstack.org/#/c/104147/ (Dan Smith's -2 on this one > seems procedural to me) Yep, we're just trying to get MineSweeper votes on them before letting them in. We already had one refactor go in without a minesweeper run that ... broke minesweeper :) --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OS or os are not acronyms for OpenStack
> OS or os is operating system. I am starting to see some people us OS or > os to mean OpenStack. This is confusing and also incorrect[0]. Except in the nova API code, where 'os' means 'openstack'. I too think that policing the use of abbreviations is silly. It's a confusing abbreviation, but seriously... --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Enabling silent Docker tests for Nova?
> Feature freeze is only a few weeks away (Sept 4). How about we just > leave it in experimental until after that big push? That seems pretty > reasonable. Joe just proposed dropping a bunch of semi-useless largeops runs. Maybe that leaves room to sneak this in? If the infra team is okay with it, I'd say we should go ahead and do it, but regardless of that decision, I'm +1 on getting this promoted so we can start the "stability timer". --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] requesting an FFE for SRIOV
> The main sr-iov patches have gone through lots of code reviews, manual > rebasing, etc. Now we have some critical refactoring work on the > existing infra to get it ready. All the code for refactoring and sr-iov > is up for review. I've been doing a lot of work on this recently, and plan to see it through if possible. So, I'll be a sponsor. In the meeting russellb said he would as well. I think he's tied up today, so I'm proxying him in here :) --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][FFE] Feature Freeze exception for juno-slaveification
> All the other patches from this blueprint have merged, the only > remaining patch really just needs a +W as it has been extensively > reviewed and already approved previously. This may be an easy > candidate since Andrew Laski, Jay Pipes and Dan Smith have reviewed > and +2'd this already. > > I am happy to sponsor this, as this is the last patch needed to finish a > BP and the patch was approved once bit needed a rebase over some objects > changes. The change from the approved version is pretty minor. Yeah, this was approved before the deadline, just failed in the gate, and is the last patch of a maintenance blueprint. It's very low risk and lets us mark off another thing. I'll sponsor this one too. --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Averting the Nova crisis by splitting out virt drivers
>> The last few days have been interesting as I watch FFEs come through. >> People post explaining their feature, its importance, and the risk >> associated with it. Three cores sign on for review. All of the ones >> I've looked at have received active review since being posted. Would >> it be bonkers to declare nova to be in "permanent feature freeze"? If >> we could maintain the level of focus we see now, then we'd be getting >> heaps more done that before. > > Agreed. Honestly, this has been a really nice flow. I'd love to figure > out what part of this focus is capturable for normal cadence. This > realistically is what I was hoping slots would provide, because I feel > like we actually move really fast when we call out 5-10 things to go > look at this week. The funny thing is, last week I was thinking how similar FF is to what slots/runways would likely provide. That is, intense directed focus on a single thing by a group of people until it's merged (or fails). Context is kept between iterations because everyone is on board for quick iterations with minimal distraction between them. It *does* work during FF, as we've seen in the past -- I'd expect we have nearly 100% merge rate of FFEs. How we arrive at a thing getting focus is different in slots/runways, but I feel the result could be the same. Splitting out the virt drivers is an easy way to make the life of a core much easier, but I think the negative impacts are severe and potentially irreversible, so I'd rather make sure we're totally out of options before we exercise it. --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] On an API proxy from baremetal to ironic
> As far as I understand it, though, that's a patch for a read-only > mode. It seems bizzare, and possibly dangerous, to proxy read > commands, but not write commands. It gives the impression that > everything's fine until it's not fine (because someone tried to use > an existing script to do a create command). IMHO, it would be better > to just tell people up front "Update your scripts to use Ironic, > because they won't work at all" instead of leading people (through > empirical evidence) to believe that their scripts will work, and then > having them discover later that something broke because they tried to > create a node. How is it dangerous? Most code making "write commands" would need to be pretty diligent about making sure that the thing being requested actually succeeded. Having the proxy allows us to return a reasonable code for those things (i.e. 403 Forbidden, perhaps) instead of just "500 Huh? What?". I was pro-proxy from the beginning, not because I think proxies are awesome, but because that's what we do when we move things out of Nova's API to other services. Some feel this is a purely admin API and that gives us license to break our own rules here, but I don't really understand where, when and why we draw that line. The code is written, it's minor, and it gives a much more graceful response to the move. We know there are people running this, with clusters in the thousands. We don't know who they all are, what they're doing with it, and we don't know that all of them are happy or expecting to immediately rewrite all of their tools. I don't really see why this is a big deal. --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] On an API proxy from baremetal to ironic
> 1) Is this tested anywhere? There are no unit tests in the patch and > it's not clear to me that there would be any Tempest coverage of this > code path. Providing this and having it break a couple of months down > the line seems worse than not providing it at all. This is obviously > fixable though. AFAIK, baremetal doesn't have any tempest-level testing at all anyway. However, I don't think our proxy code breaks, like, ever. I expect that unit tests for this stuff is plenty sufficient. > 2) If we think maintaining compatibility for existing users is that > important, why aren't we proxying everything? Is it too > difficult/impossible due to the differences between Baremetal and > Ironic? And if they're that different, does it still make sense to > allow one to look like the other? As it stands, this isn't going to > let deployers use their existing tools without modification anyway. Ideally we'd proxy everything, based on our current API guarantees. However, I think the compromise of just the show/index stuff came about because it would be extremely easy to do, provide some measure of continuity, and provide us a way to return something nicer for the create/update operations than a 500. It seemed like a completely fair and practical balance. --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Proposal to add Matt Riedemann to nova-core
> Please respond with +1/-1, or any further comments. +1 from me -- Matt has been helping a lot lately. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] FreeBSD hypervisor (bhyve) driver
> So just to clarify: the native driver for another hypervisor (bhyve) > would not be accepted into Nova even if it met testing coverage > criteria? As I said the libvirt route is an option we consider, but we > would like to have the possibility of a native FreeBSD api integration > as well, similar to what you can have for non-libvirt hypervisor apis > available already in Nova. I'd _really_ prefer to see a libvirt-based approach over a native one. In the presence of one, I would hope we would not need to take on the maintenance burdon of a native driver. In the absence, I'd want to know why. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan
> My overall concern, and I think the other guys doing this for virt > drivers will agree, is trying to scope down the exposure to unrelated > failures. But, if it's not related to your driver, then it also failed in the upstream gate, right? If it didn't fail the upstream gate, then it is some weird interaction. Of course, for spurious failures, you'll still have the case where you recheck something and it passes the next time, but that's exactly what we have today. I don't see a new CI job for a given driver being a "monstrous" amount of work above and beyond what we have to do today to triage the issues. Of course, driver CI is not a single-time task that you never have to babysit... I think that running as close as possible to the upstream gate is crucial for us to gain the level of comfort we have with the stuff that is actually tested upstream. If you pass more often because you don't include any of the tests you don't strictly have to run, and don't have a set of folks ready to triage failures, then you don't have quality parity with the other stuff. That's kinda the whole point of us doing this, no? --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Docker] Environment variables
> eg use a 'env_' prefix for glance image attributes > > We've got a couple of cases now where we want to overrides these > same things on a per-instance basis. Kernel command line args > is one other example. Other hardware overrides like disk/net device > types are another possibility > > Rather than invent new extensions for each, I think we should > have a way to pass arbitrary attributes alon with the boot > API call, that a driver would handle in much the same way as > they do for glance image properties. Basically think of it as > a way to custom any image property per instance created. Personally, I think having a bunch of special case magic namespaces (even if documented) is less desirable than a proper API to do something like this. Especially a namespace that someone else could potentially use legitimately that would conflict. To me, this feels a lot like what I'm worried this effort will turn into, which is making containers support in Nova look like a bolt-on thing with a bunch of specialness required to make it behave. Anyone remember this bolt-on gem? nova boot --block-device-mapping vda=965453c9-02b5-4d5b-8ec0-3164a89bf6f4:::0 --flavor=m1.tiny --image=6415797a-7c03-45fe-b490-f9af99d2bae0 BFV I found that one amidst hundreds of forum threads of people confused about what incantation of magic they were supposed to do to make it actually boot from volume. Just MHO. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Future meeting times
> I am fine with this, but I will never be attending the 1400 UTC > meetings, as I live in utc-8 I too will miss the 1400UTC meetings during the majority of the year. During PDT I will be able to make them, but will be uncaffeinated. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Live upgrades and major rpc versions
> Using this approach I think we can support live upgrades from N-1 to N > while still being able to drop some backwards compatibility code each > release cycle. Agreed. We've been kinda slack about bumping the RPC majors for a while, which means we end up with a lot of cruft and comments like "#NOTE: Remove this in Grizzly" still in the code. Major numbers are free, we should use them more :) > Thoughts? I wonder if it's worth also saying something in the checklist about "if the API hasn't changed in this release, no need to bump". Just so we don't get to RPC version 23.0 for the console API, or something else that doesn't change much. I don't have a feeling for how this would translate to objects, if at all, so I'll reserve judgment on that for the moment. Mechanically, it's a very similar thing, but we've not had enough churn in the object APIs to know if being this proactive is at all warranted. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Detect changes in object model
> If an object A contains another object or object list (called > sub-object), any change happened in the sub-object can't be detected > by obj_what_changed() in object A. Well, like the Instance object does, you can override obj_what_changed() to expose that fact to the caller. However, I think it might be good to expand the base class to check, for any NovaObject fields, for the obj_what_changed() of the child. How does that sound? --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Detect changes in object model
> Sounds good to me. The list base objects don't have methods to make changes > to the list - so it would be a case of iterating looking at each object in > the list. That would be ok. Hmm? You mean for NovaObjects that are lists? I hesitate to expose lists as changed when one of the objects inside has changed because I think that sends the wrong message. However, I think it makes sense to have a different method on lists for "are any of your contents changed?" I'll cook up a patch to implement what I'm talking about so you can take a look. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The extra_resource in compute node object
> This patch set makes the extra_resources a list of object, instead of > opaque json string. How do you think about that? Sounds better to me, I'll go have a look. > However, the compute resource object is different with current > NovaObject, a) it has no corresponding table, but just a field in > another table, and I assume it will have no save/update functions. b) > it defines the functions for the object like alloc/free etc. Not sure > if this is correct direction. Having a NovaObject that isn't backed by a conventional SQLAlchemy model is fine with me, FWIW. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Detect changes in object model
> ObjectListBase has a field called objects that is typed
> fields.ListOfObjectsField('NovaObject'). I can see methods for count
> and index, and I guess you are talking about adding a method for "are
> any of your contents changed" here. I don't see other list operations
> (like append, insert, remove, pop) that modify the list. If these
> were included they would have to mark the list as changed so it is
> picked up when looking for changes.
>
> Do you see these belonging here or would you expect those to go in a
> sub-class if they were wanted?
Well, I've been trying to avoid implying the notion that a list of
things represents the content of the database. Meaning, I don't think it
makes sense for someone to get a list of Foo objects, add another Foo to
the list and then call save() on the list. I think that ends up with the
assumption that the list matches the contents of the database, and if I
add or remove things from the list, I can save() the contents to the
database atomically. That definitely isn't something we can or would
want to support.
That said, if we make the parent object consider the child to be dirty
if any of its contents are dirty or the list itself is dirty (i.e. the
list of objects has changed) that should give us the desired behavior
for change tracking, right?
--Dan
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Detect changes in object model
> Hi Dan, are you going to cook a patch to expand the base class? Or we can do > that ourselves? Yeah, I'll try to get to that today. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Neutron] nova-network in Icehouse and beyond
> Effective immediately, I would like to unfreeze nova-network > development. I fully support this plan, while also agreeing that Neutron is the future of networking for OpenStack. As we have seen with recent performance-related gate failures, we cannot continue to ignore nova-network while the rest of the system moves forward. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Neutron] nova-network in Icehouse and beyond
> I was thinking for the upgrade process that we could leverage the port > attach/detach BP done by Dan Smith a while ago. This has libvirt support > and there are patches pending approval for Xen and Vmware. Not sure about > the other drivers. > > If the guest can deal with the fact that the nova port is being removed > and a new logical port is added then we may have the chance of no down > time. If this works then we may need to add support for nova-network port > detach and we may have a seamless upgrade path. That's a good thought for sure. However, it would be much better if we could avoid literally detaching the VIF from the guest and instead just swap where the other end is plugged into. With virtual infrastructure, that should be pretty easy to do, unless you're switching actual L2 networks. If you're doing the latter, however, you might as well reboot the guest I think. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Neutron] nova-network in Icehouse and beyond
> If necessary the tasks work could be done solely as an extension, but I > would really prefer to avoid that so I'll get this ball rolling quickly. I agree that doing it as a bolt-on to v3 would be significantly less favorable than making it an integrated feature of the API. IMHO, if a server create operation doesn't return a task, then we failed, as that is (to me) one of the primary cases where a task object is important. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Scheduler] Will the Scheuler use Nova Objects?
> I'm of the opinion that the scheduler should use objects, for all the > reasons that Nova uses objects, but that they should not be Nova > objects. Ultimately what the scheduler needs is a concept of capacity, > allocations, and locality of resources. But the way those are modeled > doesn't need to be tied to how Nova does it, and once the scope expands > to include Cinder it may quickly turn out to be limiting to hold onto > Nova objects. Yeah, my response to the original question was going to be something like: "If the scheduler was staying in Nova, it would use NovaObjects like the rest of Nova. Long-term Gantt should use whatever it wants and the API between it and Nova will be something other than RPC and thus something other than NovaObject anyway." I think the point you're making here is that the models used for communication between Nova and Gantt should be objecty, regardless of what the backing implementation is on either side. I totally agree with that. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] do nova objects work for plugins?
> Basically, if object A has object B as a child, and deserialization > finds object B to be an unrecognized version, it will try to back > port the object A to the version number of object B. Right, which is why we rev the version of, say, the InstanceList when we have to rev Instance itself, and why we have unit tests to makes sure that happens. > It is not reasonable to bump the version of the compute_node when > new external plugin is developed. So currently the versioning seems > too rigid to implement extensible/pluggable objects this way. So we're talking about an out-of-tree closed-source plugin, right? IMHO, Nova's versioning infrastructure is in place to make Nova able to handle upgrades; adding requirements for supporting out-of-tree plugins wouldn't be high on my priority list. > A reasonable alternative might be for all objects to be deserialized > individually within a tree data structure, but I’m not sure what > might happen to parent/child compatability without some careful > tracking. I think it would probably be possible to make the deserializer specify the object and version it tripped over when passing the whole thing back to conductor to be backleveled. That seems reasonably useful to Nova itself. > Another might be to say that nova objects are for nova use only and > that’s just tough for plugin writers! Well, for the same reason we don't provide a stable virt driver API (among other things) I don't think we need to be overly concerned with allowing arbitrary bolt-on code to hook in at this point. Your concern is, I assume, allowing a resource metric plugin to shove actual NovaObject items into a container object of compute node metrics? Is there some reason that we can't just coerce all of these to a dict-of-strings or dict-of-known-primitive-types to save all of this complication? I seem to recall the proposal that led us down this road being "store/communicate arbitrary JSON blobs", but certainly there is a happy medium? Given that the nova meetup is next week, perhaps that would be a good time to actually figure out a path forward? --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] massive number of new errors in logs with oslo.messaging
> And the error messages, which look like this:
>
> Returning exception Unexpected task state: expecting [u'scheduling',
> None] but the actual state is deleting to caller
>
> don't make sense -- at least in the English language.
It's missing some grouping operator to help with order of operations.
What it's saying is:
(I am) Returning exception:
{
Unexpected task state: expecting [u'scheduling', None] but the actual
state is deleting
}
...to the caller.
The inner exception is a thing and the outer pieces are a thing. The
inner means that some instance update was attempted, but should be
aborted if the instance state is not what we think it is.
--Dan
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] massive number of new errors in logs with oslo.messaging
> The inner exception is a thing and the outer pieces are a thing. The > inner means that some instance update was attempted, but should be > aborted if the instance state is not what we think it is. And looking a little closer, I think it means that the messaging.expected_exceptions decorator isn't doing the right thing in that case. That sort of exception should be passed back to the client without verbose logging since it's an "okay thing to happen" in that case. It was working before the oslo.messaging change, but I think something is making it not properly exclude the UnexpectedTaskState exception... https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L113 --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] ceilometer unit tests broke because of a nova patch
>> Whats the underlying problem here? nova notifications aren't >> versioned? Nova should try to support ceilometer's use case so >> it sounds like there is may be a nova issue in here as well. > > Oh you're far from it. > > Long story short, the problem is that when an instance is detroyed, > we need to poll one last time for its CPU, IO, etc statistics to > send them to Ceilometer. The only way we found to do that in Nova > is to plug a special notification driver that intercepts the > deletion notification in Nova, run the pollsters, and then returns > to Nova execution. Doesn't this just mean that Nova needs to do an extra poll and send an extra notification? Using a special notification driver, catching the delete notification, and polling one last time seems extremely fragile to me. It makes assumptions about the order things happen internally to nova, right? What can be done to make Ceilometer less of a bolt-on? That seems like the thing worth spending time discussing... --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ceilometer] ceilometer unit tests broke because of a nova patch
> We don't have to add a new notification, but we have to add some > new datas in the nova notifications. At least for the delete > instance notification to remove the ceilometer nova notifier. > > A while ago, I have registered a blueprint that explains which > datas are missing in the current nova notifications: > > https://blueprints.launchpad.net/nova/+spec/usage-data-in-notification > > https://wiki.openstack.org/wiki/Ceilometer/blueprints/remove-ceilometer-nova-notifier This seems like a much better way to do this. I'm not opposed to a nova plugin, but if it's something that lives outside the nova tree, I think there's going to be a problem of constantly chasing internal API changes. IMHO, a plugin should live (and be tested) in the nova tree and provide/consume a stableish API to/from Ceilometer. So, it seems like we've got the following options: 1. Provide the required additional data in our notifications to avoid the need for a plugin to hook into nova internals. 2. Continue to use a plugin in nova to scrape the additional data needed during certain events, but hopefully in a way that ties the plugin to the internal APIs in a maintainable way. Is that right? Personally, I think #1 is far superior to #2. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] [TripleO] Goal setting // progress towards integration
> I would also like to see CI (either third party or in the gate) for > the nova driver before merging it. There's a chicken and egg problem > here if its in the gate, but I'd like to see it at least proposed as a > review. Yeah, I think that the existing nova-baremetal driver is kinda frozen in a pre-deprecation state right now, which gives it a special pass on the CI requirement. To me, I think it makes sense to avoid ripping it out since it's already on ice. However, for the Ironic driver, I would definitely rather see real CI up _and_ working before we merge it. I think that probably means it will be a post-icehouse thing at this point, unless that effort is farther along than I think. At the Nova meetup this week, we had a serious discussion about ripping out major drivers that might not make the deadline. I don't think it makes sense to rip those out and merge another without meeting the requirement. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Db interaction: conductor vs objects vs apis
> - "fat model" approach - put the db interaction in objects If it's just DB interaction, then yes, in the object for sure. > - put the db interactions in the conductor itself There is a reasonable separation between using conductor for mechanics (i.e. API deferring a long-running activity to conductor) and using it for (new) DB proxying. At this point. the former is okay and the latter is not, IMHO. That said, any complex data structure going over RPC should be an object as well, so that we have version tracking on the structure itself. We recently had a case where we broke upgrades because the *format* of an RPC method argument was changed, that argument was not an object, and we ended up with some very ugly failures as a result :) --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Future of the Nova API
> - We want to make backwards incompatible changes to the API > and whether we do it in-place with V2 or by releasing V3 > we'll have some form of dual API support burden. IMHO, the cost of maintaining both APIs (which are largely duplicated) for almost any amount of time outweighs the cost of localized changes. > - Not making backwards incompatible changes means: > - retaining an inconsistent API > - not being able to fix numerous input validation issues > - have to forever proxy for glance/cinder/neutron with all > the problems that entails. The neutron stickiness aside, I don't see a problem leaving the proxying in place for the foreseeable future. I think that it's reasonable to mark them as deprecated, encourage people not to use them, and maybe even (with a core api version to mark the change) say that they're not supported anymore. I also think that breaking our users because we decided to split A into B and C on the backend kind of sucks. I imagine that continuing to do that at the API layer (when we're clearly going to keep doing it on the backend) is going to earn us a bit of a reputation. > - Backporting V3 infrastructure changes to V2 would be a > considerable amount of programmer/review time While acknowledging that you (and others) have done that for v3 already, I have to think that such an effort is much less costly than maintaining two complete overlapping pieces of API code. > - The V3 API as-is has: > - lower maintenance > - is easier to understand and use (consistent). > - Much better input validation which is baked-in (json-schema) > rather than ad-hoc and incomplete. In case it's not clear, there is no question that the implementation of v3 is technically superior in my mind. So, thanks for that :) IMHO, it is also: - twice the code - different enough to be annoying to convert existing clients to use - not currently different enough to justify the pain > - Proposed way forward: > - Release the V3 API in Juno with nova-network and tasks support > - Feature freeze the V2 API when the V3 API is released > - Set the timeline for deprecation of V2 so users have a lot > of warning This feels a lot like holding our users hostage in order to get them to move. We're basically saying "We tweaked a few things, fixed some spelling errors, and changed some date stamp formats. You will have to port your client, or no new features for you!" That's obviously a little hyperbolic, but I think that deployers of APIv2 would probably feel like that's the story they have to give to their users. > Firstly I'd like to step back a bit and ask the question whether we > ever want to fix up the various problems with the V2 API that involve > backwards incompatible changes. These range from inconsistent naming > through the urls and data expected and returned, to poor and > inconsistent input validation and removal of all the proxying Nova > does to cinder, glance and neutron. I believe the answer to this is > yes - inconsistencies in the API make it harder to use (eg do I have a > instance or a server, and a project or a tenant just to name a > couple) and more error prone and proxying has caused several painful to > fix issues for us. I naively think that we could figure out a way to move things forward without having to completely break older clients. It's clear that other services (with much larger and more widely-used APIs) are able to do it. That said, I think the corollary to the above question is: do we ever want to knowingly break an existing client for either of: 1. arbitrary user-invisible backend changes in implementation or service organization 2. purely cosmetic aspects like spelling, naming, etc IMHO, we should do whatever we can to avoid breaking them except for the most extreme cases. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Future of the Nova API
> The API layer is a actually quite a very thin layer on top of the > rest of Nova. Most of the logic in the API code is really just > checking incoming data, calling the underlying nova logic and then > massaging what is returned in the correct format. So as soon as you > change the format the cost of localised changes is pretty much the > same as duplicating the APIs. In fact I'd argue in many cases its > more because in terms of code readability its a lot worse and > techniques like using decorators for jsonschema for input validation > are a lot harder to implement. And unit and tempest tests still need > to be duplicated. Making any change to the backend is double the effort with the two trees as it would be with one API. I agree that changing/augmenting the format of a call means some localized "if this then that" code, but that's minor compared to what it takes to do things on the backend, IMHO. > I don't understand why this is also not seen as forcing people off > V2 to V3 which is being given as a reason for not being able to set > a reasonable deprecation time for V2. This will require major changes > for people using the V2 API to change how they use it. Well, deprecating them doesn't require the change. Removing them does. I think we can probably keep the proxying in a deprecated form for a very long time, hopefully encouraging new users to "do it right" without breaking existing users who don't care. Hopefully losing out on the functionality they miss by not talking directly to Neutron (for example) will be a good carrot to avoid using the proxy APIs. > In all the discussions we've (as in the Nova group) had over the API > there has been a pretty clear consensus that proxying is quite > suboptimal (there are caching issues etc) and the long term goal is > to remove it from Nova. Why the change now? This is just MHO, of course. I don't think I've been party to those conversations. I understand why the proxying is bad, but that's a different issue from whether we drop it and break our users. > I strongly disagree here. I think you're overestimating the amount of > maintenance effort this involves and significantly underestimating > how much effort and review time a backport is going to take. Fair enough. I'm going from my experience over the last few cycles of changing how the API communicates with the backend. This is something we'll have to continue to evolve over time, and right now it Sucks Big Time(tm) :) >> - twice the code > For starters, It's not twice the code because we don't do things > like proxying and because we are able to logically separate out > input validation jsonschema. You're right, I should have said "twice the code for changes between the API and the backend". > Eg just one simple example, but how many people new to the API get > confused about what they are meant to send when it asks for > instance_uuid when they've never received one - is at server uuid - > and if so what's the difference? Do I have to do some sort of > conversion? Similar issues around project and tenant. And when > writing code they have to remember for this part of the API they pass > it as server_uuid, in another instance_uuid, or maybe its just id? > All of these looked at individually may look like small costs or > barriers to using the API but they all add up and they end up being > imposed over a lot of people. Yup, it's ugly, no doubt. I think that particular situation is probably (hopefully?) covered up by the various client libraries (and/or docs) that we have. If not, I think it's probably something we can improve from an experience perspective on that end. But yeah, I know the public API docs would still have that ambiguity. > And how is say removing proxying or making *any* backwards > incompatible change any different? It's not. That's why I said "maybe remove it some day" :) > Well if you never deprecate the only way to do it is to maintain the > old API forever (including test). And just take the hit on all that > involves. Sure. Hopefully people that actually deploy and support our API will chime in here about whether they think that effort is worth not telling their users to totally rewrite their clients. If we keep v2 and v3, I think we start in icehouse with a very large surface, which will increase over time. If we don't, then we start with v2 and end up with only the delta over time. > What about the tasks API? We that discussed at the mid cycle summit > and decided that the alternative backwards compatible way of doing it > was too ugly and we didn't want to do that. But that's exactly what > we'd be doing if we implemented them in the v2 API and it would be a > feature which ends up looking bolted because of the otherwise > significant non backwards compatible API changes we can't do. If we version the core API and let the client declare the version it speaks in a header, we could iterate on that interface right? If they're version =X return the task. We
Re: [openstack-dev] [nova] Future of the Nova API
> So the deprecation message in the patch says:
>
>LOG.warning(_('XML support has been deprecated and will be
> removed in the Juno release.'))
>
> perhaps that should be changed :-)
Maybe, but I think we can continue with the plan to rip it out in Juno.
In the past when we've asked, there has been an overwhelming amount of
"meh" regarding removing it. We've considered it several times, and
we've now drawn a line in the sand. Personally, I'm fine with doing
this, and I think the support from the core team that +A'd the heck out
of it probably means that there is wide support for it.
> In terms of user facing changes we can't do a whole lot - because they
> are inherently changes in how users communicate with API. And not just
> in terms of parameter names, but where and how they access the
> functionality (eg url paths change). In the past we've made
> mistakes as to where or how functionality should appear, leading to
> weird inconsistencies.
>
> So either we can't fix them or in cases where we preserve backwards
> compatibility we end up with dual maintenance cost (our test load
> still doubles), but often having to be implemented in a way which costs
> us more in terms of readability because the code becomes spaghetti.
I think it can be done without it turning into a mess. Non-trivial for
sure, but not impossible. And if not, I still expect most users would
prefer stability over purity.
--Dan
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Future of the Nova API
> onSharedStorage = True
> on_shared_storage = False
This is a good example. I'm not sure it's worth breaking users _or_
introducing a new microversion for something like this. This is
definitely what I would call a "purity" concern as opposed to "usability".
Things like the twenty different datetime formats we expose _do_ seem
worth the change to me as it requires the client to parse a bunch of
different formats depending on the situation. However, we could solve
that with very little code by just exposing all the datetimes again in
proper format:
{
"updated_at": "%(random_weirdo)s",
"updated_at_iso": "%(isotime)s",
}
Doing the above is backwards compatible and doesn't create code
organizations based on any sort of pasta metaphor. If we introduce a
discoverable version tag so the client knows if they will be available,
I think we're good.
URL inconsistencies seem "not worth the trouble" and I tend to think
that the "server" vs. "instance" distinction probably isn't either, but
I guess I'm willing to consider it.
Personally, I would rather do what we can/need in order to provide
features in a compatible way, fix real functional issues (like the
datetimes), and not ask users to port to a new API to clear up a bunch
of CamelCase inconsistencies. Just MHO.
--Dan
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Future of the Nova API
> I thought micro versioning was so we could make backwards compatible changes. > If we make breaking changes we need to support the old and the new for > a little while. Adding a new field alongside an old one in a structure that we return is not a breaking change to me, IMHO. We can clean up the datestamp formats that we return in that way: return both. For datestamps that the client passes (are there any of these?) we don't have to honor both and do conflict resolution if they disagree, we just honor the new one, clearly. > For return values: > * get new clients to send Accepts headers, to version the response > * this amounts to the "major" version > * for those request the new format, they get the new format > * for those getting the old format, they get the old format Yes, I think this is quite reasonable. I honestly think that in most cases we can avoid *ever* breaking the original return format without the code turning into a mess, and I think that's what we should shoot for. > We could port the V2 classes over to the V3 code, to get the code benefits. Yes. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Future of the Nova API
> I think we need to find an alternative way to support the new and old > formats, like Accepts Headers, and retro-fitting a version to > extensions so we can easily advertise new attributes, to those parsers > that will break when they encounter those kinds of things. Agreed. > Now I am tempted to say we morph the V3 code to also produce the V2 > responses. And change the v3 API, so thats easier to do, and easier > for clients to move (like don't change URLs unless we really have to). > I know the risk for screwing that up is enormous, but maybe that makes > the most sense? It seems far easier to port the architectural awesomeness of v3 to v2 in terms of code organization (which can be done without altering the format), and then start extending v2 to support new formats that we want. Trying to take a thing with a thousand small changes and add support to optionally not send those small changes seems harder to me than adding the important ones into v2. It will also help us revisit what changes we want to make, and hopefully we would reconsider taking on the versioning pain of a bunch of CamelCase changes :) --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Future of the Nova API
> +1, seems would could explore for another cycle just to find out that > backporting everything to V2 isn't going to be what we want, and now > we've just wasted more time. > If we say it's just deprecated and frozen against new features, then > it's maintenance is just limited to bug fixes right? No, any time we make a change to how the api communicates with compute, conductor, or the scheduler, both copies of the API code have to be changed. If we never get rid of v2 (and I don't think we have a good reason to right now) then we're doing that *forever*. I do not want to sign up for that. I'm really curious what deployers like RAX, HP Cloud, etc think about freezing V2 to features and having to deploying V3 to get them. Does RAX expose V3 right now? Also curious if RAX/HP/etc see the V3 value statement when compared to what it will mean for their users. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Future of the Nova API
> Yeah, so objects is the big one here. Objects, and everything else. With no-db-compute we did it for a couple cycles, then objects, next it will be retooling flows to conductor, then dealing with tasks, talking to gantt, etc. It's not going to end any time soon. > So what kind of reaction are the Keystone people getting to that? Do > they plan on removing their V2 API at some point? Or just maintain it > with bug fixes forever? Yep, that would be good data. We also need to factor in the relative deployment scale of nova installations vs. keystone installations in the world (AFAIK, RAX doesn't use keystone for example). --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Future of the Nova API
> This would reduce the amount of duplication which is required (I doubt > we could remove all duplication though) and whether its worth it for say > the rescue example is debatable. But for those cases you'd only need to make > the modification in one file. Don't forget the cases where the call chain changes -- where we end up calling into conductor instead of compute, or changing how we fetch complicated information (like BDMs) that we end up needing to send to something complicated like the run_instance call. As we try to evolve compute/api.py to do different things, the changes we have to drive into the api/openstack/ code will continue. However, remember I've maintained that I think we can unify a lot of the work to make using almost the same code work for multiple ways of accessing the API. I think it's much better to structure it as multiple views into the same data. My concern with the v2 -> v3 approach, is that I think instead of explicitly duplicating 100% of everything and then looking for ways to squash the two pieces back together, we should be making calculated changes and supporting just the delta. I think that if we did that, we'd avoid making simple naming and CamelCase changes as I'm sure we'll try to avoid starting from v3. Why not start with v2? We've already established that we can get a version from the client on an incoming request, so why wouldn't we start with v2 and evolve the important pieces instead of explicitly making the decision to break everyone by moving to v3 and *then* start with that practice? --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Future of the Nova API
> So I was thinking about this and Ken'ichi has basically said pretty > much the same thing in his reply to this thread. I don't think it > makes client moves any easier though - this is all about lowering our > maintenance costs. So, in the other fork of this thread, I think you said we can't improve v2 because we're concerned about its incredible fragility. The above statement seems to imply that we can totally rewrite it as decorators on top of v3? I don't get that :) --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Future of the Nova API
> We may need to differentiate between breaking the API and breaking > corner-case behavior. Totally agreed. > In one case you force everyone in the ecosystem to > adapt (the libraries, the end user code). In the other you only > (potentially) affect those that were not following the API correctly. The problem is that the API spec is too loose right now, which is definitely a problem. However, I think I'd much rather tighten things down and deal with the potential fallout of someone's client breaking and saying "oh, I thought 'red' was a valid uuid" than whole rewrites. > We could make a V3 that doesn't break the API, only breaks behavior in > error cases due to its stronger input validation. A V3 that shouldn't > break code that was following the API, nor require heavy library > changes. It's still a major API bump because behavior may change and > some end users will be screwed in the process, but damage is more > limited, so V2 could go away after a shorter deprecation period. What's the difference between saying "/v2 will return a 404 after K" and saying "If your client doesn't declare support for revision 2 of these calls" we'll return a 405, 406, 410, etc? Actually, 412 seems to be exactly this case. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Future of the Nova API
> Users actually care about the latter. If the API accepts 'red' as a > valid UUID then that is part of the implicit contract. Yeah, but right now, many of those things are "would fail on postgres and succeed on mysql" (not uuids necessarily, but others). Since we can't honor them in all cases, I don't see the changes (versioned, for warning) as quite so bad as the alternative. My point was that unless we decide to stop working on the core, we're going to have minor changes like that regardless. If we decide to freeze v2 and move to v3, then we have to basically emulate those bugs _and_ maintain v3. If we version v2 and move forward, we;re in better shape, IMHO. > If we do go down this route of changing the v2 API like this, it would > be nice to bump the minor version as well, so when a user has to > distinguish between the two versions they don't have to say Havana V2 > vs Icehouse V2, they can just say v2.1. Yes, I'm totally all for introducing versioning into v2. Definitely. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Future of the Nova API
> So if we make backwards incompatible changes we really need a major > version bump. Minor versions don't cut it, because the expectation is > you have API stability within a major version. I disagree. If the client declares support for it, I think we can very reasonably return new stuff. If we take what we have today in v2 and call that 2, then we could make novaclient send this header: Accept: application/json;version=2 Then, we have a common method in the api code called get_client_version(req), which returns 2 if it's missing, otherwise it returns the version the client declared. When we want to return something new, we do so only if get_client_version(req) >= 3. I think that if we did that, we could support tasks in v2, properly, today. If we _have_ to deprecate something (with the same care, concern, and delay as we've previously proposed deprecating v2 in general) we can return a proper HTTP status code if the client declares a crufty version (or no version at all). The nice thing about that, is that we don't have to deprecate all of v2 just to deprecate something we don't feel we can support anymore for some good technical reason. When we come along and decide that the API should really be organized totally differently than it is today, such that a total rewrite makes sense, we can do that and call it v3. However, for the level of change that we've currently done in v3, I think the above would work just fine and avoid the churn. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Future of the Nova API
> So I think once we start returning different response codes, or > completely different structures (such as the tasks change will be), it > doesn't matter if we make the change in effect by invoking /v2 prefix > or /v3 prefix or we look for a header. Its a major api revision. I > don't think we should pretend otherwise. I think you're missing my point. The current /v2 and /v3 versions are about 10,000 revisions apart. I'm saying we have the client declare support for a new version every time we need to add something new, not to say that they support what we currently have as /v3. So it would be something like: version=v2: Current thing version=v3: added simple task return for server create version=v4: added the event extension version=v5: added a new event for cinder to the event extension version=v6: changed 200 to 202 for three volumes calls ...etc Obviously using a header instead of a url doesn't help if the gap between v2 and v3 is what it is today. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Future of the Nova API
> Sure, but that's still functionally equivalent to using the /v2 prefix. > So we could chuck the current /v3 code and do: > > /v2: Current thing > /v3: invalid, not supported > /v4: added simple task return for server create > /v5: added the event extension > /v6: added a new event for cinder to the event extension > > and it would be equivalent. Yep, sure. This seems more likely to confuse people or clients to me, but if that's how we decided to do it, then that's fine. The approach to _what_ we version is my concern. > And arguably, anything that is a pure "add" could get away with either a > minor version or not touching the version at all. Only "remove" or > "modify" should have the potential to break a properly-written application. Totally agree! --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Future of the Nova API
> I do think client headers instead of urls have some pragmatic > approach here that is very attractive. Will definitely need a good > chunk of plumbing to support that in a sane way in the tree that > keeps the overhead from a review perspective low. Aside from some helper functions to make this consistent, and some things to say "here's the code I want to return, but compat clients need /this/ code", I think this actually gets us most of the way there: http://paste.openstack.org/show/70310/ (super hacky not-working fast-as-I-could-type prototype, of course) Some care to the protocol we use, versioning rules, etc is definitely required. But plumbing-wise, I think it's actually not so bad. With the above, I think we could stop doing new extensions for every tweak right now. Of course, cleaning up the internals of v2 to be more like v3 now is a separate effort, which is non-trivial and important. However, v3 started as a copy of v2, so we should know exactly what that takes. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Future of the Nova API
> So whilst we still have extensions (and that's a separate debate) we > need versioning on a per extension basis. Otherwise people are forced > to upgrade their extensions in lockstep with each other. I think that some people would argue that requiring the extensions to go together linearly is a good thing, from the point of view of a consistent API. I'm not sure how I feel about that, actually, but I think that's a different discussion. However, what I think I really want, which I mentioned in IRC after I sent this was: using something like servers:version=2. That could be namespaced by extension, or we could define boxes of functionality that go together, like core:version=1, volume-stuff:version=1, etc. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] What is the currently accepted way to do plugins
> In a chat with Dan Smith on IRC, he was suggesting that the important > thing was not to use class paths in the config file. I can see that > internal implementation should not be exposed in the config files - > that way the implementation can change without impacting the nova > users/operators. > > Sandy, I'm not sure I really get the security argument. Python > provides every means possible to inject code, not sure plugins are so > different. Certainly agree on choosing which plugins you want to use > though. Yeah, so I don't think there's any security reason why one is better than the other. I think that we've decided that providing a class path is ugly, and I agree, especially if we have entry points at our disposal. Now, the actual concern is not related to any of that, but about whether we're going to open this up as a new thing we support. In general, my reaction to adding new APIs people expect to be stable is "no". However, I understand why things like the resource reporting and even my events mechanism are very useful for deployers to do some plumbing and monitoring of their environment -- things that don't belong upstream anyway. So I'm conflicted. I think that for these two cases, as long as we can say that it's not a stable interface, I think it's probably okay. However, things like we've had in the past, where we provide a clear plug point for something like "Compute manager API class" are clearly off the table, IMHO. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] What is the currently accepted way to do plugins
> How about using 'unstable' as a component of the entrypoint group? > E.g., "nova.unstable.events"… Well, this is a pretty heavy way to ensure that the admin gets the picture, but maybe appropriate :) What I don't think we want is the in-tree plugins having to hook into something called "unstable". But, if we handle those one way and then clearly find and loop through any "unstable $things" then maybe that's a happy medium. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Concrete Proposal for Keeping V2 API
> What I'd like to do next is work through a new proposal that includes > keeping both v2 and v3, but with a new added focus of minimizing the > cost. This should include a path away from the dual code bases and to > something like the "v2.1" proposal. I think that the most we can hope for is consensus on _something_. So, the thing that I'm hoping would mostly satisfy the largest number of people is: - Leaving v2 and v3 as they are today in the tree, and with v3 still marked experimental for the moment - We start on a v2 proxy to v3, with the first goal of fully implementing the v2 API on top of v3, as judged by tempest - We define the criteria for removing the current v2 code and marking the v3 code supported as: - The v2 proxy passes tempest - The v2 proxy has sign-off from some major deployers as something they would be comfortable using in place of the existing v2 code - The v2 proxy seems to us to be lower maintenance and otherwise preferable to either keeping both, breaking all our users, deleting v3 entirely, etc - We keep this until we either come up with a proxy that works, or decide that it's not worth the cost, etc. I think the list of benefits here are: - Gives the v3 code a chance to address some of the things we have identified as lacking in both trees - Gives us a chance to determine if the proxy approach is reasonable or a nightmare - Gives a clear go/no-go line in the sand that we can ask deployers to critique or approve It doesn't address all of my concerns, but at the risk of just having the whole community split over this discussion, I think this is probably (hopefully?) something we can all get behind. Thoughts? --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db][nova] NovaObject.save() needs its own DB transaction
> However, it presents a problem when we consider NovaObjects, and > dependencies between them. I disagree with this assertion, because: > For example, take Instance.save(). An > Instance has relationships with several other object types, one of which > is InstanceInfoCache. Consider the following code, which is amongst what > happens in spawn(): > > instance = Instance.get_by_uuid(uuid) > instance.vm_state = vm_states.ACTIVE > instance.info_cache.network_info = new_nw_info > instance.save() > > instance.save() does (simplified): > self.info_cache.save() > self._db_save() > > Both of these saves happen in separate db transactions. This has always been two DB calls, and for a while recently, it was two RPCs, each of which did one call. > This has at least 2 undesirable effects: > > 1. A failure can result in an inconsistent database. i.e. info_cache > having been persisted, but instance.vm_state not having been persisted. > > 2. Even in the absence of a failure, an external reader can see the new > info_cache but the old instance. I think you might want to pick a different example. We update the info_cache all the time asynchronously, due to "time has passed" and other non-user-visible reasons. > New features continue to add to the problem, > including numa topology and pci requests. NUMA and PCI information are now created atomically with the instance (or at least, passed to SQLA in a way I expect does the insert as a single transaction). We don't yet do that in save(), I think because we didn't actually change this information after creation until recently. Definitely agree that we should not save the PCI part without the base instance part. > I don't think we can reasonably remove the cascading save() above due to > the deliberate design of objects. Objects don't correspond directly to > their datamodels, so save() does more work than just calling out to the > DB. We need a way to allow cascading object saves to happen within a > single DB transaction. This will mean: > > 1. A change will be persisted either entirely or not at all in the event > of a failure. > > 2. A reader will see either the whole change or none of it. This is definitely what we should strive for in cases where the updates are related, but as I said above, for things (like info cache) where it doesn't matter, we should be fine. > Note that there is this recently approved oslo.db spec to make > transactions more manageable: > > https://review.openstack.org/#/c/125181/11/specs/kilo/make-enginefacade-a-facade.rst,cm > > Again, while this will be a significant benefit to the DB api, it will > not solve the problem of cascading object saves without allowing > transaction management at the level of NovaObject.save(): we need to > allow something to call a db api with an existing session, and we need > to allow something to pass an existing db transaction to NovaObject.save(). I don't agree that we need to be concerned about this at the NovaObject.save() level. I do agree that Instance.save() needs to have a relationship to its sub-objects that facilitates atomicity (where appropriate), and that such a pattern can be used for other such hierarchies. > An obvious precursor to that is removing N309 from hacking, which > specifically tests for db apis which accept a session argument. We then > need to consider how NovaObject.save() should manage and propagate db > transactions. Right, so I believe that we had more consistent handling of transactions in the past. We had a mechanism for passing around the session between chained db/api methods to ensure they happened atomically. I think Boris led the charge to eliminate that, culminating with the hacking rule you mentioned. Maybe getting back to the justification for removing that facility would help us understand the challenges we face going forward? > [1] At a slight tangent, this looks like an artifact of some premature > generalisation a few years ago. It seems unlikely that anybody is going > to rewrite the db api using an ORM other than sqlalchemy, so we should > probably ditch it and promote it to db/api.py. We've had a few people ask about it, in terms of rewriting some or all of our DB API to talk to a totally non-SQL backend. Further, AFAIK, RAX rewrites a few of the DB API calls to use raw SQL queries for performance (or did, at one point). I'm quite happy to have the implementation of Instance.save() make use of primitives to ensure atomicity where appropriate. I don't think that's something that needs or deserves generalization at this point, and I'm not convinced it needs to be in the save method itself. Right now we update several things atomically by passing something to db/api that gets turned into properly-related SQLA objects. I think we could do the same for any that we're currently cascading separately, even if the db/api update method uses a transaction to ensure safety. --Dan signature.asc Description: OpenPGP digital signature ___
Re: [openstack-dev] [oslo.db][nova] NovaObject.save() needs its own DB transaction
> * Flavor.save() makes an unbounded number of db calls in separate > transactions. This is actually part of the design of the original flavors public API. Since we can add and remove projects/specs individually, we avoid ending up with just one or the other group of values, for competing requests. But as I said before, your point is entirely valid for the cases where it's relevant. > * Instance.save() cascades saves to security groups, each of which is > saved in a separate transaction. > > We can push these into the db layer, but it is my understanding that > NovaObjects are supposed to manage their own mapping of internal state > -> db representation. If we push this into the db api, we're violating > the separation of concerns. If we're going to do this, we'd better > understand, and be able to articulate, *why* we don't want to allow > NovaObjects to manage a db transaction when required. The pattern I > outlined below is very simple. The DB API isn't a stable interface, and exists purely to do what we need it to do. So, if pushing the above guarantees into the DB API works for the moment, we can do that. If we needed to change something about it in the future, we could. If you look at it right now, it has some very (very) specific queries and updates that serve very directed purposes. I don't see pushing atomicity expectations into the DB API as a problem. > I'm proposing a pattern which is always safe and is simple to reason > about. I would implement it everywhere. I don't think there are any > downsides. Cool, sounds like a spec. I'd say propose it and we'll see where it goes in review. Thanks! --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] deleting the pylint test job
> However, because of nova objects pylint is progressively less and less > useful. So the fact that no one else looked at it means that people > didn't seem to care that it was provably broken. I think it's better > that we just delete the jobs and save a node on every nova patch instead. Agreed. > Project Config Proposed here - https://review.openstack.org/#/c/136846/ +1'd > If you -1 that you own fixing it, and making nova objects patches > sensible in pylint. Ooh, wait, maybe I want to see someone do that second part :) --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Versioned objects cross project sessions next steps
> 3. vish brought up one draw back of versioned objects: the difficulty in > cherry picking commits for stable branches - Is this a show stopper?. After some discussion with some of the interested parties, we're planning to add a third .z element to the version numbers and use that to handle backports in the same way that we do for RPC: https://review.openstack.org/#/c/134623/ > Next steps: > - Jay suggested making a second spec that would lay out what it would > look like if we used google protocol buffers. > - Dan: do you need some help in making this happen, do we need some > volunteers? I'm not planning to look into this, especially since we discussed it a couple years ago when deciding to do what we're currently doing. If someone else does, creates a thing that is demonstrably more useful than what we have, and provides a migration plan, then cool. Otherwise, I'm not really planning to stop what I'm doing at the moment. > - Are there any other concrete things we can do to get this usable by > other projects in a timely manner? To be honest, since the summit, I've not done anything with the current oslo spec, given the potential for doing something different that was raised. I know that cinder folks (at least) are planning to start copying code into their tree to get moving. I think we need a decision to either (a) dump what we've got into the proposed library (or incubator) and plan to move forward incrementally or (b) each continue doing our own thing(s) in our own trees while we wait for someone to create something based on GPB that does what we want. --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Spring cleaning nova-core
> I'm going to be honest and say I'm confused here. > > We've always said we expect cores to maintain an average of two > reviews per day. That's not new, nor a rule created by me. Padraig is > a great guy, but has been working on other things -- he's done 60 > reviews in the last 60 days -- which is about half of what we expect > from a core. > > Are we talking about removing the two reviews a day requirement? Please, no. A small set of nova-core does most of the reviews right now anyway. Honestly, I feel like two a day is a *really* low bar, especially given how much time I put into it. We need to be doing more reviews all around, which to me means a *higher* expectation at the low end, if anything. --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Spring cleaning nova-core
> The argument boils down to there is a communications cost to adding > someone to core, and therefore there is a maximum size before the > communications burden becomes to great. I'm definitely of the mindset that the core team is something that has a maximum effective size. Nova is complicated and always changing; keeping everyone on top of current development themes is difficult. Just last week, we merged a patch that bumped the version of an RPC API without making the manager tolerant of the previous version. That's a theme we've had for a while, and yet it was still acked by two cores. A major complaint I hear a lot is "one core told me to do X and then another core told me to do !X". Obviously this will always happen, but I do think that the larger and more disconnected the core team becomes, the more often this will occur. If all the cores reviewed at the rate of the top five and we still had a throughput problem, then evaluating the optimal size would be a thing we'd need to do. However, even at the current size, we have (IMHO) communication problems, mostly uninvolved cores, and patches going in that break versioning rules. Making the team arbitrarily larger doesn't seem like a good idea to me. > I will say that I am disappointed that we have cores who don't > regularly attend our IRC meetings. That makes the communication much > more complicated. Agreed. We alternate the meeting times such that this shouldn't be hard, IMHO. --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [tc] [PTL] Cascading vs. Cells – summit recap and move forward
> [joehuang] Could you pls. make it more clear for the deployment mode > of cells when used for globally distributed DCs with single API. Do > you mean cinder/neutron/glance/ceilometer will be shared by all > cells, and use RPC for inter-dc communication, and only support one > vendor's OpenStack distribution? How to do the cross data center > integration and troubleshooting with RPC if the > driver/agent/backend(storage/network/sever) from different vendor. Correct, cells only applies to single-vendor distributed deployments. In both its current and future forms, it uses private APIs for communication between the components, and thus isn't suited for a multi-vendor environment. Just MHO, but building functionality into existing or new components to allow deployments from multiple vendors to appear as a single API endpoint isn't something I have much interest in. --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
>> 2. There's no way to add an existing server to this "group". > > In the original API there was a way to add existing servers to the > group. This didn't make it into the code that was submitted. It is > however supported by the instance group db API in nova. > >> 3. There's no way to remove members from the group > > In the original API there was a way to remove members from the group. > This didn't make it into the code that was submitted. Well, it didn't make it in because it was broken. If you add an instance to a group after it's running, a migration may need to take place in order to keep the semantics of the group. That means that for a while the policy will be being violated, and if we can't migrate the instance somewhere to satisfy the policy then we need to either drop it back out, or be in violation. Either some additional states (such as being queued for inclusion in a group, etc) may be required, or some additional footnotes on what it means to be in a group might have to be made. It was for the above reasons, IIRC, that we decided to leave that bit out since the semantics and consequences clearly hadn't been fully thought-out. Obviously they can be addressed, but I fear the result will be ... ugly. I think there's a definite possibility that leaving out those dynamic functions will look more desirable than an actual implementation. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Trove] Pluggable conductor manager
> I'd like to propose the ability to support a pluggable trove conductor > manager. Currently the trove conductor manager is hard-coded [1][2] and > thus is always 'trove.conductor.manager.Manager'. I'd like to see this > conductor manager class be pluggable like nova does [3]. Note that most of us don't like this and we're generally trying to get rid of these sorts of things. I actually didn't realize that conductor.manager was exposed in the CONF, and was probably just done to mirror other similar settings. Making arbitrary classes pluggable like this without a structured and stable API is really just asking for trouble when people think it's a pluggable interface. So, you might not want to use "because nova does it" as a reason to add it to trove like this :) --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][neutron] VIF event callbacks implementation
> Aside from creating a sort of cyclic dependency between the two, it > is my understanding that Neutron is meant to be a "stand alone" > service capable of being consumed by other compute managers (i.e. > oVirt). This breaks that paradigm. > So my question is: Why use API and not RPC? > > I saw that there is already a notification system in Neutron that > notifies on each port update (among other things) which are > currently consumed by Ceilometer. Why not have Nova use those > notifications to decide that a VIF got plugged correctly, floating > IPs changed, and so on? To your point above, having either service sit on the other's RPC bus ties them together far closer than having them consume each other's REST API, IMHO. Further, Nova's internal RPC mechanics are controlled pretty tightly for upgrade compatibility and I don't think I'd want another service sitting on that bus that we have to worry about when we're coordinating a change across releases (which we do quite often). We consume Neutron's services via the REST API because that's what is exposed externally and guaranteed to be stable -- the same goes for Neutron consuming anything from Nova. > I believe the rationale here was that nova's API interface is only > currently exposed via a rest API over http so leveraging this > existing framework seemed like a good place to do it. In addition, > there didn't seem to be an obvious advantage to using RPC rather > than the rest interface. Lastly, this new interface that nova exposes > is generic and not neutron specific as it can be used for other type > of notifications that things want to send nova. I added Dan Smith to > CC to keep me honest here as I believe this was the rationale. Yeah, we've already got plans in place to get Cinder to use the interface to provide us more detailed information and eliminate some polling. We also have a very purpose-built notification scheme between nova and cinder that facilitates a callback for a very specific scenario. I'd like to get that converted to use this mechanism as well, so that it becomes "the way you tell nova that things it's waiting for have happened." --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][neutron] VIF event callbacks implementation
> I have additional concern that API is something that's user facing > so basically now Nova is exposing some internal synchronization > detail to the outside world. We have lots of admin-only APIs. > Does it make sense that the user would now be able to send messages > to this API? Potentially. Right now you can ask for a refresh of their network info if things are stale, and some RAX people mentioned that they would like to be able to trigger network resets and some other sorts of things through that mechanism. > Not sure why RPC is more coupled than API. Maybe you could explain? Because we change our RPC APIs all the time. If an external component has to consume our RPC messages reliably, that means either we have to make sure that component tracks the changes, or we avoid making them. On the other hand, our REST APIs are specifically designed and carefully monitored to maintain compatibility with external things. > Currently it's a very specific API putting a burden on Neutron to now > know what is VIF and what state is necessary for this VIF to be > working, instead of having these calculations in Nova (which is of > course aware of VIF, and of Port). It's optional, and for better integration with Neutron and Nova. I don't really think that it's as nova-specific as you're implying. IIRC, the neutron side just fires a notification any time a port changes state right now. > I wasn't suggesting touching Nova's RPC but rather utilize the > existing notifications sent from Neutron to achieve the same logic. > So not sure what changes you believe are to be coordinated from > Nova's POV. Neutron consuming Nova's RPC or Nova consuming Neutron's RPC.. either way, it's not how I'd choose to go about it. > It could similarly be a queue with some defined message format. I think that's what we've implemented, no? It's a well-defined format that describes an event. It has nothing nova-specific in it. > We could alternatively provide a "callback" functionality that > allows various clients to receive notifications from Neutron, > specifying an address to send these details to. Sure, if you want to go extend this mechanism to remove the static configuration and allow for dynamic registration of who receives these messages, I would expect that would be fine. We'd need to figure out how a single Nova deployment will manage registering with Neutron in an efficient way, but I'm sure that could be done. > Not sure how you consider this "mechanism" something generic since > it's facilitating only Nova while there might be a number of > different services interested in this information. Now Neutron needs > to be aware of VIF and Nova's expectations of Neutron in regards to > that VIF, which is highly tightly coupled. > > Using a notification scheme where any subscriber can receive the > event from Neutron/Cinder/etc and handle it how it needs instead > would be much more decoupled, IMHO. Well, I'm not sure why Cinder would need to receive notifications from Neutron, but I understand what you mean. Like I said, nothing about how it's currently architected prevents this from happening, AFAIK, other than the fact that it's currently managed by static configuration. If you want to add a subscription interface to register for these so that neutron can supply notifications to multiple entities dynamically, then that seems like a natural next step. I'm sure Cinder would prefer that as well. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Informal meeting before SR-IOV summit presentation
> It would be nice to have an informal discussion / unconference session > before the actual summit session on SR-IOV. During the previous IRC > meeting, we were really close to identifying the different use cases. > There was a dangling discussion on introducing another level of > indirection between the vnic_types exposed via the nova boot API and how > it would be represented internally. It would be ideal to have these 2 > discussions converged before the summit session. What would be the purpose of doing that before the session? IMHO, a large part of being able to solve this problem is getting everyone up to speed on what this means, what the caveats are, and what we're trying to solve. If we do some of that outside the scope of the larger audience, I expect we'll get less interaction (or end up covering it again) in the session. That said, if there's something I'm missing that needs to be resolved ahead of time, then that's fine, but I expect the best plan is to just keep the discussion to the session. Afterwards, additional things can be discussed in a one-off manner, but getting everyone on the same page is largely the point of having a session in the first place IMHO. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Need help with a gnarly Object Version issue
> On a compute manager that is still running the old version of the code > (i.e using the previous object version), if a method that hasn’t yet > been converted to objects gets a dict created from the new version of > the object (e.g. rescue, get_console_output), then object_compat() > decorator will call the _/from_db/_object() method in > objects.Instance. Because this is the old version of the object > code, it expects user_data to be a field in dict, and throws a key error. Yeah, so the versioning rules are that for a minor version, you can only add things to the object, not remove them. > 1) Rather than removing the user_data field from the object just > set it to a null value if its not requested. Objects have a notion of "unset" which is what you'd want here. Fields that are not set can be lazy-loaded when touched, which might be a reasonable way out of the box here if user_data is really only used in one place. It would mean that older clients would lazy-load it when needed, and going forward we'd be specific about asking for it when we want. However, the problem is that instance defines the fields it's willing to lazy-load, and user_data isn't one of them. That'd mean that we need to backport a change to allow it to be lazy-loaded, which means we should probably just backport the thing that requests user_data when needed instead. > 2) Add object versioning in the client side of the RPC layer for > those methods that don’t take objects. I'm not sure what you mean here. > I’m open to other ideas, and general guidance around how deletion of > fields from Objects is meant to be handled ? It's meant to be handled by rev-ing the major version, since removing something isn't a compatible operation. Note that *conductor* has knowledge of the client-side version of an object on which the remotable_classmethod is being called, but that is not exposed to the actual object implementation in any way. We could, perhaps, figure out a sneaky way to expose that, which would let you honor the old behavior if we know the object is old, or the new behavior otherwise. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][QA] Disabling the v3 API tests in the gate
I think it'd be OK to move them to the experimental queue and a periodic nightly job until the v2.1 stuff shakes out. The v3 API is marked experimental right now so it seems fitting that it'd be running tests in the experimental queue until at least the spec is approved and microversioning starts happening in the code base. I think this is reasonable. Continuing to run the full set of tests on every patch for something we never expect to see the light of day (in its current form) seems wasteful to me. Plus, we're going to (presumably) be ramping up tests on v2.1, which means to me that we'll need to clear out some capacity to make room for that. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][QA] Disabling the v3 API tests in the gate
Thats true, though I was suggesting as v2.1microversions rolls out we drop the test out of v3 and move it to v2.1microversions testing, so there's no change in capacity required. Right now we run a full set over /v2 and a full set over /v3. Certainly as we introduce /v2.1 we'll need full coverage on that as well, before we start introducing any microversion-based changes, right? Dropping /v3 should make room for /v2.1, and then go from there when adding new things to /v2.1 via microversions. When we are able to drop /v2 (or rather move /v2.1 to /v2) then we actually gain back the capacity. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] FFE request: clean-up-legacy-block-device-mapping
> Why accept it? > > * It's low risk but needed refactoring that will make the code that has > been a source of occasional bugs. > * It is very low risk internal refactoring that uses code that has been > in tree for some time now (BDM objects). > * It has seen it's fair share of reviews Yeah, this has been conflict-heavy for a long time. If it hadn't been, I think it'd be merged by now. The bulk of this is done, the bits remaining have seen a *lot* of real review. I'm happy to commit to reviewing this, since I've already done so many times :) --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Conductor support for networking in Icehouse
> However, when attempting to boot an instance, the Nova network service > fails to retrieve network information from the controller. Adding the > the database keys resolves the problem. I'm using > the 2014.1~b3-0ubuntu1~cloud0 packages on Ubuntu 12.04. Can you file a bug with details from the logs? --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Conductor support for networking in Icehouse
> https://bugs.launchpad.net/nova/+bug/1290568 Thanks. Note that the objects work doesn't really imply that the service doesn't hit the database. In fact, nova-compute stopped hitting the database before we started on the objects work. Anyway, looks like there are still some direct-to-database things lingering in the linux_net module. I'm not sure those will get resolved before icehouse at this point... --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Conductor support for networking in Icehouse
> Hmm... I guess the blueprint summary led me to believe that nova-network > no longer needs to hit the database. Yeah, using objects doesn't necessarily mean that the rest of the direct database accesses go away. However, I quickly cooked up the rest of what is required to get this done: https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/1290568,n,z Review would be great. The last patch wedges the database like we do in compute to make sure that the tests pass without talking to the database itself. Would be a nice feature for icehouse to say that multihost compute nodes are now db-clean. Thanks! --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] FFE Request: Ephemeral RBD image support
> I'm confused as to why we arrived at the decision to revert the commits > since Jay's patch was accepted. I'd like some details about this > decision, and what new steps we need to take to get this back in for Juno. Jay's fix resolved the immediate problem that was reported by the user. However, after realizing why the bug manifested itself and why it didn't occur during our testing, all of the core members involved recommended a revert as the least-risky course of action at this point. If it took almost no time for that change to break a user that wasn't even using the feature, we're fearful about what may crop up later. We talked with the patch author (zhiyan) in IRC for a while after making the decision to revert about what the path forward for Juno is. The tl;dr as I recall is: 1. Full Glance v2 API support merged 2. Tests in tempest and nova that exercise Glance v2, and the new feature 3. Push the feature patches back in --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] [neutron] Top Gate Race - Bug 1248757 - test_snapshot_pattern fails with paramiko ssh EOFError
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 > Here is the latest marked fail - > http://logs.openstack.org/28/79628/4/check/check-tempest-dsvm-neutron/11f8293/ So, > looking at this a little bit, you can see from the n-cpu log that it is getting failures when talking to neutron. Specifically, from neutronclient: throwing ConnectionFailed : timed out _cs_request ConnectionFailed: Connection to neutron failed: Maximum attempts reached - From a brief look at neutronclient, it looks like it tries several times to send a request, before it falls back to the above error. Given the debug "timed out" log there, I would assume that neutron's API isn't accepting the connection in time. Later in the log, it successfully reaches neutron again, and then falls over again in the same way. This is a parallel job, so load is high, which makes me suspect just a load issue. - From talking to salv-orlando on IRC just now, it sounds like this might just be some lock contention on the Neutron side, which is slowing it down enough to cause failures occasionally. - --Dan -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTIbxoAAoJEBeZxaMESjNVB4IH/0wzaRhW/xkuUbFxNsSbRRt5 8EJdBkDJHfFQW6VQM6GqmvyZOVFkTLOhdMGF1dgWLBTTkGhmOVRiwdkim059sPd4 3EwUH3ZhSQg8n/rSAoS0rb1nFKaCt6D76DNJR5LXBCd89k6d/0q8SAkOgwNg7H82 oS17CjnLYvUfvF0JqSmKNt4ter1zMSXMZXNe8z09mKqZBTC4vNWIskv2yLgUbecv Sb6NVc+HFkCk3t5MlKlM8fnLIoF2b4F0w0rCSJPV9txXWL2ijiaFIncyTYSFSuOp NE1kdEAuZOIUnnZW3udEyb4QQS3HhRbVvRHJbnTAOVLGw5ijp+1V5FoipizA3v0= =lVal -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack vs. SQLA 0.9
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 > Because of where we are in the freeze, I think this should wait > until Juno opens to fix. Icehouse will only be compatible with > SQLA 0.8, which I think is fine. I expect the rest of the issues > can be addressed during Juno 1. Agreed. I think we have some other things to check before we make this move, like how we currently check to see if something is loaded in a SQLA object. ISTR it changed between 0.8 and 0.9 and so likely tests would not fail, but we'd lazy load a bunch of stuff that we didn't intend to. Even without that, I think it's really way too late to make such a switch. - --Dan -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTIeATAAoJEBeZxaMESjNVm8QH/0kjEjXYTHuj3jmuiL0P8ccy KVMaXTL3NmIhaNm1UD/OcWIebgkOKk1BjYAloSRRewulvt0XcK5yr272FLhwuLqr IJBtF15/4pG1b9B8Ol/sOlgAUzcgQ68pu8jIHRd7S5cxjWlEuCP7y2H3pUG38rfq lqUZhrltMpBbcZ0/ewG1BlIgfCWjuv6c/U+S8K2D4zcKkfuOG2hfzPk4ZEy99+wh UYiLfaW+dvku8rN6Lll+6S8VfKM1V+I9hFpKs2exxbX65KJinNgymHxLAj2iQD6Y Ubpk8LO2DElpUm2gULgUqKh0kddmXL7Cuqa2/B5Bm3BAa89CAUVny4ASAWk868c= =Qet4 -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] An analysis of code review in Nova
> Just to answer this point, despite the review latency, please don't be > tempted to think one big change will get in quicker than a series of > little, easy to review, changes. All changes are not equal. A large > change often scares me away to easier to review patches. > > Seems like, for Juno-1, it would be worth cancelling all non-urgent > bug fixes, and doing the refactoring we need. > > I think the aim here should be better (and easier to understand) unit > test coverage. Thats a great way to drive good code structure. Review latency will be directly affected by how good the refactoring changes are staged. If they are small, on-topic and easy to validate, they will go quickly. They should be linearized unless there are some places where multiple sequences of changes make sense (i.e. refactoring a single file that results in no changes required to others). As John says, if it's just a big "change everything" patch, or a ton of smaller ones that don't fit a plan or process, then it will be slow and painful (for everyone). > +1 sounds like a good first step is to move to oslo.vmware I'm not sure whether I think that refactoring spawn would be better done first or second. My gut tells me that doing spawn first would mean that we could more easily validate the oslo refactors because (a) spawn is impossible to follow right now and (b) refactoring it to smaller methods should be fairly easy. The tests for spawn are equally hard to follow and refactoring it first would yield a bunch of more unit-y tests that would help us follow the oslo refactoring. However, it sounds like the osloificastion has maybe already started and that refactoring spawn will have to take a backseat to that. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Backwards incompatible API changes
> If we managed to break Horizon, its likely we've broken (or will break) > other people's scripts or SDKs. > > The patch was merged in October (just after Icehouse opened) and so has > been used in clouds that do CD for quite a while. After some discussion > on IRC I think we'll end up having to leave this backwards incompatible > change in there - given there are most likely users who now rely on > both sets of behaviour there is no good way to get out of this > situation. I've added a note to the Icehouse release notes. So, my first reaction was to keep this change since we've had it in the wild since October, we know that we have CDing public and private clouds, and because it makes us match our own docs. The original decision was made to fix the problem, knowing that it might break a client, even though it wasn't expected that we would in reality. That clearly wasn't quite right. I know that our primary delivery mechanism is releases right now, and so if we decide to revert before this gets into a release, that's cool. However, I think we need to be looking at CD as a very important use-case and I don't want to leave those folks out in the cold. Could we consider a middle road? What if we made the extension silently tolerate an add-myself operation to a flavor, (potentially only) right after create? Yes, that's another change, but it means that old clients (like horizon) will continue to work, and new clients (which expect to automatically get access) will continue to work. We can document in the release notes that we made the change to match our docs, and that anyone that *depends* on the (admittedly weird) behavior of the old broken extension, where a user doesn't retain access to flavors they create, may need to tweak their client to remove themselves after create. I'm really okay with any of the three options, but given that the revert-or-not dichotomy ends up definitely breaking someone, I think the third option is what I'd prefer. Given that we're already looking at a change, lets make it a change that is least offensive. nd...discuss! --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] nova-compute not re-establishing connectivity after controller switchover
> Any ideas on what might be going on would be appreciated. This looks like something that should be filed as a bug. I don't have any ideas off hand, bit I will note that the reconnection logic works fine for us in the upstream upgrade tests. That scenario includes starting up a full stack, then taking down everything except compute and rebuilding a new one on master. After the several minutes it takes to upgrade the controller services, the compute host reconnects and is ready to go before tempest runs. I suspect your case wedged itself somehow other than that, which definitely looks nasty and is worth tracking in a bug. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Rolling upgrades in icehouse
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 > Where can I obtain more information about this feature? - From the blog post that I've yet to write :D > Does above imply that database is upgraded along with control > service update as well? Yes, but only for the services that interact directly with the database. The services that do *not* need to be upgraded atomically with the schema are: nova-compute and nova-network. > One more question - is there an initiative to make icehouse > database schema work with havana based control services ? It depends on what you mean by "control services". For icehouse, the incremental step that we're making is that all the controller services must be upgraded atomically with the database schema. That means api, scheduler, conductor, etc. A havana compute node is sufficiently isolated from the data schema that it will continue to work with an icehouse conductor, allowing you to upgrade compute nodes independently after the controller services are updated. This was just our first step at providing some capability for this. We hope to continue to increase the capabilities (and decrease the amount that must be done atomically) going forward. Hope that helps! - --Dan -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTMIP6AAoJEBeZxaMESjNVSrYH/ixFKup4jRu5THMq5+X9td/S 0lJfTTTBUki2ikmi/mvSPN4Dtfes+SdfkK71EF09B2Za+rq29A4QTLf0RQHSqeFR NpOzTf/baqxUcWroDe/HNLakVd2KnBwh1n3XhEU7Wy+7wzYLl9uLQ/ZguyjazfZv vt7aAs/VtLpYARx4MdK3vopjSuSdVlfHP+0vhPTzoxyDSRzudDh7FRddGLEjjVlX WUHWNePNmdgRzKAFarvyw3qipEuR4kaPqZh3bWr4fIxB6ZQzOA+fa5hkIg3vnD3D 0sLznbZkKevLxhSEX+ml3Gfk2Ax3UVIdAai5JxH+LAka2tiVCrwHQMeKyu7lxFw= =JYp4 -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Nova] PTL Candidacy
Hi all, I would like to run for the OpenStack Compute (Nova) PTL position. Qualifications - I have been working almost exclusively on Nova since mid-2012, and have been on the nova-core team since late 2012. I am also a member of nova-drivers, where I help to target and prioritize blueprints to help shape and focus the direction of the project. I spend a lot of time reviewing code from all over the Nova tree, and am regularly in the top five reviewers: http://russellbryant.net/openstack-stats/nova-reviewers-90.txt and I have sustained that level of activity consistently over time: http://russellbryant.net/openstack-stats/nova-reviewers-365.txt My focus since I started has been on improving Nova's live upgrade capabilities, which started with significant contributions to completion of the no-db-compute blueprint, creation of the conductor service, and most recently the concept and implementation for the NovaObject work. I have been in or at the top of the list of contributors by commit count for a few cycles now: https://github.com/openstack/nova/graphs/contributors?from=2012-07-25&to=2014-03-22&type=c http://www.ohloh.net/p/novacc/contributors http://www.stackalytics.com/?release=icehouse&metric=commits&project_type=openstack&module=&company=&user_id=danms Icehouse Accomplishments - This past cycle, I worked to get us to the point at which we could successfully perform live upgrades for a subset of scenarios from Havana to the Icehouse release. With the help of many folks, this is now a reality, with an upstream gating test to prevent regressions going forward. This is largely possible due to the no-db-compute and NovaObject efforts in the past, which provide us an architecture of version compatibility. Late in the Icehouse cycle, I also worked with developers from Neutron to design and implement a system for coordination between services. This allows us to better integrate Nova's network cache and instance modification tasks with Neutron's processes for increased reliability and performance. Looking forward to Juno Clearly, as Nova continues to grow, the difficult task of scaling the leadership is increasingly important. In the Icehouse cycle, we gained some momentum around this, specifically with involving the entire nova-drivers team in the task of targeting and prioritizing blueprints. The creation of the nova-specs repo will help organize the task of proposing new work, but will add some additional steps to the process. I plan to continue to lean on the drivers team as a whole for keeping up with blueprint-related tasks. Further, we gained blueprint and bug czars in John Garbutt and Tracy Jones, both of which have done an excellent job of wrangling the "paperwork" involved with tracking these items. I think that delegation is extremely important, and something we should attempt to replicate for other topics. The most tactile issue around scaling the project is, of course, the review bandwidth and latency. Russell did a fantastic job of keeping fresh blood on the nova-core team, which both encourages existing members to exhibit a high level of activity, as well as encourages other contributors to aim for the level of activity and review quality needed to be on the core team. I plan to continue to look for ways to increase communication between the core team members, as well as keep it packed with people capable of devoting time to the important task of reviewing code submissions. Another excellent win for the Nova project in Icehouse was the requirement for third-party CI testing of our virtualization drivers. Not only did this significantly improve our quality and regression-spotting abilities on virt drivers, but it also spurred other projects to require the same from their contributing vendors. Going forward, I think we need to increase focus on the success rate for each of these systems which will help us trust them when they report failure. Additionally, I think it is important for us to define a common and minimum set of functions that a virt driver must support. Currently, our hypervisor support matrix shows a widely-varying amount of support for some critical things that a user would expect from a driver integrated in our tree. Thanks for your consideration! --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Decorator behavior
> At run time there are decorators that behave in an unexpected manner.
> For instance, in nova/compute/manager.py when ComputeManager's
> resize_instance method is called, the "migration" positional argument
> is somehow added to kwargs (paired with the "migration" key) and is
> stripped out of the args parameters when the decorators are called.
> However, when this same method is called in
> nova/tests/compute/test_compute_mgr.py, the "migration" positional
> argument remains in args when the decorators are called. In other
> words at run time the decorators see these arguments:
>
> (self, context, [], {'migration': migration, 'image': image,
> 'instance': instance, 'reservations': reservations})
>
> while when running a test case, they see these arguments:
>
> (self, context, [instance, image, reservations, migration,
> instance_type], {})
All RPC-called methods get called with all of their arguments as keyword
arguments. I think this explains the runtime behavior you're seeing.
Tests tend to differ in this regard because test writers are human and
call the methods in the way they normally expect, passing positional
arguments when appropriate.
--Dan
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Dropping or weakening the 'only import modules' style guideline - H302
>> So I'm a soft -1 on dropping it from hacking. Me too. > from testtools import matchers > ... > > Or = matchers.Or > LessThan = matchers.LessThan > ... This is the right way to do it, IMHO, if you have something like matchers.Or that needs to be treated like part of the syntax. Otherwise, module-only imports massively improves the ability to find where something comes from. I also think that machine-enforced style, where appropriate, is very helpful in keeping our code base readable. Repeated patterns and style help a lot, and anything that can be easily machine-enforced is a win in my book. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][neutron] Changing to external events
> If feels like the right sequence is: > > - Deploy the new code in Nova and at the same time set > vif_plugging_is-fatal=False, so that Nova will wait for Neutron, but > will still continue if the event never turns up (which is kind of like > the code was before, but with a wait) Yes, but set the timeout=0. This will cause nova to gracefully accept the events when they start showing up, but the computes will not wait for them. > - Then update Neutron (with presumably some additional config) > so that it starts sending events Right, after that, set fatal=True and timeout=300 (or whatever) and you're good to go. > Is that right, and any reason why the default for vif_plugging_is_fatal > shouldn’t be False insated of True to make this sequence less dependent > on matching config changes ? Yes, because the right approach to a new deployment is to have this enabled. If it was disabled by default, most deployments would never turn it on. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] nova-specs
> There may be some consistency work needed. I spent some time/text in > justification around no security impact in a spec. I was guided > specifically that None was a better statement. I think you're referring to me. What I said was, you went into a lot of depth explaining why there was no security impact for things that I felt were fairly common. That said, even in code reviews, you're going to get opinions from lots of different people, and they're not all going to match up 100%. That's okay, and to be expected, IMHO. It's obviously desirable for us to have a core (and drivers) team that all have roughly the same opinion on what is suitable and not, but expecting total consistency is not realistic or necessary, I think. Remember that just because someone -1s a patch, doesn't mean that every single comment they made was -1 worthy on its own. Often times I will -1 for a spelling mistake and then make a bunch of other purely-opinion comments which don't necessarily need to change. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] nova-specs
> I'm not asking for 100% consistency. I'm just raising it since it seems > to be early in the process change and want to work out these kinds of > things. If it turns out to be an outlier then great. Sure, and the spec reviewers are learning in this process as well. It takes a certain amount of us seeing what other reviewers think is reasonable and not before we establish a baseline. I'm sure that after we've done this for a cycle or two, things will look a lot more consistent :) --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] nova-specs
> Do we really want to -1 for spelling mistake in nova-specs? I do, yes. These documents are intended to be read by deployers and future developers. I think it's really important that they're useful in that regard. > This is really a bad news for non-native speaker like me because I'm > really not sensitive to the spelling even after checking again and > again. Well, I certainly understand it can be frustrating. When I do it, I provide the proper spelling in my comment, and I think I've been generally pretty responsive to re-reviewing things quickly when it was just a spelling error. Usually, however, there is something else to go wrong with the -1 so it needs to be respun anyway. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Release criticality of bug 1365606 (get_network_info efficiency for nova-network)
> and I don't see how https://review.openstack.org/#/c/121663/ is actually > dependent on https://review.openstack.org/#/c/119521/. Yeah, agreed. I think that we _need_ the fix patch in Juno. The query optimization is good, and something we should take, but it makes me nervous sliding something like that in at the last minute without more exposure. Especially given that it has been like this for more than one release, it seems like Kilo material to me. --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] object field naming question
> The value it adds (and that an underscore would add in hvtype ->
> hv_type) is that the name would match the naming style for the vast
> majority of everything else in OpenStack. See, for examples:
Agreed.
> As mentioned in the review, I disagree on this point, since "doing a
> cleanup afterwards" would mean having to increment the
> nova.objects.SupportedInstance model VERSION as soon as it went into
> master. I say let's make the quick change now and avoid having to write
> code like this in the next patchset:
>
> if client_version <= (1,0):
> # We renamed hvtype to hv_type in 1.1
> self.hv_type = fields.get('hvtype')
Right, this becomes RPC debt if we think we might change it later. We
definitely want to get it right the first time whenever possible.
--Dan
signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Pulling nova/virt/hardware.py into nova/objects/
> OK, so in reviewing Dan B's patch series that refactors the virt > driver's get_available_resource() method [1], I am stuck between two > concerns. I like (love even) much of the refactoring work involved in > Dan's patches. They replace a whole bunch of our nested dicts that are > used in the resource tracker with real objects -- and this is something > I've been harping on for months that really hinders developer's > understanding of Nova's internals. dict['line1'] = 'Agreed, this is extremely important stuff.' dict['line2'] = 'The current dict mess that we have there is ' dict['line3'] = 'really obscure and confusing.' reply = jsonutils.dumps(dict) > However, all of the object classes that Dan B has introduced have been > unversioned objects -- i.e. they have not derived from > nova.objects.base.NovaObject. This means that these objects cannot be > sent over the wire via an RPC API call. In practical terms, this issue > has not yet reared its head, because the resource tracker still sends a > dictified JSON representation of the object's fields directly over the > wire, in the same format as Icehouse, therefore there have been no > breakages in RPC API compatibility. Right, so the blueprint for this work states that it's not to be sent over the RPC wire or stored in the database. However, it already is in some cases (at least the ComputeNode object has the unversioned JSONified version of some of these hardware models in it). If the modeling is purely for internal-to-compute-node purposes, then it's all good. However, it surely seems like with the pending scheduler isolation work, we're in a spot where we are building two parallel model hierarchies, and I'm not really sure why. > My proposal is that before we go and approve any BPs or patches that add > to nova/virt/hardware.py, we first put together a patch series that > moves the object models in nova/virt/hardware.py to being full-fledged > objects in nova/objects/* I'm not sure that just converting them all to NovaObjects is really necessary here. If it's all stuff that is going to go over the wire eventually as part of the resource tracker's expansion, then probably so. If there are bits of the model that only serve to let the resource tracker do its calculations, then perhaps it doesn't make sense to require those be NovaObjects. Regardless, it sounds like we need some discussion on how best to proceed here. Since it's entirely wrapped up in the scheduler work, we should definitely try to make sure that what we're doing here fits with those plans. Last I heard, we weren't sure where we were going to draw the line between nova bits and scheduler bits, so erring on the side of "more versioned interfaces" seems safest to me. --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Pulling nova/virt/hardware.py into nova/objects/
> As there are multiple interfaces using non versioned dicts and as we are > looking at reducing technical debt by Kilo, there are different > blueprints which can be worked in parallel. I don't think I disagree with anything above, but I'm not sure what you're getting at. I think the parallelism we should avoid is building models that mirror things we need to send to a remote service and just require conversion. If we're modeling things between the virt driver and the RT (or interface to the RT) that get aggregated or transformed in some way before they leave the service, then that's fine. > Here, the virt-to-RT interface has to be objectified, hence Dan's work. > On the other end of the RT, the RT-to-scheduler interface has to be > objectified, hence Jay and mine's work. Right, but if the RT moves out of the compute service, then we'll need to be using versioned models, regardless of if it's RPC or REST or whatever. So *if* that's going to happen, building an unversioned object hierarchy that then necessarily has to be converted to another form might be work we don't need to do. > I hope we will provide a clear big picture and a roadmap for the Summit > so we could give you more insights. Right, since that part is still not fully defined, it's hard to know the best course of action going forward. I'd hate to delay any of this work until after summit, but if you think that would be the most efficient, I guess it's only a couple weeks away. > Totally agreed. Here there is no need to version the interface as the > virt/Rt interface is not RPC based and purely internal to nova-compute. Well, unless the RT is moved outside the compute node, which is (I think) what is being proposed, no? --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Pulling nova/virt/hardware.py into nova/objects/
> The rationale behind two parallel data model hiercharies is that the > format the virt drivers report data in, is not likely to be exactly > the same as the format that the resoure tracker / scheduler wishes to > use in the database. Yeah, and in cases where we know where that line is, it makes sense to use the lighter-weight modeling for sure. > FWIW, my patch series is logically split up into two parts. THe first > 10 or so patches are just thought of as general cleanup and useful to > Nova regardless of what we decide todo. The second 10 or so patches > are where the objects start appearing & getting used & the controversial > bits needing mor detailed discussion. Right, so after some discussion I think we should go ahead and merge the bottom of this set (all of them are now acked I think) and continue the discussion on the top half where the modeling is introduced. Thanks! --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Taking a break..
> I won’t have much time for OpenStack, but I’m going to continue to > hang out in the channels. Nope, sorry, veto. Some options to explain your way out: 1. Oops, I forgot it wasn't April 2. I have a sick sense of humor; I'm getting help for it 3. I've come to my senses after a brief break from reality Seriously, I don't recall a gerrit review for this terrible plan... > Anyway, it’s been fun… the project has grown like crazy! Keep on > trucking... And while I won’t be active much, don’t be afraid to ping > me! Well, I for one am really sorry to see you go. I'd be lying if I said I hope that your next opportunity leaves you daydreaming about going back to OpenStack before too long. However, if not, good luck! --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] questions on object/db usage
>When I fix some bugs, I found that some code in > nova/compute/api.py > sometimes we use db ,sometimes we use objects do we have > any criteria for it? I knew we can't access db in compute layer code, > how about others ? prefer object or db direct access? thanks Prefer objects, and any remaining db.* usage anywhere (other than the object code itself) is not only a candidate for cleanup, it's much appreciated :) --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] pci pass through turing complete config options?
> If we really need that level of arbitrary complexity and future name > values we should then just: > > pci_passthrough_cfg = /etc/nova/pci_pass.yaml I hate to have to introduce a new thing like that, but I also think that JSON-encoded config variable strings are a nightmare. They lead to bugs like this: https://bugs.launchpad.net/nova/+bug/1383345 So I'd rather see something that is a little easier to manage. Also, moving things like this out to a separate file makes it easy to generate/update that file automatically, which is probably a useful thing for something like PCI. --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][CI] nova-networking or neutron netwokring for CI
> Are current nova CI platforms configured with nova-networking or with > neutron networking? Or is networking in general not even a part of the > nova CI approach? I think we have several that only run on Neutron, so I think it's fine to just do that. --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Consistency, efficiency, and safety of NovaObject.save()
> An initial inconsistency I have noticed is that some objects refresh > themselves from the database when calling save(), but others don't. I agree that it would be ideal for all objects to behave the same in this regard. I expect that in practice, it's not necessary for all objects to do this, and since it is an extra step/query in some places, it's not without cost to just always refresh (there is no reason to refresh a Flavor object AFAIK, for instance). > The lack of consistency in behaviour is obviously a problem, and I > can't think of any good reason for a second select for objects which > do that. However, I don't think it is good design for save() to > refresh the object at all, and the reason is concurrency. The cached > contents of a Nova object are *always* potentially stale. A refresh > does nothing to change that, because the contents are again > potentially stale as soon as it returns. Handling this requires > concurrency primitives which we don't currently have (see the larger > context I mentioned above). Refreshing an object's contents might > reduce the probability of a race, but it doesn't fix it. Callers who > want a refresh after save can always call object.refresh(), but for > others it's just wasted hits on the db. Doing a refresh() after a save() is more than just another DB hit, it's another RPC round-trip, which is far more expensive. There is a lot of code in the compute manager (and elsewhere I'm sure) that expects to get back the refreshed object after a save (our instance update functions have always exhibited this behavior, so there is code built to expect it). Any time something could change async on the object that might affect what we're about to do will benefit from this implicit refreshing. A good example is when we're doing something long-running on an instance and it is deleted underneath us. If we didn't get the deleted=True change after a save(), we might go continue doing a lot of extra work before we notice it the next time. It's not that doing so prevents the object's contents from being stale, it's that it reduces the amount of time before we notice a change, and avoids us needing to explicitly check. Any code we have that can't tolerate the object being stale is broken anyway. > Refresh on save() is also arbitrary. Why should the object be updated > then rather than at any other time? The timing of an update in thread > X is unrelated to the timing of an update in thread Y, but it's a > problem whenever it happens. Because it's not about cache consistency, it's about the expense required to do any of these things. To save(), we *have* to make the round-trip, so why not get a refresh at the same time? In cases where we explicitly want to refresh, we call refresh(), but otherwise we use natural synchronization points like save() to do that. > Can anybody see a problem if we didn't fetch the row at all, and > simply updated it? Absent locking or compare-and-swap this is > effectively what we're already doing, and it reduces the db cost of > save to a single update statement. The difference would be that the > object would remain stale without an explicit refresh(). Value > munging would remain unaffected. At least for instance, I don't want to do away with the implicit refreshing. I would be up for any of these options: 1. Documenting which objects do and don't auto-refresh 2. Making the case for non-Instance objects to not auto-refresh 3. Making them all auto-refresh for consistency, with the appropriate re-tooling of the db/api code to minimize performance impact. > Additionally, Instance, InstanceGroup, and Flavor perform multiple > updates on save(). I would apply the same rule to the sub-updates, > and also move them into a single transaction such that the updates > are atomic. Yep, no complaints about fixing these non-atomic updates, of course :) --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Consistency, efficiency, and safety of NovaObject.save()
> I personally favour having consistent behaviour across the board. How > about updating them all to auto-refresh by default for consistency, > but adding an additional option to save() to disable it for particular > calls? I think these should be two patches: one to make them all auto-refresh, and another to make it conditional. That serves the purpose of (a) bisecting a regression to one or the other, and (b) we can bikeshed on the interface and appropriateness of the don't-refresh flag :) > I also suggest a tactical fix to any object which fetches itself twice > on update (e.g. Aggregate). I don't see that being anything other than an obvious win, unless there is some obscure reason for it. But yeah, seems like a good thing to do. --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Cells] Cells subgroup and meeting times
> I am proposing Wednesday as the meeting day since it's more open than > Tues/Thurs so finding a meeting room at almost any time should be > feasible. My opening bid is alternating between 1700 and 2200 UTC. > That should provide options that aren't too early or too late for most > people. Is this fine for everyone or should it be adjusted a bit? Both work fine for me. --Dan signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
