Re: [Openstack] Easy API

2010-12-30 Thread Jorge Williams
Hi Andy, The delegated auth pattern in Easy API seems to match our existing blueprint for authentication in OpenStack. blueprint: https://blueprints.launchpad.net/nova/+spec/nova-authn spec: http://wiki.openstack.org/openstack-authn Have you taken a look at the blueprint? Are we on the same p

Re: [Openstack] Easy API

2010-12-30 Thread Andy Smith
Hi Jorge, I don't really think we are on the same page for very long, your spec goes quite far down the path of specific authorization schemes where as mine really only takes the first step. In general the spirit seems to be the same, that we want another service to be able to handle authenticati

Re: [Openstack] Easy API

2010-12-30 Thread Jorge Williams
Hi Andy, >From your description, I don't think we're that far apart at all. In fact, I >think we're proposing almost exactly the same thing -- at least in terms of >auth. The purpose of our blueprint is *not* to provide a framework for >creating an authentication service, but rather to simpl

Re: [Openstack] Some insight into the number of instances Nova needs to spin up...

2010-12-30 Thread Ewan Mellor
Those are interesting figures, but it would also be interesting to know how many of those are kept running. Our 1M host target is irrelevant if all those instances are shut down again a day later. If Amazon are adding *and keeping* 70k VMs a day then that's a very different matter. Ewan. > -

[Openstack] Housekeeping specs

2010-12-30 Thread Thierry Carrez
Hi everyone, We have a few housekeeping tasks that are basically a shared effort and never completely ended, like improving our test code coverage, docstrings or pylint scores. Generic blueprints are very bad at tracking that kind of effort, since those are not assigned to specific people, or tar

Re: [Openstack] Some insight into the number of instances Nova needs to spin up...

2010-12-30 Thread Pete Zaitcev
On Wed, 29 Dec 2010 19:27:09 + Erik Carlin wrote: > The 1M host limit still seems reasonable to me. [] In my opinion, such numbers are completely out of whack. Google's Chubby article says that the busiest Chubby has 90,000 clients (not hosts!) and the biggest datacenter has 10,000 systems.

Re: [Openstack] Some insight into the number of instances Nova needs to spin up...

2010-12-30 Thread Bret Piatt
We can break this down into a list of measureable test cases. Many of these tests aren't just single host system tests. They require an integrated deployment to find and eliminate the bottleneck. I'm certain I'm missing additional items we would want to measure so consider this an off the top of

Re: [Openstack] Some insight into the number of instances Nova needs to spin up...

2010-12-30 Thread Jay Pipes
On Wed, Dec 29, 2010 at 2:27 PM, Erik Carlin wrote: > We know Amazon is highly, highly elastic.  While the instances launched > per day is impressive, we know that many of those instances have a short > life. OK, good point. But, this begs the question: what should Nova's priority be? Elasticit

Re: [Openstack] Some insight into the number of instances Nova needs to spin up...

2010-12-30 Thread Erik Carlin
You are right. The 1M number was VMs not hosts. At least, that was from one scale discussion we had within Rackspace. I'm not sure what the "official" nova target limits are and I can't find anything on launchpad that defines it. If there is something, could someone please send me a link. I'm

Re: [Openstack] Some insight into the number of instances Nova needs to spin up...

2010-12-30 Thread Rick Clark
Actually it was 1M hosts and Ivthink 45 million vms. It was meant to be across all regions. Jason Seats set the number arbitrarily, but it is a good target to not let us forget about scaling while we design. I think eventually all loads will be more ephemeral. So, I think I agree with your num

Re: [Openstack] Some insight into the number of instances Nova needs to spin up...

2010-12-30 Thread Erik Carlin
Jay - Few comments below... Erik On 12/30/10 10:43 AM, "Jay Pipes" wrote: >On Wed, Dec 29, 2010 at 2:27 PM, Erik Carlin >wrote: >> We know Amazon is highly, highly elastic. While the instances launched >> per day is impressive, we know that many of those instances have a short >> life. > >OK

Re: [Openstack] Some insight into the number of instances Nova needs to spin up...

2010-12-30 Thread Erik Carlin
I suggest we consider the limits of a single nova deployment, not across all regions. To Pete's point, at a certain scale, people will break into parallel, independent nova deployments. A single, global, deployment becomes untenable at scale. I agree with Pete that 1M hosts (and 45M Vms) is a bi

Re: [Openstack] Some insight into the number of instances Nova needs to spin up...

2010-12-30 Thread Bret Piatt
These numbers seem very large today but a decade from now they won't at all. We don't need to design everything for the requirements in 2020 but we should have some view on where things are going. More swag calculations... In 2020... 60,000,000 servers worldwide (I'm assuming this doesn't grow su

Re: [Openstack] Some insight into the number of instances Nova needs to spin up...

2010-12-30 Thread Erik Carlin
Agree the scale limits will increase over time. I'm thinking more about what the near term (cactus or cactus+1) limits should be. The model we have been using is a single nova deployment per metro area (called a region). That metro area could be comprised of multiple Dcs and failure isolation dom

Re: [Openstack] Easy API

2010-12-30 Thread Matt Dietz
Thanks for the reply, I have to confess that I'm still concerned about the idea of yet another API for several reasons. * While I appreciate the idea of a compatibility layer of sorts, we already have a weird digression in available features between Amazon and Openstack. I think a 3rd API that

Re: [Openstack] Easy API

2010-12-30 Thread Sandy Walsh
Thanks Andy ... that answers many of our questions. I haven't had an opportunity to look at Easy API yet, but I don't think there is any reason to break compatibility with the RS/OS API. Here's how I see the flow ('Client' is the client-side command line tool): 1. Client determines Verb of Inte

[Openstack] [RFC] OpenStack API

2010-12-30 Thread John Purrier
The timing of this is interesting, as it comes when there is a lot of discussion about EasyAPI. I would like to encourage discussions about the EasyAPI extensibility mechanisms versus the OpenStack API v1.1 extension mechanism. Jorge, you will need to let folks see the proposed updates to the OpenS

[Openstack] [RFC] OpenStack Image Formats

2010-12-30 Thread John Purrier
A discussion in order to set a common understanding of supported image formats in OpenStack and how the project can approach the issues surrounding various hypervisors, cross-cloud interoperability, and potentially setting some necessary development work items. Once the community has reached consen

Re: [Openstack] [RFC] OpenStack Scope and projects

2010-12-30 Thread Andy Smith
I would quickly say this should be on the wiki so that it can be kept up to date as it is discussed, it is a document not an email. On Thu, Dec 30, 2010 at 12:59 PM, John Purrier wrote: > I would like to present for discussion a statement on the scope and > projects that are considered core to O

Re: [Openstack] [RFC] OpenStack API

2010-12-30 Thread Sandy Walsh
Thanks John. That's a good summary of the plan. I still have a few questions, if you could indulge me ... Seems like there are two issues being debated, one customer-facing, one Nova-developer-facing: Customer-facing: Will OS 1.1 remain backward compatible with 1.0? In other words, is 1.1 truly

Re: [Openstack] [RFC] OpenStack API

2010-12-30 Thread John Purrier
Hi Sandy, the easy one first. the OpenStack API is versioned, and so is backward compatible. The client will negotiate the highest version it knows about, the API returns will be consistent with that version. As for the Easy API.. The on-going discussion is excellent and I believe we are flushi

Re: [Openstack] [RFC] OpenStack Scope and projects

2010-12-30 Thread John Purrier
I can create wiki pages, where should they live? John From: Andy Smith [mailto:andys...@gmail.com] Sent: Thursday, December 30, 2010 3:20 PM To: John Purrier Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] [RFC] OpenStack Scope and projects I would quickly say this should be

Re: [Openstack] [RFC] OpenStack Scope and projects

2010-12-30 Thread Andy Smith
http://wiki.openstack.org I would assume, under whatever names you think are best, just so that there is a definitive doc that can be kept abreast of the discussion. On Thu, Dec 30, 2010 at 1:43 PM, John Purrier wrote: > I can create wiki pages, where should they live? > > > > John > > > > *From

Re: [Openstack] [RFC] OpenStack API

2010-12-30 Thread Jorge Williams
On Dec 30, 2010, at 1:59 PM, John Purrier wrote: Jorge, you will need to let folks see the proposed updates to the OpenStack API for version 1.1. Absolutely, I will post the spec once we settle on the new set of API features. The API extension mechanism and the transition to the OpenStack n

[Openstack] Glance/Nova Snapshot Changes

2010-12-30 Thread Rick Harris
In developing Nova's instance snapshots, I've run into a little snag revolving around somed design decisions in both Nova and Glance. I have a plan that I'd like to move forward with ASAP, so please, if you see any problems or have any objections, raise them now. Problem === OpenStack's API

Re: [Openstack] [RFC] OpenStack API

2010-12-30 Thread Jorge Williams
On Dec 30, 2010, at 3:43 PM, John Purrier wrote: Hi Sandy, the easy one first… the OpenStack API is versioned, and so is backward compatible. The client will negotiate the highest version it knows about, the API returns will be consistent with that version. Right. We create new versions only