Well, this all depends on how granular you setup your zones. If you have a zone hierarchy that goes all the way down to the rack level, your requests would be sufficient without needing to identify a particular object (volume).
Within a particular service such as nova, you could allow 'near:volume-001' which would know how to parse the object and lookup the zone, but you could also get the zone tag for 'volume-001' and use that for the 'near' value as well. Again, if you want to hide the topology, you can have a simple zone ID<->real location service shared between projects and only expose the IDs via public APIs. -Eric On Fri, Feb 11, 2011 at 12:38:18PM -0800, Vishvananda Ishaya wrote: > I just don't think you can get enough information from zone. > Volumes/Instances might > need to be literally in the same rack. The same could be true for > lbs/firewalls. Coarse > zones might be good for some cases, butI think locality to actual objects is > really useful > in the long run. We don't necessarily need to provide it for all objects, > but we should > for the ones where it makes sense. We can fallback to course zones for > objects > that don't need it. > > Vish > > On Feb 11, 2011, at 12:22 PM, Eric Day wrote: > > > If we go the admin-api lookup route, I worry about this as the number > > of services grow. If nova needs to support launching instances near > > any other object in an openstack service, we may be looking at: swift > > objects, glance images, firewalls, load balancers, queue workers, > > database masters, database replicas, etc. Should nova need to speak all > > of those APIs to lookup 'near' request tokens? If we can abstract away > > the actual object and instead have a zone (now suggesting completely > > opaque so we can keep it accurate without giving away topology), we > > can simply pass these zones IDs and let the deployment only need to > > understand how to resolve zones (one service) rather than all services. > > > > For the swift case you mention, the client requesting an instance could > > ask for current zones for all replicas of a swift object and pass all > > three zones into the nova request: near-on-of:a,b,c where a,b,c are > > the three opaque zones names. Nova could then pick the best zone of the > > three to launch an instance on without need to talk to swift directly. > > > > -Eric > > > > On Fri, Feb 11, 2011 at 12:01:47PM -0800, Vishvananda Ishaya wrote: > >> Yes, i think that cross-project should be requests across an admin api. I > >> don't think high level zone info is enough information for the scheduler > >> to decide where the instance should go. Especially with things like > >> swift, where there are three copies that may be in different zones > >> altogether. > >> > >> Vish > >> On Feb 11, 2011, at 11:55 AM, Eric Day wrote: > >> > >>> Hey Vish! > >>> > >>> On Fri, Feb 11, 2011 at 12:10:14PM -0600, Vishvananda Ishaya wrote: > >>>> I think that providers may not wish to expose the internal structure of > >>>> their network to the degree that you are suggesting. I prefer the idea > >>>> of > >>>> near other object with an internal lookup into zone. > >>> > >>> Providers can always obscure the naming to hide topology, or > >>> services can have options to only expose a certain depth of > >>> the hierarchy (ie, truncate at the 'datacenter level', hide all > >>> room/rack/switch zone level details). So even though volume-001 > >>> may be in rack12.room2.dc1.dfw.rackspace.com, the API only returns > >>> dc1.dfw.rackspace.com. We can of course allow for both too when the > >>> granularity is needed within a project: > >>> > >>> Within nova requests, allow: > >>> > >>> near-volume:volume-001 > >>> > >>> But across projects: > >>> in-zone: dc1.dfw.rackspace.com > >>> > >>> If we don't expose object zones at all, how would you support > >>> cross-project lookups? If nova gets a request 'near:swift-12345', > >>> would it be configured to query swift to get the zone via an admin API? > >>> > >>> We could also expose detailed object zones but keep them completely > >>> opaque, and instead require each service to allow pluggable > >>> scheduler/zone mapping. For example, a volume may return zone > >>> '87af1b7c', which internally all services could lookup and translate > >>> to where that maps in the deployment hierarchy. I was hoping DNS names > >>> were abstract enough to reduce complexity, but perhaps they are not. :) > >>> > >>> -Eric > >>> > >>>> On Feb 11, 2011 9:38 AM, "Eric Day" <e...@oddments.org> wrote: > >>>>> Hi Sandy, > >>>>> > >>>>> I agree with using tags for full scheduler selection, it's something > >>>>> I've been pushing for from the start. The request contains any number > >>>>> of k/v pairs, the services provide any number of k/v pairs, and the > >>>>> scheduler performs a match (some required, some optional, ...). I see > >>>>> the URI/zone as one of those tags, not something we need to overload > >>>>> to contain all of the capabilities. It should only be a hierarchical > >>>>> "location", which may be geographic location, organizational location > >>>>> (dept, ...), or some other type (however you decide to construct > >>>>> your zones). > >>>>> > >>>>> For example, imagine a dynamic del.icio.us tag that allowed for domain > >>>>> name filtering on bookmarks (give me all bookmarks with tags [book > >>>>> review] [domain:slashdot.org]). For Nova, this means issuing requests > >>>>> like "create instance with [GPU] [Fast disk] [zone:dc1.example.com]". > >>>>> > >>>>> The important thing is that this is not a tag specific to a particular > >>>>> service. For example, Swift would never care or need to understand a > >>>>> 'GPU' tag, but it can share and understand zone tags. > >>>>> > >>>>> -Eric > >>>>> > >>>>> On Fri, Feb 11, 2011 at 12:40:44PM +0000, Sandy Walsh wrote: > >>>>>> Heh, hate to be the one to bust up the URI love-fest :) > >>>>>> > >>>>>> The issue I have with a single URI being used as the heuristic for node > >>>> selection is that it is very rigid. > >>>>>> > >>>>>> Different business units have different views on the network: > >>>>>> * Operations may view it as geography/data centers. > >>>>>> * Consumers may view it as technical ability (gpu's, fast disk, good > >>>> inter-server speed, etc) > >>>>>> * Sales/marketing may view it as the number of martinis they can buy ;) > >>>>>> > >>>>>> Trees become unmanageable/hard to visualize for users beyond a couple > >>>> hundred nodes. We are lucky that our geographical/DC-based hierarchy is > >>>> relatively flat. This is why I was initially pushing for a tag-based > >>>> system for selection (aka Zone/Host Capabilities). > >>>>>> > >>>>>> Consider the way delicio.us works. They manage many millions of URL's > >>>> and tags are an effective way to slice & dice your way through the data: > >>>>>> "Show me all the URL's on [OpenStack] [Python] [Zones] [Scheduler]" ... > >>>> blam. > >>>>>> > >>>>>> This is also the way the old Trader services worked: > >>>>>> "I want a [wax transfer] [color] printer that can has [30ppm] and > >>>> [300dpi] on [Floor 2]" > >>>>>> > >>>>>> "Near" simply has to mean the distance in zones from the most-optimal > >>>> zones, based on the tags. > >>>>>> > >>>>>> "I want a new instance with [GPU] and [Fast Disk] [Good inter-instance > >>>> network speed] [near] [DRW] [DC1]" > >>>>>> * where "[near]" implies "as close as possible to" in zone distance. > >>>>>> > >>>>>> Personally I don't like overloading the zone name to have a > >>>> "meaningful" URI when we can get the same functionality with > >>>> Capabilities/Tags already. And we already know we need Capability > >>>> support > >>>> anyway. Especially if it means enforcing a rigid hierarchy. > >>>>>> > >>>>>> $0.02 > >>>>>> > >>>>>> -S > >>>>>> > >>>>>> > >>>>>> > >>>>>> ________________________________________ > >>>>>> From: openstack-bounces+sandy.walsh=rackspace....@lists.launchpad.net > >>>> [openstack-bounces+sandy.walsh=rackspace....@lists.launchpad.net] on > >>>> behalf of Eric Day [e...@oddments.org] > >>>>>> Sent: Friday, February 11, 2011 4:30 AM > >>>>>> To: Justin Santa Barbara > >>>>>> Cc: openstack@lists.launchpad.net; Devin Carlen > >>>>>> Subject: Re: [Openstack] Allowing clients to pass capability requests > >>>> through tags? > >>>>>> > >>>>>> The main reason I was proposing full location/zone of objects is to > >>>>>> allow this type of 'near' scheduling to happen without understanding > >>>>>> what the actual object is. For example, imagine we want to start an > >>>>>> instance near a particular swift object. We could query the swift > >>>>>> object and in the metadata there could be a 'zone' tag (well, three, > >>>>>> one for each copy). For example: > >>>>>> > >>>>>> get swift-12345: zone=rack12.room2.dc1.dfw.rackspace.com > >>>>>> > >>>>>> I can now use that zone name to: > >>>>>> > >>>>>> create_instance: openstack:near=rack12.room2.dc1.dfw.rackspace.com > >>>>>> > >>>>>> The deployment can decide what 'near' is (perhaps a measure of link > >>>>>> speed or latency). This way a particular deployment that uses the > >>>>>> same URI/zone names across projects can account for locality without > >>>>>> knowing what objects from different services are. If it were just > >>>>>> 'near=swift-12345', it would need to understand what a swift object > >>>>>> was and perform that lookup to find out where it is. > >>>>>> > >>>>>> So you can still grab a zone tag from a volume you created: > >>>>>> > >>>>>> get vol-000001: rack4.room2.dc1.dfw.rackspace.com > >>>>>> > >>>>>> and use the zone to launch an instance with: > >>>>>> > >>>>>> create_instance: openstack:near=rack4.room2.dc1.dfw.rackspace.com > >>>>>> > >>>>>> We can also write schedulers/tools for a particular deployment > >>>>>> that understands the zones to just say 'always prefer in > >>>>>> dc1.dfw.rackspace.com', because power is cheaper there right now, or > >>>>>> 'test.dc1.dfw.rackspace.com' because that is my test zone (perhaps > >>>>>> only enabled for certain accounts in the scheduler too). > >>>>>> > >>>>>> -Eric > >>>>>> > >>>>>> On Thu, Feb 10, 2011 at 03:38:42PM -0800, Justin Santa Barbara wrote: > >>>>>>> I think the blueprint was largely complementary to the multi-zone > >>>> stuff; > >>>>>>> this is more about how the client _requests_ a particular > >>>>>>> location/capability through the API. The multi-zone blueprint seems > >>>> to be > >>>>>>> more about how nova would satisfy those requests (in a non-trivial > >>>> zone > >>>>>>> structure.) > >>>>>>> The root motivator is indeed getting a 'good' connection to a storage > >>>>>>> volume. I'm thinking of iSCSI SAN storage here, so in my case this > >>>>>>> probably means the SAN device with the least number of switches in > >>>>>>> between. There could well be SAN devices in each rack (e.g. Solaris > >>>>>>> volume nodes), or the devices could even be running on the host > >>>> nodes, and > >>>>>>> I don't believe that zones in the EC2 sense are sufficient here. > >>>>>>> But I guess that if the zone hierarchy went all the way down to the > >>>> rack > >>>>>>> (or machine), that would work. So I could create a volume and it > >>>> would > >>>>>>> come back with a location of "rack4.room2.dc1.dfw.rackspace.com" and > >>>> I > >>>>>>> could then request allocation of machines in that same rack? Is that > >>>> the > >>>>>>> vision of the nested zones? > >>>>>>> I do have a concern that long-term if we _only_ use zones, that's > >>>> trying > >>>>>>> to multiplex a lot of information into the zone hierarchy, and we can > >>>>>>> really only put one attribute in there. I also like the flexibility > >>>> of > >>>>>>> the 'openstack:near=vol-000001' request, because then the cloud can > >>>> decide > >>>>>>> how near to place the instance based on its knowledge of the > >>>> topology, and > >>>>>>> the clients can be oblivious to the storage system and arrangement. > >>>> But, > >>>>>>> my immediate requirement would indeed be satisfied if the zones went > >>>> down > >>>>>>> to the rack/machine level. > >>>>>>> An alternative way to look at zones and instance-types is that > >>>> they're > >>>>>>> actually just fail-if-not-satisfiable tags of the creation request > >>>>>>> (openstack:+zone=us-east-1a and openstack:+instancetype=m1.large) > >>>> They're > >>>>>>> only distinguished attributes because AWS doesn't have an > >>>>>>> extensibility mechanism, which this blueprint would give us. > >>>>>>> Justin > >>>>>>> > >>>>>>> On Thu, Feb 10, 2011 at 3:12 PM, Devin Carlen <devcam...@me.com> > >>>> wrote: > >>>>>>> > >>>>>>> I haven't totally digested this blueprint yet but it seems like there > >>>> is > >>>>>>> some overlap with what is being discussed with the multi zone > >>>> metadata > >>>>>>> stuff. One approach might be to handle this awt the scheduler level > >>>>>>> though and try to ensure things are always in the same zone when > >>>>>>> appropriate. > >>>>>>> I think the bigger question you raise is how to request local volumes > >>>>>>> when possible, yes? > >>>>>>> > >>>>>>> Devin > >>>>>>> On Feb 10, 2011, at 3:37 PM, Justin Santa Barbara > >>>> <jus...@fathomdb.com> > >>>>>>> wrote: > >>>>>>> > >>>>>>> Does anyone have any thoughts/objections on the blueprint I posted > >>>> for > >>>>>>> allowing clients to pass capability-requests through tags? I'm > >>>>>>> planning on starting implementation soon, so if people think this is > >>>> a > >>>>>>> bad idea I'd rather know before I start coding! > >>>>>>> Blueprint: > >>>> > >>>> https://blueprints.launchpad.net/nova/+spec/use-metadata-tags-for-capabilities > >>>>>>> Wiki: > >>>> > >>>> https://blueprints.launchpad.net/nova/+spec/use-metadata-tags-for-capabilities > >>>>>>> And a quick TLDR: > >>>>>>> API clients need a way to request e.g. placement of machines near > >>>> each > >>>>>>> other / near volumes, or that a volume be created with a particular > >>>>>>> RAID level, or that a machine be created in a HIPAA compliant > >>>>>>> environment. (This is complementary to the work on hierarchical zones > >>>>>>> & URL naming, I believe) > >>>>>>> I propose using the instance tags for this, e.g. specifying > >>>>>>> openstack:near=vol-000001 when creating an instance to request > >>>>>>> locating the instance 'close to' that volume. > >>>>>>> By default these requests would be best-effort and > >>>> ignored-if-unknown; > >>>>>>> if the client wants to specify that something is required and should > >>>>>>> fail if not understood or not satisfiable, they could use a "+" e.g. > >>>>>>> openstack:+location=*.dc1.north.rackspace.com > >>>>>>> Controversially (?), this would not be supported for clients using > >>>> the > >>>>>>> AWS API, because tags can only be specified once the instance has > >>>>>>> already been created. > >>>>>>> Feedback appreciated! > >>>>>>> Justin > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> Mailing list: https://launchpad.net/~openstack > >>>>>>> Post to : openstack@lists.launchpad.net > >>>>>>> Unsubscribe : https://launchpad.net/~openstack > >>>>>>> More help : https://help.launchpad.net/ListHelp > >>>>>> > >>>>>>> _______________________________________________ > >>>>>>> Mailing list: https://launchpad.net/~openstack > >>>>>>> Post to : openstack@lists.launchpad.net > >>>>>>> Unsubscribe : https://launchpad.net/~openstack > >>>>>>> More help : https://help.launchpad.net/ListHelp > >>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> Mailing list: https://launchpad.net/~openstack > >>>>>> Post to : openstack@lists.launchpad.net > >>>>>> Unsubscribe : https://launchpad.net/~openstack > >>>>>> More help : https://help.launchpad.net/ListHelp > >>>>>> > >>>>>> > >>>>>> Confidentiality Notice: This e-mail message (including any attached or > >>>>>> embedded documents) is intended for the exclusive and confidential use > >>>> of the > >>>>>> individual or entity to which this message is addressed, and unless > >>>> otherwise > >>>>>> expressly indicated, is confidential and privileged information of > >>>> Rackspace. > >>>>>> Any dissemination, distribution or copying of the enclosed material is > >>>> prohibited. > >>>>>> If you receive this transmission in error, please notify us immediately > >>>> by e-mail > >>>>>> at ab...@rackspace.com, and delete the original message. > >>>>>> Your cooperation is appreciated. > >>>>> > >>>>> _______________________________________________ > >>>>> Mailing list: https://launchpad.net/~openstack > >>>>> Post to : openstack@lists.launchpad.net > >>>>> Unsubscribe : https://launchpad.net/~openstack > >>>>> More help : https://help.launchpad.net/ListHelp _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
