Hi Vladimir, I think my previous email was a bit confusing. I did not mean placement of scheduling module should be anywhere else. I agree that it should be on the scheduler nodes.
What I meant was similar to what you explained in the last paragraph about back-ref to array... Whether the volume driver can accommodate multiple Vendor/Type, like SM, or a single one, it will need to select an actual array at some point. My suggestion was only to put enough intelligence in the generic scheduler (via plugins etc.) to be able to not only select the node to handle the request, but also to select the actual array (what I called storage backend) to host the volume. I don't think we should take for granted that there will be a one-to-one mapping between arrays and Storage Nodes. I can imagine a situation where a cloud provider might have more than one SN handling an array to account for node failures etc. and having a one-many model for array-SN could be overkill, so it could be many-many. Thanks, Renuka. From: Vladimir Popovski [mailto:vladi...@zadarastorage.com] Sent: Wednesday, October 19, 2011 11:20 AM To: Renuka Apte; Armando Migliaccio; Reddin, Tim (Cloud Services); Chuck Thier Cc: openstack-volume@lists.launchpad.net Subject: RE: [Openstack-volume] Basic volume-type aware scheduler Hi Renuka, >>> I had one question about the proposal. From what I understand, it seems to >>> assume that different types of storage backends (actual infrastructure >>> which houses the volume) will be controlled by different types of volume >>> drivers (or at least nodes). Is this a good assumption to make? Yes, we will have drivers per each storage Vendor (or even per Vendor/Type if Vendor will be unable to provide one generic driver). I can imagine that we may have "generic" drivers that will be able to operate with many different vendors. Some vendors may decide not to provide their own schedulers, but instead of that to provide data in the format that will be compliant with some generic scheduler. Regarding placement of scheduling module: I suppose it will be better to perform scheduling on actual scheduler nodes who may have broader picture about connectivity. Think about the situation when your Backend for some reason is inconsistently connected to nodes (nova-volume node A can see storage arrays 1 & 2, B can see 2 & 3 and C - 1 & 3). You are right about double-scheduling, because if nova-volume node (let's call it storage node or SN for simplicity) can operate with multiple arrays it will need to know which one should host it. I was thinking more about one-to-one connection of array to SN or case as ours, where SN has actual local storage. So, what this proposal is lacking is some sort of back-ref to array where volume should be actually hosted. When scheduler will make a decision about host it should not only forward it to SN, but include some opaque data that might be used by the driver. This opaque data might be the entire dict of key/value pairs used by the scheduler or some special blob reported by driver for this storage class. It will help us to have only one Scheduler (centralized or distributed), which seems to be more compliant with general Openstack model. To be more specific about the approach: Capabilities reported by SN to Scheduler might be like: "volume": [ {"type": "SATA", "RPM": 7200, ..., "location": "aaa", "access path": "a1"}, {"type": "SATA", "RPM": 15000, ..., "location": "bbb", "access path": "b1"}, {"type": "SAS", "RPM": 7200, ..., "location": "aaa", "access path": "c1"}, ... ] Create volume from Scheduler to SN: def create_volume(self, context, volume_id, snapshot_id=None, req_spec=None): ... Where req_spec will be something like: req_spec = {"type": "SATA", "RPM": 7200, ..., "location": "aaa", "access path": "a1"} The alternate approach might be to have: "volume": [ {"type": "SATA", "RPM": 7200, ..., "blob": "xxx " }, {"type": "SATA", "RPM": 15000, ..., "blob": "yyy" }, {"type": "SAS", "RPM": 7200, ..., "blob": "zzz" }, ... ] and req_spec = {"blob": "xxx "} This will require to change what schedule_create_volume() returns - not only host, but host + some special data. Regards, -Vladimir -----Original Message----- From: openstack-volume-bounces+vladimir=zadarastorage....@lists.launchpad.net<mailto:zadarastorage....@lists.launchpad.net> [mailto:openstack-volume-bounces+vladimir<mailto:openstack-volume-bounces%2Bvladimir>=zadarastorage....@lists.launchpad.net<mailto:zadarastorage....@lists.launchpad.net>] On Behalf Of Renuka Apte Sent: Wednesday, October 19, 2011 10:17 AM To: Armando Migliaccio; Reddin, Tim (Cloud Services); Chuck Thier Cc: openstack-volume@lists.launchpad.net<mailto:openstack-volume@lists.launchpad.net> Subject: Re: [Openstack-volume] Basic volume-type aware scheduler Hi, I had one question about the proposal. From what I understand, it seems to assume that different types of storage backends (actual infrastructure which houses the volume) will be controlled by different types of volume drivers (or at least nodes). Is this a good assumption to make? Instead do you think scheduling on the basis of backends instead of nodes is a better idea. The reason I find this more logical is, in the case of compute, we schedule on the basis of a node, because the node is actually housing (running) the VM. For volume drivers, the node is simply the brain, while some real storage that the node can reach, is where the volume is housed. Each driver can report whatever opaque key/value pairs they wish to, and plug in rules, just as you mentioned, to help the scheduler make this decision. In the absence of this, if we have symmetric volume drivers, that can reach most of the storage backends, we end up in a situation where the scheduler has not really shortlisted anything, and has left another round of scheduling to the driver. This is especially true if at some point we support multiple drivers per node, as you suggested. Whatever I said can certainly be accommodated in Vladimir's current proposal by doing two rounds of scheduling (pick node, pick storage from node). What I am asking is, if nodes, in general, need the ability to figure out which backend to create this volume on, why not push it into the generic scheduler itself? Thanks, Renuka. -----Original Message----- From: openstack-volume-bounces+renuka.apte=citrix....@lists.launchpad.net<mailto:openstack-volume-bounces+renuka.apte=citrix....@lists.launchpad.net> [mailto:openstack-volume-bounces+renuka.apte=citrix....@lists.launchpad.net]<mailto:[mailto:openstack-volume-bounces+renuka.apte=citrix....@lists.launchpad.net]> On Behalf Of Armando Migliaccio Sent: Wednesday, October 19, 2011 9:06 AM To: Reddin, Tim (Cloud Services); Chuck Thier Cc: openstack-volume@lists.launchpad.net<mailto:openstack-volume@lists.launchpad.net> Subject: Re: [Openstack-volume] Basic volume-type aware scheduler I agree with Tim. As I understand it, Vlad is talking Policies and Chuck is talking mechanisms. Those are not to be confused, but rather to be kept separate. Chuck's proposal may turn things to be simpler, but definitely not more flexible. This is because system implementation details will dictate policies according to which decisions are made...unless I am completely missing the point. Armando > -----Original Message----- > From: openstack-volume- > bounces+armando.migliaccio=citrix....@lists.launchpad.net<mailto:bounces+armando.migliaccio=citrix....@lists.launchpad.net> > bounces+[mailto:openstack-<mailto:openstack-> > volume-bounces+armando.migliaccio=citrix....@lists.launchpad.net<mailto:volume-bounces+armando.migliaccio=citrix....@lists.launchpad.net>] > On > volume-bounces+Behalf Of > Reddin, Tim (Cloud Services) > Sent: 19 October 2011 16:14 > To: Chuck Thier > Cc: > openstack-volume@lists.launchpad.net<mailto:openstack-volume@lists.launchpad.net> > Subject: Re: [Openstack-volume] Basic volume-type aware scheduler > > I think we need to differentiate between what we present to the > consumer, (which may be a simple set of flavors) and the attributes > which the scheduler using to make selection decisions. > > If I am reading the proposal correctly a driver can advertize a ( > mostly) opaque set of attributes which the scheduler can use to make > placement decisions. > > Doesn't this kind of approach facilitate the kind of placement > capabilites that Renuka was seeking? Without introducing any topology > or vendor specific awareness? > > In addituon wont the maintenance of these kind of attributes > facilitate an admin api in a vendor agnostic way? > > Tim > > > On 19 Oct 2011, at 15:52, "Chuck Thier" > <cth...@gmail.com<mailto:cth...@gmail.com>> wrote: > > > Hi Vladimir, > > > > I agree that we need a volume-type aware scheduler, and thanks for > > taking this on. I had envisioned it a bit different though. I was > > thinking that the cluster operator would define the volume types > > (similar to how they define vm flavors). Each type would have a > > mapping to a driver, and the scheduler would use this mapping to > > determine which driver to send the incoming request to. > > > > I think this makes things simpler and more flexible. For example an > > operator may not want to expose every capability of some storage > > system that they have added. > > > > -- > > Chuck > > > > > > On Tue, Oct 18, 2011 at 2:36 PM, Vladimir Popovski > > <vladi...@zadarastorage.com<mailto:vladi...@zadarastorage.com>> wrote: > >> Hi All, > >> > >> > >> > >> I've registered a new blueprint for volume-type aware scheduler: > >> > >> https://blueprints.launchpad.net/nova/+spec/volume-type-scheduler > >> > >> > >> > >> Please take a look at specification attached to it explaining main > >> principles (feel free to add/change parts of it). > >> > >> > >> > >> Regards, > >> > >> -Vladimir > >> > >> > >> > >> > >> > >> -- > >> Mailing list: https://launchpad.net/~openstack-volume > >> Post to : > >> openstack-volume@lists.launchpad.net<mailto:openstack-volume@lists.launchpad.net> > >> Unsubscribe : https://launchpad.net/~openstack-volume > >> More help : https://help.launchpad.net/ListHelp > >> > >> > > > > -- > > Mailing list: https://launchpad.net/~openstack-volume > > Post to : > > openstack-volume@lists.launchpad.net<mailto:openstack-volume@lists.launchpad.net> > > Unsubscribe : https://launchpad.net/~openstack-volume > > More help : https://help.launchpad.net/ListHelp > -- > Mailing list: https://launchpad.net/~openstack-volume > Post to : > openstack-volume@lists.launchpad.net<mailto:openstack-volume@lists.launchpad.net> > Unsubscribe : https://launchpad.net/~openstack-volume > More help : https://help.launchpad.net/ListHelp -- Mailing list: https://launchpad.net/~openstack-volume Post to : openstack-volume@lists.launchpad.net<mailto:openstack-volume@lists.launchpad.net> Unsubscribe : https://launchpad.net/~openstack-volume More help : https://help.launchpad.net/ListHelp -- Mailing list: https://launchpad.net/~openstack-volume Post to : openstack-volume@lists.launchpad.net<mailto:openstack-volume@lists.launchpad.net> Unsubscribe : https://launchpad.net/~openstack-volume More help : https://help.launchpad.net/ListHelp
-- Mailing list: https://launchpad.net/~openstack-volume Post to : openstack-volume@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-volume More help : https://help.launchpad.net/ListHelp
