Hi Mike, Is there any use case to have a more generic signature for updateStoragePool API ?
e.g void updateStoragePool(StoragePool storagePool, Map<E,V> updateDetails); // not too sure of what should be E & V as of now. To start with E & V can be String types or Enums for better static checks. instead of below void updateCapacity(StoragePool storagePool, Long capacityBytes, Long capacityIops); Regards, Amit *CloudByte Inc.* <http://www.cloudbyte.com/> On Fri, Jun 20, 2014 at 10:37 AM, Mike Tutkowski < mike.tutkow...@solidfire.com> wrote: > I just wanted to update those who are interested in this thread about work > I've done on this over the past couple days. > > This gist is that I've added a new method to the PrimaryDataStoreLifeCycle > interface that has the following signature (this code is not yet checked > in): > > void updateCapacity(StoragePool storagePool, Long capacityBytes, Long > capacityIops); > > This method can be invoked from StorageManagerImpl when the > updateStoragePool API is called. > > This gives the storage plug-in that backs the primary storage in question > an opportunity to update the backend storage it represents, if that makes > sense to do in your particular case (for example, changing the size and/or > IOPS of a volume). > > There is a related enhancement to the resizeVolume API that I plan to > tackle next week. That one will be particularly useful for managed storage > plug-ins. > > Also, I have been collecting input on the generic key/value proposal here: > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=42566111 > > That may turn into a considerable amount of work. I was initially thinking > it could be fit into 4.5, but it might be 4.6. > > Thanks for any feedback! > > > On Thu, Jun 12, 2014 at 11:09 PM, Mike Tutkowski < > mike.tutkow...@solidfire.com> wrote: > >> As I think about this more, there are two situations we should cover: >> >> 1) Non-managed storage that has control over IOPS. >> >> When you invoke the createStoragePool API, you can pass in capacityIops. >> >> We should support modifying capacityIops via the updateStoragePool API. >> >> 2) Managed storage that has control over IOPS. >> >> In this environment, there is a 1:1 mapping between a SAN volume and a >> CloudStack volume. >> >> This is where we need to augment the resizeVolume API to accept - in a >> similar fashion to size - a new value for Min and/or Max IOPS. >> >> For example, a resizeVolume can be initiated by simply selecting a new >> Disk Offering. >> >> In this situation, the size and IOPS are part of the new Disk Offering >> (i.e. neither size nor IOPS can be marked as custom) and when the resize >> method of the storage plug-in is invoked, it will have a chance to change >> the size and/or IOPS. >> >> I would say we should perform a bit of analysis in the CloudStack volume >> logic to NOT stop resize from being invoked on the storage plug-in IF the >> volume size is the same, but the IOPS are different. This way the volume >> can be online as long as the user is only changing the IOPS of the volume. >> >> I also think it's only a problem for XenServer for the VDI to have its >> size changed dynamically. >> >> I plan to draw a flowchart for this soon. Once I do that I think it will >> be easier to talk in detail. >> >> Thanks! >> >> >> On Thu, Jun 12, 2014 at 12:59 PM, Mike Tutkowski < >> mike.tutkow...@solidfire.com> wrote: >> >>> From what I understand about the resizeVolume API, to change the size of >>> a given volume, you must either: >>> >>> 1) pass in a new Disk Offering (if the current Disk Offering the volume >>> uses does not allow for a custom size) >>> >>> or >>> >>> 2) pass in the ID of the volume and a new size (if the current Disk >>> Offering the volume uses does allow for a custom size) >>> >>> Either way, if you are shrinking the volume's size, you then have to >>> pass in 'true' for the 'shrinkok' parameter. >>> >>> One thing we should do is support this same concept with IOPS. At the >>> time being, both Min and Max IOPS can be custom (set by user) or non custom >>> (set by admin). This is a direct parallel to custom size or non-custom >>> size. If the user is using a non-custom IOPS setting and wants to switch to >>> a custom IOPS setting, he should be able to do so by switching to a Disk >>> Offering with custom IOPS. Of course we should support doing this while the >>> volume is attached. >>> >>> If arbitrary key/value pairs can be associated with Disk Offerings, then >>> you should be able to get the new key/value pairs by switching to a new >>> Disk Offering. We'd want to allow this to work with the volume in the >>> attached state, as well. >>> >>> Perhaps we should allow this all to happen online (volume attached) >>> UNLESS doing what we're about to do will change the size of the volume. >>> Then we can fail the OP and tell them to detach the volume and re-run the >>> OP. >>> >>> What are you thoughts on that? >>> >>> Also, I think volumeResize only works for data disks at the time being. >>> >>> In my mind, volumeResize is a bit of a misnomer now. We are really >>> allowing the user to resize their Disk Offering now in terms of not only >>> size, but IOPS, and even arbitrary key/value pairs. This is still all done >>> by selecting a new Disk Offering (or - if you have a custom size or custom >>> IOPS disk offering already - by passing in the ID of the volume and the new >>> size and/or IOPS). >>> >>> Let's brainstorm on this a bit and see which way makes sense to go. >>> >>> Thanks! >>> >>> >>> On Thu, Jun 12, 2014 at 9:48 AM, Punith S <punit...@cloudbyte.com> >>> wrote: >>> >>>> sure mike. >>>> >>>> and i have one question, >>>> >>>> which existing volume api are we gonna use for changing disk >>>> offerings(properties) dynamically ? >>>> since resize api will not allow unless the disk is detached ! >>>> >>>> thanks. >>>> >>>> >>>> On Thu, Jun 12, 2014 at 11:37 AM, Mike Tutkowski < >>>> mike.tutkow...@solidfire.com> wrote: >>>> >>>>> Sounds good - let me give some thought as to how we should break up >>>>> the work. >>>>> >>>>> My GSoC student from Tunisia will be helping us, as well. >>>>> >>>>> >>>>> On Wed, Jun 11, 2014 at 8:34 AM, Punith S <punit...@cloudbyte.com> >>>>> wrote: >>>>> >>>>>> yes it sounds good, thanks for the proposal mike, >>>>>> >>>>>> yeah right now i have implemented prototype of my proposal, since its >>>>>> not generic we shall implement your proposal for 4.5. >>>>>> on the other hand, for 4.5 i'm supporting nfs protocol and resize >>>>>> feature for managed storage in xenserver, now trying to implement >>>>>> snapshot >>>>>> and support root disk for vm's. >>>>>> and yes if we can club together, i can start working on this proposal >>>>>> for data disk and get the prototype ready. >>>>>> what do you think ? >>>>>> >>>>>> thanks. >>>>>> >>>>>> >>>>>> On Wed, Jun 11, 2014 at 3:53 AM, Mike Tutkowski < >>>>>> mike.tutkow...@solidfire.com> wrote: >>>>>> >>>>>>> I'll send out a [PROPOSAL] e-mail so others who may not be following >>>>>>> this e-mail thread have a better chance to comment on the feature. >>>>>>> >>>>>>> >>>>>>> On Tue, Jun 10, 2014 at 2:58 PM, Mike Tutkowski < >>>>>>> mike.tutkow...@solidfire.com> wrote: >>>>>>> >>>>>>>> Here is that design document I was referring to, Punith: >>>>>>>> >>>>>>>> >>>>>>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=42566111 >>>>>>>> >>>>>>>> I've been working with a student in Tunisia who is participating in >>>>>>>> Google Summer of Code (GSoC) (I'm his mentor). >>>>>>>> >>>>>>>> He'll be working on part of this as will I. (He is also working on >>>>>>>> another related task not listed here.) >>>>>>>> >>>>>>>> Feel free to join us, if you have time available, as we can divide >>>>>>>> out coding and testing among the three of us. >>>>>>>> >>>>>>>> Talk to you later! >>>>>>>> Mike >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Jun 10, 2014 at 10:18 AM, Mike Tutkowski < >>>>>>>> mike.tutkow...@solidfire.com> wrote: >>>>>>>> >>>>>>>>> I plan to draw up a design document surrounding generic key/value >>>>>>>>> pairs today. >>>>>>>>> >>>>>>>>> Perhaps you can take a look at it when you have time, Punith? >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Jun 10, 2014 at 10:06 AM, Mike Tutkowski < >>>>>>>>> mike.tutkow...@solidfire.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Punith, >>>>>>>>>> >>>>>>>>>> Yeah, I hear you about the number of permutations involved. >>>>>>>>>> >>>>>>>>>> Traditionally Compute and Disk Offerings have been immutable. It >>>>>>>>>> makes it easier from an accounting point of view for chargeback and >>>>>>>>>> billing. >>>>>>>>>> >>>>>>>>>> You should definitely feel free to extend the CloudStack API. I >>>>>>>>>> think NetApp did this for one of their storage features in the >>>>>>>>>> recent past. >>>>>>>>>> This way vendor-specific capabilities can be more easily offered >>>>>>>>>> without >>>>>>>>>> making it look like all vendors support those particular features. >>>>>>>>>> >>>>>>>>>> I do not yet have any code in master related to generic >>>>>>>>>> keys/values. I'm still designing this. >>>>>>>>>> >>>>>>>>>> How does your schedule look for the 4.5 release? Do you think you >>>>>>>>>> might have available cycles to help out with this generic key/value >>>>>>>>>> implementation? >>>>>>>>>> >>>>>>>>>> Talk to you later! >>>>>>>>>> Mike >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Jun 10, 2014 at 7:09 AM, Punith S <punit...@cloudbyte.com >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> hi mike, >>>>>>>>>>> >>>>>>>>>>> thanks for the reply, i like your approach which is a very >>>>>>>>>>> generic way and also we only need to do a few changes to the current >>>>>>>>>>> cloudstack, >>>>>>>>>>> >>>>>>>>>>> but on the other hand we are tying every property of the vendor >>>>>>>>>>> to a disk offering through key/value pairs, since we offer lot of >>>>>>>>>>> properties like i mentioned, this can create a lot of permutation >>>>>>>>>>> combinations of disk offerings, for say if i need to turn >>>>>>>>>>> deduplication On >>>>>>>>>>> for a specific volume , should i have to create a new disk offering >>>>>>>>>>> having >>>>>>>>>>> current properties with deduplication On? >>>>>>>>>>> >>>>>>>>>>> is this approach already implemented in the current master ? >>>>>>>>>>> >>>>>>>>>>> and also like you mentioned about exposing a new api, is it okay >>>>>>>>>>> if i expose our own api in my util by extending the PlugableService >>>>>>>>>>> like in >>>>>>>>>>> network plugins ? >>>>>>>>>>> >>>>>>>>>>> thanks. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Jun 10, 2014 at 1:17 AM, Mike Tutkowski < >>>>>>>>>>> mike.tutkow...@solidfire.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Allow me to follow this up with more detail (with regards to >>>>>>>>>>>> what Chris and I talked about): >>>>>>>>>>>> >>>>>>>>>>>> As you are aware, today the way you associate a Compute >>>>>>>>>>>> Offering (CO) or a Disk Offering (DO) with a Primary Storage (PS) >>>>>>>>>>>> is via >>>>>>>>>>>> storage tagging. >>>>>>>>>>>> >>>>>>>>>>>> This has some benefits and drawbacks. >>>>>>>>>>>> >>>>>>>>>>>> One benefit is being able to have some level of vendor >>>>>>>>>>>> independence from the point of view of the CO or DO. For example, >>>>>>>>>>>> if the >>>>>>>>>>>> storage tag of a DO is "Fast", then this can be satisfied by PS >>>>>>>>>>>> that >>>>>>>>>>>> describes itself as "Fast", regardless of vendor. >>>>>>>>>>>> >>>>>>>>>>>> A major drawback with the storage-tagging approach, however, is >>>>>>>>>>>> that you are not easily able to leverage vendor-specific features, >>>>>>>>>>>> which is >>>>>>>>>>>> often why you bought storage from the vendor in question to begin >>>>>>>>>>>> with. >>>>>>>>>>>> >>>>>>>>>>>> Ideally we do not want to add each vendor's features into the >>>>>>>>>>>> system as properties that can be seen by the admin regardless of >>>>>>>>>>>> whether or >>>>>>>>>>>> not the underlying storage he's actually using supports the >>>>>>>>>>>> feature in >>>>>>>>>>>> question. >>>>>>>>>>>> >>>>>>>>>>>> This coarse approach, however, was sort of business as usual >>>>>>>>>>>> when I started in with CloudStack 1.5 years ago. >>>>>>>>>>>> >>>>>>>>>>>> That being the case, when I added QoS options to CS, I did so >>>>>>>>>>>> in a way where the admin would see Min IOPS and Max IOPS options >>>>>>>>>>>> regardless >>>>>>>>>>>> of whether or not his storage actually supported those controls (to >>>>>>>>>>>> mitigate this a bit in the GUI, the admin has to explicitly select >>>>>>>>>>>> "Storage >>>>>>>>>>>> QoS" from a combobox). >>>>>>>>>>>> >>>>>>>>>>>> We leverage the same use model with Hypervisor QoS: The admin >>>>>>>>>>>> sees these options regardless of whether or not they actually >>>>>>>>>>>> apply on the >>>>>>>>>>>> hypervisor where the VM gets deployed. >>>>>>>>>>>> >>>>>>>>>>>> Going forward, we want to implement a more fine-grain and >>>>>>>>>>>> generic approach. >>>>>>>>>>>> >>>>>>>>>>>> We would like to have a storage provider field for the CO and >>>>>>>>>>>> DO windows (this equates to the name of one and only one storage >>>>>>>>>>>> provider). >>>>>>>>>>>> If the admin inputs a specific storage provider and does not use >>>>>>>>>>>> the >>>>>>>>>>>> storage tags field, he can enter in an arbitrary number of >>>>>>>>>>>> key/value pairs >>>>>>>>>>>> in another text field (perhaps we would provide a nice entry >>>>>>>>>>>> dialog to make >>>>>>>>>>>> this easier in the GUI). These key value pairs can be passed into >>>>>>>>>>>> the >>>>>>>>>>>> storage driver when it's asked to create (or update) a volume and >>>>>>>>>>>> the >>>>>>>>>>>> storage driver can decide what each and every key/value pair means. >>>>>>>>>>>> >>>>>>>>>>>> What do you think about this approach? >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Jun 9, 2014 at 1:14 PM, Mike Tutkowski < >>>>>>>>>>>> mike.tutkow...@solidfire.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Punith, >>>>>>>>>>>>> >>>>>>>>>>>>> This kind of a feature is something Chris Suich and I >>>>>>>>>>>>> discussed a while back. >>>>>>>>>>>>> >>>>>>>>>>>>> We talked about leveraging arbitrary key/value pairs to make >>>>>>>>>>>>> this happen (OpenStack does something similar). The key/value >>>>>>>>>>>>> pairs would >>>>>>>>>>>>> be vendor specific. >>>>>>>>>>>>> >>>>>>>>>>>>> If we take a key/value approach, we might be able to make this >>>>>>>>>>>>> all work the way things work today when the user wants to change >>>>>>>>>>>>> an >>>>>>>>>>>>> existing Compute Offering and/or Disk Offering. >>>>>>>>>>>>> >>>>>>>>>>>>> For example, the user would pick a new Compute Offering (with >>>>>>>>>>>>> presumably has different key/value pairs) and CloudStack could >>>>>>>>>>>>> inform the >>>>>>>>>>>>> applicable storage provider, who could update the volume in >>>>>>>>>>>>> question. >>>>>>>>>>>>> >>>>>>>>>>>>> This way we don't need to introduce a new API command and the >>>>>>>>>>>>> use model for the user doesn't really change. >>>>>>>>>>>>> >>>>>>>>>>>>> What are you thoughts on this? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Mike >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Jun 9, 2014 at 8:08 AM, Punith S < >>>>>>>>>>>>> punit...@cloudbyte.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> hi guys, >>>>>>>>>>>>>> >>>>>>>>>>>>>> since most of the third party storage providers have been >>>>>>>>>>>>>> implementing 1:1 mapping(managed storage) between a >>>>>>>>>>>>>> volume(dataset) and a >>>>>>>>>>>>>> vm disk(vdi/vmdk) for guaranteeing the Qos, i would like to >>>>>>>>>>>>>> propose a new >>>>>>>>>>>>>> feature to dynamically change the volume properties supported by >>>>>>>>>>>>>> storage >>>>>>>>>>>>>> vendors such as IOPS, Deduplication, Compression, Grace, >>>>>>>>>>>>>> Syncronization, >>>>>>>>>>>>>> Latency etc, depending on properties and features supported by >>>>>>>>>>>>>> respective >>>>>>>>>>>>>> storage vendors. hence providing more flexibility for users. >>>>>>>>>>>>>> >>>>>>>>>>>>>> in case of using default cloudstack storage provider, we can >>>>>>>>>>>>>> change the properties of the vdi/vmdk files apart from resizing >>>>>>>>>>>>>> the >>>>>>>>>>>>>> volume(vdi/vmdk). >>>>>>>>>>>>>> >>>>>>>>>>>>>> changes in management server include, >>>>>>>>>>>>>> >>>>>>>>>>>>>> new async web api ChangeVolumePropertiesCmd, >>>>>>>>>>>>>> new method in VolumeApiService for vo and dao validation >>>>>>>>>>>>>> implementations. >>>>>>>>>>>>>> new method in VolumeServiceManager for supporting callback >>>>>>>>>>>>>> and calling the respective storage provider driver's >>>>>>>>>>>>>> implementation. >>>>>>>>>>>>>> new method in PrimaryDataStoreDriver interface for >>>>>>>>>>>>>> implementing respective features according to their storage >>>>>>>>>>>>>> product. >>>>>>>>>>>>>> >>>>>>>>>>>>>> changes in UI include, >>>>>>>>>>>>>> new changing volume properties widget in volume section, >>>>>>>>>>>>>> showing different properties depending upon listed storage >>>>>>>>>>>>>> providers. >>>>>>>>>>>>>> >>>>>>>>>>>>>> any suggestions and feedbacks ? >>>>>>>>>>>>>> >>>>>>>>>>>>>> thanks >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> regards, >>>>>>>>>>>>>> >>>>>>>>>>>>>> punith s >>>>>>>>>>>>>> cloudbyte.com >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> *Mike Tutkowski* >>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>>>>>> e: mike.tutkow...@solidfire.com >>>>>>>>>>>>> o: 303.746.7302 >>>>>>>>>>>>> Advancing the way the world uses the cloud >>>>>>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™* >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> *Mike Tutkowski* >>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>>>>> e: mike.tutkow...@solidfire.com >>>>>>>>>>>> o: 303.746.7302 >>>>>>>>>>>> Advancing the way the world uses the cloud >>>>>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™* >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> regards, >>>>>>>>>>> >>>>>>>>>>> punith s >>>>>>>>>>> cloudbyte.com >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *Mike Tutkowski* >>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>>> e: mike.tutkow...@solidfire.com >>>>>>>>>> o: 303.746.7302 >>>>>>>>>> Advancing the way the world uses the cloud >>>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™* >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Mike Tutkowski* >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>>> e: mike.tutkow...@solidfire.com >>>>>>>>> o: 303.746.7302 >>>>>>>>> Advancing the way the world uses the cloud >>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™* >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Mike Tutkowski* >>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>>> e: mike.tutkow...@solidfire.com >>>>>>>> o: 303.746.7302 >>>>>>>> Advancing the way the world uses the cloud >>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™* >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Mike Tutkowski* >>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>> e: mike.tutkow...@solidfire.com >>>>>>> o: 303.746.7302 >>>>>>> Advancing the way the world uses the cloud >>>>>>> <http://solidfire.com/solution/overview/?video=play>*™* >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> regards, >>>>>> >>>>>> punith s >>>>>> cloudbyte.com >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Mike Tutkowski* >>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>> e: mike.tutkow...@solidfire.com >>>>> o: 303.746.7302 >>>>> Advancing the way the world uses the cloud >>>>> <http://solidfire.com/solution/overview/?video=play>*™* >>>>> >>>> >>>> >>>> >>>> -- >>>> regards, >>>> >>>> punith s >>>> cloudbyte.com >>>> >>> >>> >>> >>> -- >>> *Mike Tutkowski* >>> *Senior CloudStack Developer, SolidFire Inc.* >>> e: mike.tutkow...@solidfire.com >>> o: 303.746.7302 >>> Advancing the way the world uses the cloud >>> <http://solidfire.com/solution/overview/?video=play>*™* >>> >> >> >> >> -- >> *Mike Tutkowski* >> *Senior CloudStack Developer, SolidFire Inc.* >> e: mike.tutkow...@solidfire.com >> o: 303.746.7302 >> Advancing the way the world uses the cloud >> <http://solidfire.com/solution/overview/?video=play>*™* >> > > > > -- > *Mike Tutkowski* > *Senior CloudStack Developer, SolidFire Inc.* > e: mike.tutkow...@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the cloud > <http://solidfire.com/solution/overview/?video=play>*™* >
