Hi, 1 and 2) CloudStack now supports managed storage for the following hypervisor types:
* Disk Offerings for XenServer and ESX as of CS 4.2 * Disk Offerings for KVM as of CS 4.3 * Compute Offerings for XenServer and ESX as of CS 4.4 The SolidFire plug-in is capable of supporting whatever CloudStack-managed configurations are available in the version of CloudStack that you are using. Rate limiting is supported for the KVM hypervisor only as of CS 4.2. Min knows the details of this KVM rate-limiting feature better than I do (he implemented it), so I have CCed him. I think this rate limiting is only from the hypervisor's point of view...without input into the storage system that actually provides the volume in question. In other words, the hypervisor doesn't actually know how many IOPS the primary storage is capable of delivering...it only knows to limit the IO it sends to the storage system for that volume. Rate limiting is not a guarantee on IOPS, though. It is simply a limit on the rate at which IO can be sent to the storage system. By the way, for the sake of simplicity, when I'm talking in terms of IOPS here, I'm assuming all of your IOPS are the same size...which is probably not the case in the real world, so you'd need to take that into account, as well. In theory, if you have primary storage that has, say, 10,000 IOPS available and you carve out 10 1,000 IOPS (rate limited) volumes on that primary storage, you may be OK. However, this depends. For many storage solutions, you still have to worry about drive groups, LUN assignments, failure scenarios, etc. when it comes to whether or not you'll actually get 1,000 IOPS for each of those 10 volumes on the storage side. In these environments, you also need to take into consideration write amplification from data protection (ex. RAID). If the combination of IOPS from all the SSDs on your primary storage is X, you will get less than X from the point of view of the clients of that storage system. All hypervisor rate limiting is doing is saying that at most you can get 1,000 IOPS for each of those 10 volumes...not that you actually will (a guarantee) get those IOPS. If performance can be guaranteed at the storage infrastructure, this becomes much easier. For example, with a SolidFire SAN, you simply specify a minimum (guaranteed) number of IOPS, a maximum number of IOPS, and a burst number of IOPS for each volume (LUN). The SAN will guarantee the minimum number of IOPS (from the point of view of the client) regardless of how the data is physically distributed on its SSDs (you don't have to worry about RAID groups or low-level details like that). This minimum number of IOPS is guaranteed even during failure scenarios. 3) Punith is referring to dynamically changing the number of IOPS available of the SAN volume and making those new IOPS available to the CloudStack volume or volumes that leverage that SAN volume. 4) Hypervisor rate limiting - by itself - does not guarantee network performance, that is true. This is true of storage QoS, as well, but rarely is the network the bottleneck (SolidFire requires a 10G Ethernet network for the storage traffic). I hope that helps, Mike On Mon, Jun 2, 2014 at 1:42 AM, Yoshikazu Nojima <m...@ynojima.net> wrote: > Punith, Mike, > > Allow me to ask questions regarding guaranteed IOPS feature. > (Since it may be off-topic to Hieu's original question, I changed the > subject.) > > Q1) Does your plugins(Solidfire, CloudBytes) support KVM? From which > version? > > In my understanding, > Current CloudStack's implementation guarantees IOPS by two regulations: > - Storage appliance (or Hypervisor) limits Volume's max IOPS and > throughput. > - Management Server ensures max IOPS performance of a storage will not > be exceeded by the sum of volumes' IOPS in the storage by controlling > volume provisioning to the storage. > > Q2) Is my understanding accurate? > > > Q3) If we make volume IOPS configurable on fly, max IOPS of a storage > can be exceeded by the sum of volumes' IOPS in the storage, cannot be? > > Q4) I think current implementation doesn't guarantee the bandwidth > between a hypervisor and a storage network SW. > If high IOPS VMs are deployed in a hypervisor, the bandwidth between > the hypervisor and a storage network SW can be bottle neck. > Why don't we implement VM deployment rule that make the sum of > volumes' throughput not to exceed the bandwidth between hypervisor and > storage network SW? > > > Thanks, > Noji > > > 2014-06-01 23:50 GMT-06:00 Mike Tutkowski <mike.tutkow...@solidfire.com>: > > Thanks, Punith - this is similar to what I was going to say. > > > > Any time a set of CloudStack volumes share IOPS from a common pool, you > > cannot guarantee IOPS to a given CloudStack volume at a given time. > > > > Your choices at present are: > > > > 1) Use managed storage (where you can create a 1:1 mapping between a > > CloudStack volume and a volume on a storage system that has QoS). As > Punith > > mentioned, this requires that you purchase storage from a vendor who > > provides guaranteed QoS on a volume-by-volume bases AND has this > integrated > > into CloudStack. > > > > 2) Create primary storage in CloudStack that is not managed, but has a > high > > number of IOPS (ex. using SSDs). You can then storage tag this primary > > storage and create Compute and Disk Offerings that use this storage tag > to > > make sure their volumes end up on this storage pool (primary storage). > This > > will still not guarantee IOPS on a CloudStack volume-by-volume basis, but > > it will at least place the CloudStack volumes that need a better chance > of > > getting higher IOPS on a storage pool that could provide the necessary > > IOPS. A big downside here is that you want to watch how many CloudStack > > volumes get deployed on this primary storage because you'll need to > > essentially over-provision IOPS in this primary storage to increase the > > probability that each and every CloudStack volume that uses this primary > > storage gets the necessary IOPS (and isn't as likely to suffer from the > > Noisy Neighbor Effect). You should be able to tell CloudStack to only > use, > > say, 80% (or whatever) of the storage you're providing to it (so as to > > increase your effective IOPS per GB ratio). This over-provisioning of > IOPS > > to control Noisy Neighbors is avoided in option 1. In that situation, you > > only provision the IOPS and capacity you actually need. It is a much more > > sophisticated approach. > > > > Thanks, > > Mike > > > > > > On Sun, Jun 1, 2014 at 11:36 PM, Punith S <punit...@cloudbyte.com> > wrote: > > > >> hi hieu, > >> > >> your problem is the bottle neck we see as a storage vendors in the > cloud, > >> meaning all the vms in the cloud have not been guaranteed iops from the > >> primary storage, because in your case i'm assuming you are running > 1000vms > >> on a xen cluster whose all vm's disks are lying on a same primary nfs > >> storage mounted to the cluster, > >> hence you won't get the dedicated iops for each vm since every vm is > >> sharing the same storage. to solve this issue in cloudstack we the third > >> party vendors have implemented the plugin(namely cloudbyte , solidfire > etc) > >> to support managed storage(dedicated volumes with guaranteed qos for > each > >> vms) , where we are mapping each root disk(vdi) or data disk of a vm > with > >> one nfs or iscsi share coming out of a pool, also we are proposing the > new > >> feature to change volume iops on fly in 4.5, where you can increase or > >> decrease your root disk iops while booting or at peak times. but to use > >> this plugin you have to buy our storage solution. > >> > >> if not , you can try creating a nfs share out of ssd pool storage and > >> create a primary storage in cloudstack out of it named as golden primary > >> storage with specific tag like gold, and create a compute offering for > your > >> template with the storage tag as gold, hence all the vm's you create > will > >> sit on this gold primary storage with high iops. and other data disks on > >> other primary storage but still here you cannot guarantee the qos at vm > >> level. > >> > >> thanks > >> > >> > >> On Mon, Jun 2, 2014 at 10:12 AM, Hieu LE <hieul...@gmail.com> wrote: > >> > >>> Hi all, > >>> > >>> There are some problems while deploying a large amount of VMs in my > >>> company > >>> with CloudStack. All VMs are deployed from same template (e.g: Windows > 7) > >>> and the quantity is approximately ~1000VMs. The problems here is low > IOPS, > >>> low performance of VM (about ~10-11 IOPS, boot time is very high). The > >>> storage of my company is SAN/NAS with NFS and Xen Server 6.2.0. All Xen > >>> Server nodes have standard server HDD disk raid. > >>> > >>> I have found some solutions for this such as: > >>> > >>> - Enable Xen Server Intellicache and some tweaks in CloudStack > codes to > >>> deploy and start VM in Intellicache mode. But this solution will > >>> transfer > >>> all IOPS from shared storage to all local storage, hence affect and > >>> limit > >>> some CloudStack features. > >>> - Buying some expensive storage solutions and network to increase > IOPS. > >>> Nah.. > >>> > >>> So, I am thinking about a new feature that (may be) increasing IOPS and > >>> performance of VMs: > >>> > >>> 1. Separate golden image in high IOPS partition: buying new SSD, > plug > >>> in > >>> Xen Server and deployed a new VM in NFS storage WITH golden image in > >>> this > >>> new SSD partition. This can reduce READ IOPS in shared storage and > >>> decrease > >>> boot time of VM. (Currenty, VM deployed in Xen Server always have a > >>> master > >>> image (golden image - in VMWare) always in the same storage > repository > >>> with > >>> different image (child image)). We can do this trick by tweaking in > VHD > >>> header file with new Xen Server plug-in. > >>> 2. Create golden primary storage and VM template that enable this > >>> feature. > >>> 3. So, all VMs deployed from template that had enabled this feature > >>> will > >>> have a golden image stored in golden primary storage (SSD or some > high > >>> IOPS > >>> partition), and different image (child image) stored in other normal > >>> primary storage. > >>> > >>> This new feature will not transfer all IOPS from shared storage to > local > >>> storage (because high IOPS partition can be another high IOPS shared > >>> storage) and require less money than buying new storage solution. > >>> > >>> What do you think ? If possible, may I write a proposal in CloudStack > >>> wiki ? > >>> > >>> BRs. > >>> > >>> Hieu Lee > >>> > >>> -- > >>> -----BEGIN GEEK CODE BLOCK----- > >>> Version: 3.1 > >>> GCS/CM/IT/M/MU d-@? s+(++):+(++) !a C++++(++++)$ ULC++++(++)$ P > >>> L++(+++)$ E > >>> !W N* o+ K w O- M V- PS+ PE++ Y+ PGP+ t 5 X R tv+ b+(++)>+++ DI- D+ G > >>> e++(+++) h-- r(++)>+++ y- > >>> ------END GEEK CODE BLOCK------ > >>> > >> > >> > >> > >> -- > >> regards, > >> > >> punith s > >> cloudbyte.com > >> > > > > > > > > -- > > *Mike Tutkowski* > > *Senior CloudStack Developer, SolidFire Inc.* > > e: mike.tutkow...@solidfire.com > > o: 303.746.7302 > > Advancing the way the world uses the cloud > > <http://solidfire.com/solution/overview/?video=play>*™* > -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud <http://solidfire.com/solution/overview/?video=play>*™*
