Yeah, either solution will fix this issue for managed storage in general (ex. SolidFire, CloudByte).
On Tue, Sep 9, 2014 at 9:58 AM, Francois Gaudreault < fgaudrea...@cloudops.com> wrote: > That's great Punith :) > > Thanks for handling this one. I am not too worried about the option, as > long as it fixes SF integration for 4.5 :) > > FG > > On Tue, Sep 9, 2014 at 11:55 AM, Mike Tutkowski < > mike.tutkow...@solidfire.com> wrote: > >> Thanks for the info, Punith! >> >> Does that mean you are fixing the issue via Option2? >> >> On Tue, Sep 9, 2014 at 7:52 AM, Punith S <punit...@cloudbyte.com> wrote: >> >>> yes mike, >>> >>> w.r.t to option 1: >>> it will be like creating a VM w.r.t ISO, where admin will have to >>> specify the os disk(ROOT) disk size. >>> >>> for option 2: >>> i have figured out the issue, post downloading the template to S3, >>> cloudstack will again download the template from S3 to staging nfs store. >>> here we need to access the file and process it with VHD processor in >>> order to calculate the virtualsize but we are skipping this process, >>> hence the virtual size is not being calculated while using the S3 or >>> swift. >>> >>> the templates already present in the staging nfs storage cannot be >>> applied to this process. >>> >>> for option 3: >>> it's convenient to calculate the template virtual size while it is being >>> copied from s3 to staged nfs store instead of staged nfs to primary, >>> since admin might be using more than one primary stores. >>> >>> i'm fixing the issue, will post the patch ASAP for 4.5.snapshot. >>> >>> thanks! >>> >>> On Tue, Sep 9, 2014 at 11:13 AM, Mike Tutkowski < >>> mike.tutkow...@solidfire.com> wrote: >>> >>>> By the way, for anyone new to this issue, this is what we're referring >>>> to here: >>>> >>>> https://issues.apache.org/jira/browse/CLOUDSTACK-7406 >>>> >>>> On Mon, Sep 8, 2014 at 11:41 PM, Mike Tutkowski < >>>> mike.tutkow...@solidfire.com> wrote: >>>> >>>>> Great :) Then a question might be, "Is it too late in the game to >>>>> interrogate the template to discover its virtual size if we're just about >>>>> to copy the template to primary storage?" >>>>> >>>>> If it's not, this might be the place to run the logic to figure out >>>>> the virtual size. >>>>> >>>>> Really, there are three big possibilities: >>>>> >>>>> 1) Just ask the end user to provide the virtual size (not commenting >>>>> here on what happens for already-uploaded templates) >>>>> >>>>> or >>>>> >>>>> 2) Figure out the virtual size when the template is copied from object >>>>> storage to secondary storage and update the DB with this info (not sure >>>>> what happens if the template has already been copied to >>>>> (secondary-storage) >>>>> NFS because it was used before) >>>>> >>>>> or >>>>> >>>>> 3) Figure out the virtual size when the template is about to be copied >>>>> from secondary storage to primary storage >>>>> >>>>> On Mon, Sep 8, 2014 at 11:35 PM, Sanjeev Neelarapu < >>>>> sanjeev.neelar...@citrix.com> wrote: >>>>> >>>>>> Mike, >>>>>> >>>>>> You are right. Template gets copied to (secondary-storage) NFS before >>>>>> being copied to primary storage >>>>>> >>>>>> -Sanjeev >>>>>> >>>>>> -----Original Message----- >>>>>> From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] >>>>>> Sent: Tuesday, September 09, 2014 10:55 AM >>>>>> To: dev@cloudstack.apache.org >>>>>> Cc: Punith S; Francois Gaudreault >>>>>> Subject: Re: S3/Swift Problem around Virtual Size >>>>>> >>>>>> Hi Will, >>>>>> >>>>>> Thanks for the input! >>>>>> >>>>>> I like the idea of storing the virtual size as metadata in S3 or >>>>>> Swift although this could require that the end user provide this value >>>>>> when >>>>>> uploading the template. >>>>>> >>>>>> However, if we have the ability to determine the virtual size of the >>>>>> template after it gets downloaded to (secondary-storage) NFS and we're >>>>>> able >>>>>> to update the database with this info, then it would seem we would never >>>>>> need to ask the user for this value. >>>>>> >>>>>> Either way, the tricky part might be if the template in object >>>>>> storage has already been downloaded to (secondary-storage) NFS (say it >>>>>> was >>>>>> used before). In this case, we won't need to download it to >>>>>> (secondary-storage) NFS again (at least not in the same zone), so we >>>>>> won't >>>>>> have an easy opportunity to figure out the virtual size upon download >>>>>> from >>>>>> object storage. >>>>>> >>>>>> I wonder if it's too late in this process if we figured out the >>>>>> virtual size before the copied template (now on (secondary-storage) NFS) >>>>>> gets copied to primary storage. If we could do it at this point, then we >>>>>> wouldn't have to worry about fixing the "legacy" situation because it >>>>>> would >>>>>> just work out naturally. We would look in the DB to see if the virtual >>>>>> size >>>>>> for this template is known and, if not, we could figure out the virtual >>>>>> size before downloading from (secondary-storage) NFS to primary storage >>>>>> each time. (Although I'm thinking this would come too late in the process >>>>>> because we may have already asked the primary-storage plug-in to create >>>>>> the >>>>>> necessary volume.) >>>>>> >>>>>> By the way, I'm assuming that a template gets copied to >>>>>> (secondary-storage) NFS before being copied to primary storage. I'm not >>>>>> super familiar with how this process works. >>>>>> >>>>>> Talk to you later, >>>>>> Mike >>>>>> >>>>>> On Mon, Sep 8, 2014 at 10:59 PM, Will Stevens <wstev...@cloudops.com> >>>>>> wrote: >>>>>> >>>>>> > My two cents on the topic. >>>>>> > >>>>>> > Ideally we would save the size in the object store metadata and >>>>>> > retrieve it from the metadata if it is set. If it is not set in the >>>>>> > object store metadata, then when it is fetched, we have to put it on >>>>>> > NFS and determine the size (then ideally save the metadata back to >>>>>> the >>>>>> > object store) and remove the NFS copy. >>>>>> > >>>>>> > This way the NFS copy approach is only ever done once and then the >>>>>> > data is populated (for backwards compatibility). For all templates >>>>>> > created after the patch, the metadata would be stored and retrieved >>>>>> > without the need for the NFS copy. >>>>>> > >>>>>> > Is this feasible? >>>>>> > >>>>>> > Will >>>>>> > >>>>>> > >>>>>> > *Will STEVENS* >>>>>> > Lead Developer >>>>>> > >>>>>> > *CloudOps* *| *Cloud Solutions Experts >>>>>> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* >>>>>> tw >>>>>> > @CloudOps_ >>>>>> > >>>>>> > On Tue, Sep 9, 2014 at 12:48 AM, Mike Tutkowski < >>>>>> > mike.tutkow...@solidfire.com> wrote: >>>>>> > >>>>>> > > Hi Punith, >>>>>> > > >>>>>> > > Thanks for putting in time on this! >>>>>> > > >>>>>> > > So, option 1 occurs after we download the template to S3 or Swift. >>>>>> > > We >>>>>> > then >>>>>> > > have to copy it to NFS so that we can determine the virtual size? >>>>>> > > Relatively slow, but it would work. >>>>>> > > >>>>>> > > If we went this route, would we then delete that template from the >>>>>> > > NFS share since its immediate purpose was so we could figure out >>>>>> the >>>>>> > > virtual size or would we leave it on that share? >>>>>> > > >>>>>> > > Option 2 is to allow the user who uploads such a template to >>>>>> specify >>>>>> > > the size needed for the root disk (i.e. at least the virtual size >>>>>> of >>>>>> > > the template...perhaps larger)? >>>>>> > > >>>>>> > > Does that sound like I understood the options? >>>>>> > > >>>>>> > > Thanks! >>>>>> > > Mike >>>>>> > > >>>>>> > > On Mon, Sep 8, 2014 at 10:34 PM, Punith S <punit...@cloudbyte.com >>>>>> > >>>>>> > wrote: >>>>>> > > >>>>>> > > > hi mike, >>>>>> > > > >>>>>> > > > i have figured out the issue, in NFS secondary storage the >>>>>> virtual >>>>>> > > > size >>>>>> > > is >>>>>> > > > been calculated by the VHD Processor by accessing the vhd >>>>>> template. >>>>>> > > > >>>>>> > > > but in case of S3, cloudstack is not able to access the template >>>>>> > > > but it only gets to know the physical size of the template! >>>>>> > > > >>>>>> > > > in order to solve this, we need to download the template from S3 >>>>>> > > > to another secondary nfs storage and access the template to >>>>>> > > > calculate the virtual size. >>>>>> > > > >>>>>> > > > if the above solution seems to be complicated, we shall have a >>>>>> > > > quick >>>>>> > fix >>>>>> > > > where virtual size i.e the size of the SAN volume shall be >>>>>> > > > created(like small-20g, medium-40g) for people using the >>>>>> templates from S3. >>>>>> > > > >>>>>> > > > any feedback's ? >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > On Tue, Sep 9, 2014 at 6:34 AM, Mike Tutkowski < >>>>>> > > > mike.tutkow...@solidfire.com> wrote: >>>>>> > > > >>>>>> > > >> Hi Punith, >>>>>> > > >> >>>>>> > > >> Have you been able to make any progress with regards to this >>>>>> > > >> Swift/S3 issue? >>>>>> > > >> >>>>>> > > >> Thanks! >>>>>> > > >> Mike >>>>>> > > >> >>>>>> > > >> On Wed, Aug 27, 2014 at 7:43 AM, Punith S >>>>>> > > >> <punit...@cloudbyte.com> >>>>>> > > wrote: >>>>>> > > >> >>>>>> > > >> > hi >>>>>> > > >> > >>>>>> > > >> > think i had a timeout problem! >>>>>> > > >> > on the second try the template has been downloaded to the S3 >>>>>> > > >> > bucket >>>>>> > > and >>>>>> > > >> > the management server shows the status as download complete >>>>>> > > >> > with >>>>>> > > >> template >>>>>> > > >> > size as 1.6G instead of its virtual size 20G. >>>>>> > > >> > >>>>>> > > >> > and i see the template's status as Download Complete but it >>>>>> > > >> > seems it >>>>>> > > is >>>>>> > > >> > not getting installed ! refer the attachment >>>>>> > > >> > >>>>>> > > >> > can anyone explain the "installing template" after the >>>>>> download >>>>>> > > >> completes ? >>>>>> > > >> > >>>>>> > > >> > >>>>>> > > >> > On Wed, Aug 27, 2014 at 9:18 AM, Marcus <shadow...@gmail.com >>>>>> > >>>>>> > wrote: >>>>>> > > >> > >>>>>> > > >> >> Per Edisons comments about not knowing the image size, can't >>>>>> > > >> >> we >>>>>> > just >>>>>> > > >> set >>>>>> > > >> >> some headers and store metadata with the template in S3 to >>>>>> > > >> >> save the virtual size when the template is registered? I'm >>>>>> > > >> >> assuming here that the >>>>>> > SSVM >>>>>> > > >> does >>>>>> > > >> >> the work of pulling the template in and uploading to S3. Or >>>>>> it >>>>>> > could >>>>>> > > be >>>>>> > > >> >> stored in the template table? >>>>>> > > >> >> On Aug 26, 2014 9:11 PM, "Francois Gaudreault" < >>>>>> > > >> fgaudrea...@cloudops.com> >>>>>> > > >> >> wrote: >>>>>> > > >> >> >>>>>> > > >> >> > Looks like your SSVM cannot reach Internet properly? >>>>>> > > >> >> > >>>>>> > > >> >> > FG >>>>>> > > >> >> > >>>>>> > > >> >> > On 2014-08-26, 11:14 AM, Punith S wrote: >>>>>> > > >> >> > >>>>>> > > >> >> >> hi francois, >>>>>> > > >> >> >> >>>>>> > > >> >> >> since i'm not having a swift setup, i'm using the s3 >>>>>> bucket. >>>>>> > > >> >> >> >>>>>> > > >> >> >> and as you recommended i got the SSVM up with seeded nfs >>>>>> > storage, >>>>>> > > >> >> >> >>>>>> > > >> >> >> post that i removed the nfs secondary storage and added >>>>>> the >>>>>> > > >> >> >> S3 >>>>>> > > with >>>>>> > > >> >> >> staging nfs store as the new sec storage, since you >>>>>> cannot >>>>>> > > >> >> >> have >>>>>> > > any >>>>>> > > >> nfs >>>>>> > > >> >> >> secondary storage while using the S3. >>>>>> > > >> >> >> >>>>>> > > >> >> >> on registering the a new template, i'm getting template >>>>>> > > >> >> >> status >>>>>> > > >> >> as*Unable >>>>>> > > >> >> >> to execute HTTP request: No route to host* in >>>>>> > > >> >> >> managementserver.log >>>>>> > > >> >> >> >>>>>> > > >> >> >> 2014-08-26 20:41:07,502 DEBUG >>>>>> [o.a.c.s.RemoteHostEndPoint] >>>>>> > > >> >> >> (Timer-24:ctx-b68380cd) Sending command >>>>>> > > >> org.apache.cloudstack.storage. >>>>>> > > >> >> >> command.DownloadProgressCommand to host: 10 >>>>>> > > >> >> >> 2014-08-26 20:41:07,507 DEBUG [c.c.a.t.Request] >>>>>> > > >> (Timer-24:ctx-b68380cd) >>>>>> > > >> >> >> Seq 10-5684105679694996125: Sending { Cmd , MgmtId: >>>>>> > 52242179434, >>>>>> > > >> via: >>>>>> > > >> >> >> 10(s-142-VM), Ver: v1, Flags: 100011, >>>>>> [{"org.apache.cloudstack. >>>>>> > > >> >> >> >>>>>> > > >> >>>>>> storage.command.DownloadProgressCommand":{"jobId":"d43a17c9-3b03- >>>>>> > > >> 4ff9- >>>>>> > > >> >> >> 8906-e1d155981e86","request":"GET_STATUS","hvm":true," >>>>>> > > >> >> >> >>>>>> > > >> >>>>>> description":"centext","maxDownloadSizeInBytes":53687091200,"id":209," >>>>>> > > >> >> >> resourceType":"TEMPLATE","installPath":"template/tmpl/2/ >>>>>> > > >> >> >> >>>>>> 209/209-2-b624436c-5f37-30d4-8eaf-81582eb0d39d","_store":{" >>>>>> > > >> >> >> >>>>>> com.cloud.agent.api.to.S3TO":{"id":14,"uuid":"e4afd7bb-39ea >>>>>> > > >> >> >> - 4128-ab93-f8a09b1d5e03","bucketName":"test-cloudstack", >>>>>> > > >> >> >> "httpsFlag":false,"created":"Aug 26, 2014 8:16:24 >>>>>> > > >> >> PM","enableRRS":false," >>>>>> > > >> >> >> maxSingleUploadSizeInBytes":5368709120}},"url":"http:// >>>>>> > > >> >> >> >>>>>> download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz >>>>>> > > >> >> >> 2 >>>>>> > > >> >> >> >>>>>> > > >> >> >>>>>> > > >> >>>>>> > > >>>>>> > >>>>>> ","format":"VHD","accountId":2,"name":"209-2-b624436c-5f37-30d4-8eaf-8 >>>>>> > 1582eb0d39d","wait":0}}] >>>>>> > > >> >> >> } >>>>>> > > >> >> >> 2014-08-26 20:41:07,556 DEBUG [c.c.a.t.Request] >>>>>> > > >> >> >> (AgentManager-Handler-10:null) Seq >>>>>> 10-5684105679694996125: >>>>>> > > >> >> Processing: { >>>>>> > > >> >> >> Ans: , MgmtId: 52242179434, via: 10, Ver: v1, Flags: 10, >>>>>> > > >> >> >> [{"com.cloud.agent.api.storage.DownloadAnswer":{" >>>>>> > > >> >> >> jobId":"d43a17c9-3b03-4ff9-8906-e1d155981e86"," >>>>>> > > >> >> >> downloadPct":0,"errorString":"No route to >>>>>> > host","downloadStatus":" >>>>>> > > >> >> >> DOWNLOAD_ERROR","installPath":"template/tmpl/2/209/209-2- >>>>>> > > >> >> >> b624436c-5f37-30d4-8eaf-81582eb0d39d","templateSize": >>>>>> > > >> >> >> 0,"templatePhySicalSize":0,"result":true,"details":"No >>>>>> > > >> >> >> route to host","wait":0}}] } >>>>>> > > >> >> >> >>>>>> > > >> >> >> but i don't see any logging happening in secondary >>>>>> storage >>>>>> > > >> >> >> vm's >>>>>> > > >> >> cloud.log >>>>>> > > >> >> >> >>>>>> > > >> >> >> not sure this error is happening due to S3! >>>>>> > > >> >> >> >>>>>> > > >> >> >> >>>>>> > > >> >> >> thanks! >>>>>> > > >> >> >> >>>>>> > > >> >> > >>>>>> > > >> >> > >>>>>> > > >> >> > -- >>>>>> > > >> >> > Francois Gaudreault >>>>>> > > >> >> > Gestionnaire de Produit | Product Manager - Cloud >>>>>> Platform & >>>>>> > > Services >>>>>> > > >> >> > t:514-629-6775 >>>>>> > > >> >> > >>>>>> > > >> >> > CloudOps Votre partenaire infonuagique | Cloud Solutions >>>>>> > > >> >> > Experts >>>>>> > > >> >> > 420 rue Guy | Montreal | Quebec | H3J 1S6 >>>>>> > > >> >> > w: cloudops.com | tw: @CloudOps_ >>>>>> > > >> >> > >>>>>> > > >> >> > >>>>>> > > >> >> >>>>>> > > >> > >>>>>> > > >> > >>>>>> > > >> > >>>>>> > > >> > -- >>>>>> > > >> > regards, >>>>>> > > >> > >>>>>> > > >> > punith s >>>>>> > > >> > cloudbyte.com >>>>>> > > >> > >>>>>> > > >> >>>>>> > > >> >>>>>> > > >> >>>>>> > > >> -- >>>>>> > > >> *Mike Tutkowski* >>>>>> > > >> *Senior CloudStack Developer, SolidFire Inc.* >>>>>> > > >> e: mike.tutkow...@solidfire.com >>>>>> > > >> o: 303.746.7302 >>>>>> > > >> Advancing the way the world uses the cloud >>>>>> > > >> <http://solidfire.com/solution/overview/?video=play>*™* >>>>>> > > >> >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > -- >>>>>> > > > regards, >>>>>> > > > >>>>>> > > > punith s >>>>>> > > > cloudbyte.com >>>>>> > > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > -- >>>>>> > > *Mike Tutkowski* >>>>>> > > *Senior CloudStack Developer, SolidFire Inc.* >>>>>> > > e: mike.tutkow...@solidfire.com >>>>>> > > o: 303.746.7302 >>>>>> > > Advancing the way the world uses the cloud >>>>>> > > <http://solidfire.com/solution/overview/?video=play>*™* >>>>>> > > >>>>>> > >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Mike Tutkowski* >>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>> e: mike.tutkow...@solidfire.com >>>>>> o: 303.746.7302 >>>>>> Advancing the way the world uses the cloud >>>>>> <http://solidfire.com/solution/overview/?video=play>*™* >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Mike Tutkowski* >>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>> e: mike.tutkow...@solidfire.com >>>>> o: 303.746.7302 >>>>> Advancing the way the world uses the cloud >>>>> <http://solidfire.com/solution/overview/?video=play>*™* >>>>> >>>> >>>> >>>> >>>> -- >>>> *Mike Tutkowski* >>>> *Senior CloudStack Developer, SolidFire Inc.* >>>> e: mike.tutkow...@solidfire.com >>>> o: 303.746.7302 >>>> Advancing the way the world uses the cloud >>>> <http://solidfire.com/solution/overview/?video=play>*™* >>>> >>> >>> >>> >>> -- >>> regards, >>> >>> punith s >>> cloudbyte.com >>> >> >> >> >> -- >> *Mike Tutkowski* >> *Senior CloudStack Developer, SolidFire Inc.* >> e: mike.tutkow...@solidfire.com >> o: 303.746.7302 >> Advancing the way the world uses the cloud >> <http://solidfire.com/solution/overview/?video=play>*™* >> > > -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud <http://solidfire.com/solution/overview/?video=play>*™*
