yes mike, i'm fixing the issue w.r.t option 2.
testing the patch is consuming much time, since i have to register the templates via S3, and it has to download via S3 to staging store. can i know when is the code freez for 4.5 ? thanks! On Tue, Sep 9, 2014 at 9:30 PM, Mike Tutkowski <mike.tutkow...@solidfire.com > wrote: > Yeah, either solution will fix this issue for managed storage in general > (ex. SolidFire, CloudByte). > > On Tue, Sep 9, 2014 at 9:58 AM, Francois Gaudreault < > fgaudrea...@cloudops.com> wrote: > >> That's great Punith :) >> >> Thanks for handling this one. I am not too worried about the option, as >> long as it fixes SF integration for 4.5 :) >> >> FG >> >> On Tue, Sep 9, 2014 at 11:55 AM, Mike Tutkowski < >> mike.tutkow...@solidfire.com> wrote: >> >>> Thanks for the info, Punith! >>> >>> Does that mean you are fixing the issue via Option2? >>> >>> On Tue, Sep 9, 2014 at 7:52 AM, Punith S <punit...@cloudbyte.com> wrote: >>> >>>> yes mike, >>>> >>>> w.r.t to option 1: >>>> it will be like creating a VM w.r.t ISO, where admin will have to >>>> specify the os disk(ROOT) disk size. >>>> >>>> for option 2: >>>> i have figured out the issue, post downloading the template to S3, >>>> cloudstack will again download the template from S3 to staging nfs store. >>>> here we need to access the file and process it with VHD processor in >>>> order to calculate the virtualsize but we are skipping this process, >>>> hence the virtual size is not being calculated while using the S3 or >>>> swift. >>>> >>>> the templates already present in the staging nfs storage cannot be >>>> applied to this process. >>>> >>>> for option 3: >>>> it's convenient to calculate the template virtual size while it is >>>> being copied from s3 to staged nfs store instead of staged nfs to primary, >>>> since admin might be using more than one primary stores. >>>> >>>> i'm fixing the issue, will post the patch ASAP for 4.5.snapshot. >>>> >>>> thanks! >>>> >>>> On Tue, Sep 9, 2014 at 11:13 AM, Mike Tutkowski < >>>> mike.tutkow...@solidfire.com> wrote: >>>> >>>>> By the way, for anyone new to this issue, this is what we're referring >>>>> to here: >>>>> >>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-7406 >>>>> >>>>> On Mon, Sep 8, 2014 at 11:41 PM, Mike Tutkowski < >>>>> mike.tutkow...@solidfire.com> wrote: >>>>> >>>>>> Great :) Then a question might be, "Is it too late in the game to >>>>>> interrogate the template to discover its virtual size if we're just about >>>>>> to copy the template to primary storage?" >>>>>> >>>>>> If it's not, this might be the place to run the logic to figure out >>>>>> the virtual size. >>>>>> >>>>>> Really, there are three big possibilities: >>>>>> >>>>>> 1) Just ask the end user to provide the virtual size (not commenting >>>>>> here on what happens for already-uploaded templates) >>>>>> >>>>>> or >>>>>> >>>>>> 2) Figure out the virtual size when the template is copied from >>>>>> object storage to secondary storage and update the DB with this info (not >>>>>> sure what happens if the template has already been copied to >>>>>> (secondary-storage) NFS because it was used before) >>>>>> >>>>>> or >>>>>> >>>>>> 3) Figure out the virtual size when the template is about to be >>>>>> copied from secondary storage to primary storage >>>>>> >>>>>> On Mon, Sep 8, 2014 at 11:35 PM, Sanjeev Neelarapu < >>>>>> sanjeev.neelar...@citrix.com> wrote: >>>>>> >>>>>>> Mike, >>>>>>> >>>>>>> You are right. Template gets copied to (secondary-storage) NFS >>>>>>> before being copied to primary storage >>>>>>> >>>>>>> -Sanjeev >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] >>>>>>> Sent: Tuesday, September 09, 2014 10:55 AM >>>>>>> To: dev@cloudstack.apache.org >>>>>>> Cc: Punith S; Francois Gaudreault >>>>>>> Subject: Re: S3/Swift Problem around Virtual Size >>>>>>> >>>>>>> Hi Will, >>>>>>> >>>>>>> Thanks for the input! >>>>>>> >>>>>>> I like the idea of storing the virtual size as metadata in S3 or >>>>>>> Swift although this could require that the end user provide this value >>>>>>> when >>>>>>> uploading the template. >>>>>>> >>>>>>> However, if we have the ability to determine the virtual size of the >>>>>>> template after it gets downloaded to (secondary-storage) NFS and we're >>>>>>> able >>>>>>> to update the database with this info, then it would seem we would never >>>>>>> need to ask the user for this value. >>>>>>> >>>>>>> Either way, the tricky part might be if the template in object >>>>>>> storage has already been downloaded to (secondary-storage) NFS (say it >>>>>>> was >>>>>>> used before). In this case, we won't need to download it to >>>>>>> (secondary-storage) NFS again (at least not in the same zone), so we >>>>>>> won't >>>>>>> have an easy opportunity to figure out the virtual size upon download >>>>>>> from >>>>>>> object storage. >>>>>>> >>>>>>> I wonder if it's too late in this process if we figured out the >>>>>>> virtual size before the copied template (now on (secondary-storage) NFS) >>>>>>> gets copied to primary storage. If we could do it at this point, then we >>>>>>> wouldn't have to worry about fixing the "legacy" situation because it >>>>>>> would >>>>>>> just work out naturally. We would look in the DB to see if the virtual >>>>>>> size >>>>>>> for this template is known and, if not, we could figure out the virtual >>>>>>> size before downloading from (secondary-storage) NFS to primary storage >>>>>>> each time. (Although I'm thinking this would come too late in the >>>>>>> process >>>>>>> because we may have already asked the primary-storage plug-in to create >>>>>>> the >>>>>>> necessary volume.) >>>>>>> >>>>>>> By the way, I'm assuming that a template gets copied to >>>>>>> (secondary-storage) NFS before being copied to primary storage. I'm not >>>>>>> super familiar with how this process works. >>>>>>> >>>>>>> Talk to you later, >>>>>>> Mike >>>>>>> >>>>>>> On Mon, Sep 8, 2014 at 10:59 PM, Will Stevens <wstev...@cloudops.com> >>>>>>> wrote: >>>>>>> >>>>>>> > My two cents on the topic. >>>>>>> > >>>>>>> > Ideally we would save the size in the object store metadata and >>>>>>> > retrieve it from the metadata if it is set. If it is not set in >>>>>>> the >>>>>>> > object store metadata, then when it is fetched, we have to put it >>>>>>> on >>>>>>> > NFS and determine the size (then ideally save the metadata back to >>>>>>> the >>>>>>> > object store) and remove the NFS copy. >>>>>>> > >>>>>>> > This way the NFS copy approach is only ever done once and then the >>>>>>> > data is populated (for backwards compatibility). For all templates >>>>>>> > created after the patch, the metadata would be stored and retrieved >>>>>>> > without the need for the NFS copy. >>>>>>> > >>>>>>> > Is this feasible? >>>>>>> > >>>>>>> > Will >>>>>>> > >>>>>>> > >>>>>>> > *Will STEVENS* >>>>>>> > Lead Developer >>>>>>> > >>>>>>> > *CloudOps* *| *Cloud Solutions Experts >>>>>>> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com >>>>>>> *|* tw >>>>>>> > @CloudOps_ >>>>>>> > >>>>>>> > On Tue, Sep 9, 2014 at 12:48 AM, Mike Tutkowski < >>>>>>> > mike.tutkow...@solidfire.com> wrote: >>>>>>> > >>>>>>> > > Hi Punith, >>>>>>> > > >>>>>>> > > Thanks for putting in time on this! >>>>>>> > > >>>>>>> > > So, option 1 occurs after we download the template to S3 or >>>>>>> Swift. >>>>>>> > > We >>>>>>> > then >>>>>>> > > have to copy it to NFS so that we can determine the virtual size? >>>>>>> > > Relatively slow, but it would work. >>>>>>> > > >>>>>>> > > If we went this route, would we then delete that template from >>>>>>> the >>>>>>> > > NFS share since its immediate purpose was so we could figure out >>>>>>> the >>>>>>> > > virtual size or would we leave it on that share? >>>>>>> > > >>>>>>> > > Option 2 is to allow the user who uploads such a template to >>>>>>> specify >>>>>>> > > the size needed for the root disk (i.e. at least the virtual >>>>>>> size of >>>>>>> > > the template...perhaps larger)? >>>>>>> > > >>>>>>> > > Does that sound like I understood the options? >>>>>>> > > >>>>>>> > > Thanks! >>>>>>> > > Mike >>>>>>> > > >>>>>>> > > On Mon, Sep 8, 2014 at 10:34 PM, Punith S < >>>>>>> punit...@cloudbyte.com> >>>>>>> > wrote: >>>>>>> > > >>>>>>> > > > hi mike, >>>>>>> > > > >>>>>>> > > > i have figured out the issue, in NFS secondary storage the >>>>>>> virtual >>>>>>> > > > size >>>>>>> > > is >>>>>>> > > > been calculated by the VHD Processor by accessing the vhd >>>>>>> template. >>>>>>> > > > >>>>>>> > > > but in case of S3, cloudstack is not able to access the >>>>>>> template >>>>>>> > > > but it only gets to know the physical size of the template! >>>>>>> > > > >>>>>>> > > > in order to solve this, we need to download the template from >>>>>>> S3 >>>>>>> > > > to another secondary nfs storage and access the template to >>>>>>> > > > calculate the virtual size. >>>>>>> > > > >>>>>>> > > > if the above solution seems to be complicated, we shall have a >>>>>>> > > > quick >>>>>>> > fix >>>>>>> > > > where virtual size i.e the size of the SAN volume shall be >>>>>>> > > > created(like small-20g, medium-40g) for people using the >>>>>>> templates from S3. >>>>>>> > > > >>>>>>> > > > any feedback's ? >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > On Tue, Sep 9, 2014 at 6:34 AM, Mike Tutkowski < >>>>>>> > > > mike.tutkow...@solidfire.com> wrote: >>>>>>> > > > >>>>>>> > > >> Hi Punith, >>>>>>> > > >> >>>>>>> > > >> Have you been able to make any progress with regards to this >>>>>>> > > >> Swift/S3 issue? >>>>>>> > > >> >>>>>>> > > >> Thanks! >>>>>>> > > >> Mike >>>>>>> > > >> >>>>>>> > > >> On Wed, Aug 27, 2014 at 7:43 AM, Punith S >>>>>>> > > >> <punit...@cloudbyte.com> >>>>>>> > > wrote: >>>>>>> > > >> >>>>>>> > > >> > hi >>>>>>> > > >> > >>>>>>> > > >> > think i had a timeout problem! >>>>>>> > > >> > on the second try the template has been downloaded to the S3 >>>>>>> > > >> > bucket >>>>>>> > > and >>>>>>> > > >> > the management server shows the status as download complete >>>>>>> > > >> > with >>>>>>> > > >> template >>>>>>> > > >> > size as 1.6G instead of its virtual size 20G. >>>>>>> > > >> > >>>>>>> > > >> > and i see the template's status as Download Complete but it >>>>>>> > > >> > seems it >>>>>>> > > is >>>>>>> > > >> > not getting installed ! refer the attachment >>>>>>> > > >> > >>>>>>> > > >> > can anyone explain the "installing template" after the >>>>>>> download >>>>>>> > > >> completes ? >>>>>>> > > >> > >>>>>>> > > >> > >>>>>>> > > >> > On Wed, Aug 27, 2014 at 9:18 AM, Marcus < >>>>>>> shadow...@gmail.com> >>>>>>> > wrote: >>>>>>> > > >> > >>>>>>> > > >> >> Per Edisons comments about not knowing the image size, >>>>>>> can't >>>>>>> > > >> >> we >>>>>>> > just >>>>>>> > > >> set >>>>>>> > > >> >> some headers and store metadata with the template in S3 to >>>>>>> > > >> >> save the virtual size when the template is registered? I'm >>>>>>> > > >> >> assuming here that the >>>>>>> > SSVM >>>>>>> > > >> does >>>>>>> > > >> >> the work of pulling the template in and uploading to S3. >>>>>>> Or it >>>>>>> > could >>>>>>> > > be >>>>>>> > > >> >> stored in the template table? >>>>>>> > > >> >> On Aug 26, 2014 9:11 PM, "Francois Gaudreault" < >>>>>>> > > >> fgaudrea...@cloudops.com> >>>>>>> > > >> >> wrote: >>>>>>> > > >> >> >>>>>>> > > >> >> > Looks like your SSVM cannot reach Internet properly? >>>>>>> > > >> >> > >>>>>>> > > >> >> > FG >>>>>>> > > >> >> > >>>>>>> > > >> >> > On 2014-08-26, 11:14 AM, Punith S wrote: >>>>>>> > > >> >> > >>>>>>> > > >> >> >> hi francois, >>>>>>> > > >> >> >> >>>>>>> > > >> >> >> since i'm not having a swift setup, i'm using the s3 >>>>>>> bucket. >>>>>>> > > >> >> >> >>>>>>> > > >> >> >> and as you recommended i got the SSVM up with seeded nfs >>>>>>> > storage, >>>>>>> > > >> >> >> >>>>>>> > > >> >> >> post that i removed the nfs secondary storage and added >>>>>>> the >>>>>>> > > >> >> >> S3 >>>>>>> > > with >>>>>>> > > >> >> >> staging nfs store as the new sec storage, since you >>>>>>> cannot >>>>>>> > > >> >> >> have >>>>>>> > > any >>>>>>> > > >> nfs >>>>>>> > > >> >> >> secondary storage while using the S3. >>>>>>> > > >> >> >> >>>>>>> > > >> >> >> on registering the a new template, i'm getting template >>>>>>> > > >> >> >> status >>>>>>> > > >> >> as*Unable >>>>>>> > > >> >> >> to execute HTTP request: No route to host* in >>>>>>> > > >> >> >> managementserver.log >>>>>>> > > >> >> >> >>>>>>> > > >> >> >> 2014-08-26 20:41:07,502 DEBUG >>>>>>> [o.a.c.s.RemoteHostEndPoint] >>>>>>> > > >> >> >> (Timer-24:ctx-b68380cd) Sending command >>>>>>> > > >> org.apache.cloudstack.storage. >>>>>>> > > >> >> >> command.DownloadProgressCommand to host: 10 >>>>>>> > > >> >> >> 2014-08-26 20:41:07,507 DEBUG [c.c.a.t.Request] >>>>>>> > > >> (Timer-24:ctx-b68380cd) >>>>>>> > > >> >> >> Seq 10-5684105679694996125: Sending { Cmd , MgmtId: >>>>>>> > 52242179434, >>>>>>> > > >> via: >>>>>>> > > >> >> >> 10(s-142-VM), Ver: v1, Flags: 100011, >>>>>>> [{"org.apache.cloudstack. >>>>>>> > > >> >> >> >>>>>>> > > >> >>>>>>> storage.command.DownloadProgressCommand":{"jobId":"d43a17c9-3b03- >>>>>>> > > >> 4ff9- >>>>>>> > > >> >> >> 8906-e1d155981e86","request":"GET_STATUS","hvm":true," >>>>>>> > > >> >> >> >>>>>>> > > >> >>>>>>> description":"centext","maxDownloadSizeInBytes":53687091200,"id":209," >>>>>>> > > >> >> >> resourceType":"TEMPLATE","installPath":"template/tmpl/2/ >>>>>>> > > >> >> >> >>>>>>> 209/209-2-b624436c-5f37-30d4-8eaf-81582eb0d39d","_store":{" >>>>>>> > > >> >> >> >>>>>>> com.cloud.agent.api.to.S3TO":{"id":14,"uuid":"e4afd7bb-39ea >>>>>>> > > >> >> >> - >>>>>>> 4128-ab93-f8a09b1d5e03","bucketName":"test-cloudstack", >>>>>>> > > >> >> >> "httpsFlag":false,"created":"Aug 26, 2014 8:16:24 >>>>>>> > > >> >> PM","enableRRS":false," >>>>>>> > > >> >> >> maxSingleUploadSizeInBytes":5368709120}},"url":"http:// >>>>>>> > > >> >> >> >>>>>>> download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz >>>>>>> > > >> >> >> 2 >>>>>>> > > >> >> >> >>>>>>> > > >> >> >>>>>>> > > >> >>>>>>> > > >>>>>>> > >>>>>>> ","format":"VHD","accountId":2,"name":"209-2-b624436c-5f37-30d4-8eaf-8 >>>>>>> > 1582eb0d39d","wait":0}}] >>>>>>> > > >> >> >> } >>>>>>> > > >> >> >> 2014-08-26 20:41:07,556 DEBUG [c.c.a.t.Request] >>>>>>> > > >> >> >> (AgentManager-Handler-10:null) Seq >>>>>>> 10-5684105679694996125: >>>>>>> > > >> >> Processing: { >>>>>>> > > >> >> >> Ans: , MgmtId: 52242179434, via: 10, Ver: v1, Flags: 10, >>>>>>> > > >> >> >> [{"com.cloud.agent.api.storage.DownloadAnswer":{" >>>>>>> > > >> >> >> jobId":"d43a17c9-3b03-4ff9-8906-e1d155981e86"," >>>>>>> > > >> >> >> downloadPct":0,"errorString":"No route to >>>>>>> > host","downloadStatus":" >>>>>>> > > >> >> >> >>>>>>> DOWNLOAD_ERROR","installPath":"template/tmpl/2/209/209-2- >>>>>>> > > >> >> >> b624436c-5f37-30d4-8eaf-81582eb0d39d","templateSize": >>>>>>> > > >> >> >> 0,"templatePhySicalSize":0,"result":true,"details":"No >>>>>>> > > >> >> >> route to host","wait":0}}] } >>>>>>> > > >> >> >> >>>>>>> > > >> >> >> but i don't see any logging happening in secondary >>>>>>> storage >>>>>>> > > >> >> >> vm's >>>>>>> > > >> >> cloud.log >>>>>>> > > >> >> >> >>>>>>> > > >> >> >> not sure this error is happening due to S3! >>>>>>> > > >> >> >> >>>>>>> > > >> >> >> >>>>>>> > > >> >> >> thanks! >>>>>>> > > >> >> >> >>>>>>> > > >> >> > >>>>>>> > > >> >> > >>>>>>> > > >> >> > -- >>>>>>> > > >> >> > Francois Gaudreault >>>>>>> > > >> >> > Gestionnaire de Produit | Product Manager - Cloud >>>>>>> Platform & >>>>>>> > > Services >>>>>>> > > >> >> > t:514-629-6775 >>>>>>> > > >> >> > >>>>>>> > > >> >> > CloudOps Votre partenaire infonuagique | Cloud Solutions >>>>>>> > > >> >> > Experts >>>>>>> > > >> >> > 420 rue Guy | Montreal | Quebec | H3J 1S6 >>>>>>> > > >> >> > w: cloudops.com | tw: @CloudOps_ >>>>>>> > > >> >> > >>>>>>> > > >> >> > >>>>>>> > > >> >> >>>>>>> > > >> > >>>>>>> > > >> > >>>>>>> > > >> > >>>>>>> > > >> > -- >>>>>>> > > >> > regards, >>>>>>> > > >> > >>>>>>> > > >> > punith s >>>>>>> > > >> > cloudbyte.com >>>>>>> > > >> > >>>>>>> > > >> >>>>>>> > > >> >>>>>>> > > >> >>>>>>> > > >> -- >>>>>>> > > >> *Mike Tutkowski* >>>>>>> > > >> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>> > > >> e: mike.tutkow...@solidfire.com >>>>>>> > > >> o: 303.746.7302 >>>>>>> > > >> Advancing the way the world uses the cloud >>>>>>> > > >> <http://solidfire.com/solution/overview/?video=play>*™* >>>>>>> > > >> >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > -- >>>>>>> > > > regards, >>>>>>> > > > >>>>>>> > > > punith s >>>>>>> > > > cloudbyte.com >>>>>>> > > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > -- >>>>>>> > > *Mike Tutkowski* >>>>>>> > > *Senior CloudStack Developer, SolidFire Inc.* >>>>>>> > > e: mike.tutkow...@solidfire.com >>>>>>> > > o: 303.746.7302 >>>>>>> > > Advancing the way the world uses the cloud >>>>>>> > > <http://solidfire.com/solution/overview/?video=play>*™* >>>>>>> > > >>>>>>> > >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Mike Tutkowski* >>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>>> e: mike.tutkow...@solidfire.com >>>>>>> o: 303.746.7302 >>>>>>> Advancing the way the world uses the cloud >>>>>>> <http://solidfire.com/solution/overview/?video=play>*™* >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Mike Tutkowski* >>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>>> e: mike.tutkow...@solidfire.com >>>>>> o: 303.746.7302 >>>>>> Advancing the way the world uses the cloud >>>>>> <http://solidfire.com/solution/overview/?video=play>*™* >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Mike Tutkowski* >>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>> e: mike.tutkow...@solidfire.com >>>>> o: 303.746.7302 >>>>> Advancing the way the world uses the cloud >>>>> <http://solidfire.com/solution/overview/?video=play>*™* >>>>> >>>> >>>> >>>> >>>> -- >>>> regards, >>>> >>>> punith s >>>> cloudbyte.com >>>> >>> >>> >>> >>> -- >>> *Mike Tutkowski* >>> *Senior CloudStack Developer, SolidFire Inc.* >>> e: mike.tutkow...@solidfire.com >>> o: 303.746.7302 >>> Advancing the way the world uses the cloud >>> <http://solidfire.com/solution/overview/?video=play>*™* >>> >> >> > > > -- > *Mike Tutkowski* > *Senior CloudStack Developer, SolidFire Inc.* > e: mike.tutkow...@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the cloud > <http://solidfire.com/solution/overview/?video=play>*™* > -- regards, punith s cloudbyte.com
