Thanks for the info, Punith! Does that mean you are fixing the issue via Option2?
On Tue, Sep 9, 2014 at 7:52 AM, Punith S <punit...@cloudbyte.com> wrote: > yes mike, > > w.r.t to option 1: > it will be like creating a VM w.r.t ISO, where admin will have to specify > the os disk(ROOT) disk size. > > for option 2: > i have figured out the issue, post downloading the template to S3, > cloudstack will again download the template from S3 to staging nfs store. > here we need to access the file and process it with VHD processor in order > to calculate the virtualsize but we are skipping this process, > hence the virtual size is not being calculated while using the S3 or swift. > > the templates already present in the staging nfs storage cannot be applied > to this process. > > for option 3: > it's convenient to calculate the template virtual size while it is being > copied from s3 to staged nfs store instead of staged nfs to primary, > since admin might be using more than one primary stores. > > i'm fixing the issue, will post the patch ASAP for 4.5.snapshot. > > thanks! > > On Tue, Sep 9, 2014 at 11:13 AM, Mike Tutkowski < > mike.tutkow...@solidfire.com> wrote: > >> By the way, for anyone new to this issue, this is what we're referring to >> here: >> >> https://issues.apache.org/jira/browse/CLOUDSTACK-7406 >> >> On Mon, Sep 8, 2014 at 11:41 PM, Mike Tutkowski < >> mike.tutkow...@solidfire.com> wrote: >> >>> Great :) Then a question might be, "Is it too late in the game to >>> interrogate the template to discover its virtual size if we're just about >>> to copy the template to primary storage?" >>> >>> If it's not, this might be the place to run the logic to figure out the >>> virtual size. >>> >>> Really, there are three big possibilities: >>> >>> 1) Just ask the end user to provide the virtual size (not commenting >>> here on what happens for already-uploaded templates) >>> >>> or >>> >>> 2) Figure out the virtual size when the template is copied from object >>> storage to secondary storage and update the DB with this info (not sure >>> what happens if the template has already been copied to (secondary-storage) >>> NFS because it was used before) >>> >>> or >>> >>> 3) Figure out the virtual size when the template is about to be copied >>> from secondary storage to primary storage >>> >>> On Mon, Sep 8, 2014 at 11:35 PM, Sanjeev Neelarapu < >>> sanjeev.neelar...@citrix.com> wrote: >>> >>>> Mike, >>>> >>>> You are right. Template gets copied to (secondary-storage) NFS before >>>> being copied to primary storage >>>> >>>> -Sanjeev >>>> >>>> -----Original Message----- >>>> From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] >>>> Sent: Tuesday, September 09, 2014 10:55 AM >>>> To: dev@cloudstack.apache.org >>>> Cc: Punith S; Francois Gaudreault >>>> Subject: Re: S3/Swift Problem around Virtual Size >>>> >>>> Hi Will, >>>> >>>> Thanks for the input! >>>> >>>> I like the idea of storing the virtual size as metadata in S3 or Swift >>>> although this could require that the end user provide this value when >>>> uploading the template. >>>> >>>> However, if we have the ability to determine the virtual size of the >>>> template after it gets downloaded to (secondary-storage) NFS and we're able >>>> to update the database with this info, then it would seem we would never >>>> need to ask the user for this value. >>>> >>>> Either way, the tricky part might be if the template in object storage >>>> has already been downloaded to (secondary-storage) NFS (say it was used >>>> before). In this case, we won't need to download it to (secondary-storage) >>>> NFS again (at least not in the same zone), so we won't have an easy >>>> opportunity to figure out the virtual size upon download from object >>>> storage. >>>> >>>> I wonder if it's too late in this process if we figured out the virtual >>>> size before the copied template (now on (secondary-storage) NFS) gets >>>> copied to primary storage. If we could do it at this point, then we >>>> wouldn't have to worry about fixing the "legacy" situation because it would >>>> just work out naturally. We would look in the DB to see if the virtual size >>>> for this template is known and, if not, we could figure out the virtual >>>> size before downloading from (secondary-storage) NFS to primary storage >>>> each time. (Although I'm thinking this would come too late in the process >>>> because we may have already asked the primary-storage plug-in to create the >>>> necessary volume.) >>>> >>>> By the way, I'm assuming that a template gets copied to >>>> (secondary-storage) NFS before being copied to primary storage. I'm not >>>> super familiar with how this process works. >>>> >>>> Talk to you later, >>>> Mike >>>> >>>> On Mon, Sep 8, 2014 at 10:59 PM, Will Stevens <wstev...@cloudops.com> >>>> wrote: >>>> >>>> > My two cents on the topic. >>>> > >>>> > Ideally we would save the size in the object store metadata and >>>> > retrieve it from the metadata if it is set. If it is not set in the >>>> > object store metadata, then when it is fetched, we have to put it on >>>> > NFS and determine the size (then ideally save the metadata back to the >>>> > object store) and remove the NFS copy. >>>> > >>>> > This way the NFS copy approach is only ever done once and then the >>>> > data is populated (for backwards compatibility). For all templates >>>> > created after the patch, the metadata would be stored and retrieved >>>> > without the need for the NFS copy. >>>> > >>>> > Is this feasible? >>>> > >>>> > Will >>>> > >>>> > >>>> > *Will STEVENS* >>>> > Lead Developer >>>> > >>>> > *CloudOps* *| *Cloud Solutions Experts >>>> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw >>>> > @CloudOps_ >>>> > >>>> > On Tue, Sep 9, 2014 at 12:48 AM, Mike Tutkowski < >>>> > mike.tutkow...@solidfire.com> wrote: >>>> > >>>> > > Hi Punith, >>>> > > >>>> > > Thanks for putting in time on this! >>>> > > >>>> > > So, option 1 occurs after we download the template to S3 or Swift. >>>> > > We >>>> > then >>>> > > have to copy it to NFS so that we can determine the virtual size? >>>> > > Relatively slow, but it would work. >>>> > > >>>> > > If we went this route, would we then delete that template from the >>>> > > NFS share since its immediate purpose was so we could figure out the >>>> > > virtual size or would we leave it on that share? >>>> > > >>>> > > Option 2 is to allow the user who uploads such a template to specify >>>> > > the size needed for the root disk (i.e. at least the virtual size of >>>> > > the template...perhaps larger)? >>>> > > >>>> > > Does that sound like I understood the options? >>>> > > >>>> > > Thanks! >>>> > > Mike >>>> > > >>>> > > On Mon, Sep 8, 2014 at 10:34 PM, Punith S <punit...@cloudbyte.com> >>>> > wrote: >>>> > > >>>> > > > hi mike, >>>> > > > >>>> > > > i have figured out the issue, in NFS secondary storage the virtual >>>> > > > size >>>> > > is >>>> > > > been calculated by the VHD Processor by accessing the vhd >>>> template. >>>> > > > >>>> > > > but in case of S3, cloudstack is not able to access the template >>>> > > > but it only gets to know the physical size of the template! >>>> > > > >>>> > > > in order to solve this, we need to download the template from S3 >>>> > > > to another secondary nfs storage and access the template to >>>> > > > calculate the virtual size. >>>> > > > >>>> > > > if the above solution seems to be complicated, we shall have a >>>> > > > quick >>>> > fix >>>> > > > where virtual size i.e the size of the SAN volume shall be >>>> > > > created(like small-20g, medium-40g) for people using the >>>> templates from S3. >>>> > > > >>>> > > > any feedback's ? >>>> > > > >>>> > > > >>>> > > > >>>> > > > On Tue, Sep 9, 2014 at 6:34 AM, Mike Tutkowski < >>>> > > > mike.tutkow...@solidfire.com> wrote: >>>> > > > >>>> > > >> Hi Punith, >>>> > > >> >>>> > > >> Have you been able to make any progress with regards to this >>>> > > >> Swift/S3 issue? >>>> > > >> >>>> > > >> Thanks! >>>> > > >> Mike >>>> > > >> >>>> > > >> On Wed, Aug 27, 2014 at 7:43 AM, Punith S >>>> > > >> <punit...@cloudbyte.com> >>>> > > wrote: >>>> > > >> >>>> > > >> > hi >>>> > > >> > >>>> > > >> > think i had a timeout problem! >>>> > > >> > on the second try the template has been downloaded to the S3 >>>> > > >> > bucket >>>> > > and >>>> > > >> > the management server shows the status as download complete >>>> > > >> > with >>>> > > >> template >>>> > > >> > size as 1.6G instead of its virtual size 20G. >>>> > > >> > >>>> > > >> > and i see the template's status as Download Complete but it >>>> > > >> > seems it >>>> > > is >>>> > > >> > not getting installed ! refer the attachment >>>> > > >> > >>>> > > >> > can anyone explain the "installing template" after the download >>>> > > >> completes ? >>>> > > >> > >>>> > > >> > >>>> > > >> > On Wed, Aug 27, 2014 at 9:18 AM, Marcus <shadow...@gmail.com> >>>> > wrote: >>>> > > >> > >>>> > > >> >> Per Edisons comments about not knowing the image size, can't >>>> > > >> >> we >>>> > just >>>> > > >> set >>>> > > >> >> some headers and store metadata with the template in S3 to >>>> > > >> >> save the virtual size when the template is registered? I'm >>>> > > >> >> assuming here that the >>>> > SSVM >>>> > > >> does >>>> > > >> >> the work of pulling the template in and uploading to S3. Or it >>>> > could >>>> > > be >>>> > > >> >> stored in the template table? >>>> > > >> >> On Aug 26, 2014 9:11 PM, "Francois Gaudreault" < >>>> > > >> fgaudrea...@cloudops.com> >>>> > > >> >> wrote: >>>> > > >> >> >>>> > > >> >> > Looks like your SSVM cannot reach Internet properly? >>>> > > >> >> > >>>> > > >> >> > FG >>>> > > >> >> > >>>> > > >> >> > On 2014-08-26, 11:14 AM, Punith S wrote: >>>> > > >> >> > >>>> > > >> >> >> hi francois, >>>> > > >> >> >> >>>> > > >> >> >> since i'm not having a swift setup, i'm using the s3 >>>> bucket. >>>> > > >> >> >> >>>> > > >> >> >> and as you recommended i got the SSVM up with seeded nfs >>>> > storage, >>>> > > >> >> >> >>>> > > >> >> >> post that i removed the nfs secondary storage and added the >>>> > > >> >> >> S3 >>>> > > with >>>> > > >> >> >> staging nfs store as the new sec storage, since you cannot >>>> > > >> >> >> have >>>> > > any >>>> > > >> nfs >>>> > > >> >> >> secondary storage while using the S3. >>>> > > >> >> >> >>>> > > >> >> >> on registering the a new template, i'm getting template >>>> > > >> >> >> status >>>> > > >> >> as*Unable >>>> > > >> >> >> to execute HTTP request: No route to host* in >>>> > > >> >> >> managementserver.log >>>> > > >> >> >> >>>> > > >> >> >> 2014-08-26 20:41:07,502 DEBUG [o.a.c.s.RemoteHostEndPoint] >>>> > > >> >> >> (Timer-24:ctx-b68380cd) Sending command >>>> > > >> org.apache.cloudstack.storage. >>>> > > >> >> >> command.DownloadProgressCommand to host: 10 >>>> > > >> >> >> 2014-08-26 20:41:07,507 DEBUG [c.c.a.t.Request] >>>> > > >> (Timer-24:ctx-b68380cd) >>>> > > >> >> >> Seq 10-5684105679694996125: Sending { Cmd , MgmtId: >>>> > 52242179434, >>>> > > >> via: >>>> > > >> >> >> 10(s-142-VM), Ver: v1, Flags: 100011, >>>> [{"org.apache.cloudstack. >>>> > > >> >> >> >>>> > > >> storage.command.DownloadProgressCommand":{"jobId":"d43a17c9-3b03- >>>> > > >> 4ff9- >>>> > > >> >> >> 8906-e1d155981e86","request":"GET_STATUS","hvm":true," >>>> > > >> >> >> >>>> > > >> >>>> description":"centext","maxDownloadSizeInBytes":53687091200,"id":209," >>>> > > >> >> >> resourceType":"TEMPLATE","installPath":"template/tmpl/2/ >>>> > > >> >> >> 209/209-2-b624436c-5f37-30d4-8eaf-81582eb0d39d","_store":{" >>>> > > >> >> >> com.cloud.agent.api.to.S3TO":{"id":14,"uuid":"e4afd7bb-39ea >>>> > > >> >> >> - 4128-ab93-f8a09b1d5e03","bucketName":"test-cloudstack", >>>> > > >> >> >> "httpsFlag":false,"created":"Aug 26, 2014 8:16:24 >>>> > > >> >> PM","enableRRS":false," >>>> > > >> >> >> maxSingleUploadSizeInBytes":5368709120}},"url":"http:// >>>> > > >> >> >> >>>> download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz >>>> > > >> >> >> 2 >>>> > > >> >> >> >>>> > > >> >> >>>> > > >> >>>> > > >>>> > ","format":"VHD","accountId":2,"name":"209-2-b624436c-5f37-30d4-8eaf-8 >>>> > 1582eb0d39d","wait":0}}] >>>> > > >> >> >> } >>>> > > >> >> >> 2014-08-26 20:41:07,556 DEBUG [c.c.a.t.Request] >>>> > > >> >> >> (AgentManager-Handler-10:null) Seq 10-5684105679694996125: >>>> > > >> >> Processing: { >>>> > > >> >> >> Ans: , MgmtId: 52242179434, via: 10, Ver: v1, Flags: 10, >>>> > > >> >> >> [{"com.cloud.agent.api.storage.DownloadAnswer":{" >>>> > > >> >> >> jobId":"d43a17c9-3b03-4ff9-8906-e1d155981e86"," >>>> > > >> >> >> downloadPct":0,"errorString":"No route to >>>> > host","downloadStatus":" >>>> > > >> >> >> DOWNLOAD_ERROR","installPath":"template/tmpl/2/209/209-2- >>>> > > >> >> >> b624436c-5f37-30d4-8eaf-81582eb0d39d","templateSize": >>>> > > >> >> >> 0,"templatePhySicalSize":0,"result":true,"details":"No >>>> > > >> >> >> route to host","wait":0}}] } >>>> > > >> >> >> >>>> > > >> >> >> but i don't see any logging happening in secondary storage >>>> > > >> >> >> vm's >>>> > > >> >> cloud.log >>>> > > >> >> >> >>>> > > >> >> >> not sure this error is happening due to S3! >>>> > > >> >> >> >>>> > > >> >> >> >>>> > > >> >> >> thanks! >>>> > > >> >> >> >>>> > > >> >> > >>>> > > >> >> > >>>> > > >> >> > -- >>>> > > >> >> > Francois Gaudreault >>>> > > >> >> > Gestionnaire de Produit | Product Manager - Cloud Platform & >>>> > > Services >>>> > > >> >> > t:514-629-6775 >>>> > > >> >> > >>>> > > >> >> > CloudOps Votre partenaire infonuagique | Cloud Solutions >>>> > > >> >> > Experts >>>> > > >> >> > 420 rue Guy | Montreal | Quebec | H3J 1S6 >>>> > > >> >> > w: cloudops.com | tw: @CloudOps_ >>>> > > >> >> > >>>> > > >> >> > >>>> > > >> >> >>>> > > >> > >>>> > > >> > >>>> > > >> > >>>> > > >> > -- >>>> > > >> > regards, >>>> > > >> > >>>> > > >> > punith s >>>> > > >> > cloudbyte.com >>>> > > >> > >>>> > > >> >>>> > > >> >>>> > > >> >>>> > > >> -- >>>> > > >> *Mike Tutkowski* >>>> > > >> *Senior CloudStack Developer, SolidFire Inc.* >>>> > > >> e: mike.tutkow...@solidfire.com >>>> > > >> o: 303.746.7302 >>>> > > >> Advancing the way the world uses the cloud >>>> > > >> <http://solidfire.com/solution/overview/?video=play>*™* >>>> > > >> >>>> > > > >>>> > > > >>>> > > > >>>> > > > -- >>>> > > > regards, >>>> > > > >>>> > > > punith s >>>> > > > cloudbyte.com >>>> > > > >>>> > > >>>> > > >>>> > > >>>> > > -- >>>> > > *Mike Tutkowski* >>>> > > *Senior CloudStack Developer, SolidFire Inc.* >>>> > > e: mike.tutkow...@solidfire.com >>>> > > o: 303.746.7302 >>>> > > Advancing the way the world uses the cloud >>>> > > <http://solidfire.com/solution/overview/?video=play>*™* >>>> > > >>>> > >>>> >>>> >>>> >>>> -- >>>> *Mike Tutkowski* >>>> *Senior CloudStack Developer, SolidFire Inc.* >>>> e: mike.tutkow...@solidfire.com >>>> o: 303.746.7302 >>>> Advancing the way the world uses the cloud >>>> <http://solidfire.com/solution/overview/?video=play>*™* >>>> >>> >>> >>> >>> -- >>> *Mike Tutkowski* >>> *Senior CloudStack Developer, SolidFire Inc.* >>> e: mike.tutkow...@solidfire.com >>> o: 303.746.7302 >>> Advancing the way the world uses the cloud >>> <http://solidfire.com/solution/overview/?video=play>*™* >>> >> >> >> >> -- >> *Mike Tutkowski* >> *Senior CloudStack Developer, SolidFire Inc.* >> e: mike.tutkow...@solidfire.com >> o: 303.746.7302 >> Advancing the way the world uses the cloud >> <http://solidfire.com/solution/overview/?video=play>*™* >> > > > > -- > regards, > > punith s > cloudbyte.com > -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud <http://solidfire.com/solution/overview/?video=play>*™*
