That's great Punith :) Thanks for handling this one. I am not too worried about the option, as long as it fixes SF integration for 4.5 :)
FG On Tue, Sep 9, 2014 at 11:55 AM, Mike Tutkowski < mike.tutkow...@solidfire.com> wrote: > Thanks for the info, Punith! > > Does that mean you are fixing the issue via Option2? > > On Tue, Sep 9, 2014 at 7:52 AM, Punith S <punit...@cloudbyte.com> wrote: > >> yes mike, >> >> w.r.t to option 1: >> it will be like creating a VM w.r.t ISO, where admin will have to specify >> the os disk(ROOT) disk size. >> >> for option 2: >> i have figured out the issue, post downloading the template to S3, >> cloudstack will again download the template from S3 to staging nfs store. >> here we need to access the file and process it with VHD processor in >> order to calculate the virtualsize but we are skipping this process, >> hence the virtual size is not being calculated while using the S3 or >> swift. >> >> the templates already present in the staging nfs storage cannot be >> applied to this process. >> >> for option 3: >> it's convenient to calculate the template virtual size while it is being >> copied from s3 to staged nfs store instead of staged nfs to primary, >> since admin might be using more than one primary stores. >> >> i'm fixing the issue, will post the patch ASAP for 4.5.snapshot. >> >> thanks! >> >> On Tue, Sep 9, 2014 at 11:13 AM, Mike Tutkowski < >> mike.tutkow...@solidfire.com> wrote: >> >>> By the way, for anyone new to this issue, this is what we're referring >>> to here: >>> >>> https://issues.apache.org/jira/browse/CLOUDSTACK-7406 >>> >>> On Mon, Sep 8, 2014 at 11:41 PM, Mike Tutkowski < >>> mike.tutkow...@solidfire.com> wrote: >>> >>>> Great :) Then a question might be, "Is it too late in the game to >>>> interrogate the template to discover its virtual size if we're just about >>>> to copy the template to primary storage?" >>>> >>>> If it's not, this might be the place to run the logic to figure out the >>>> virtual size. >>>> >>>> Really, there are three big possibilities: >>>> >>>> 1) Just ask the end user to provide the virtual size (not commenting >>>> here on what happens for already-uploaded templates) >>>> >>>> or >>>> >>>> 2) Figure out the virtual size when the template is copied from object >>>> storage to secondary storage and update the DB with this info (not sure >>>> what happens if the template has already been copied to (secondary-storage) >>>> NFS because it was used before) >>>> >>>> or >>>> >>>> 3) Figure out the virtual size when the template is about to be copied >>>> from secondary storage to primary storage >>>> >>>> On Mon, Sep 8, 2014 at 11:35 PM, Sanjeev Neelarapu < >>>> sanjeev.neelar...@citrix.com> wrote: >>>> >>>>> Mike, >>>>> >>>>> You are right. Template gets copied to (secondary-storage) NFS before >>>>> being copied to primary storage >>>>> >>>>> -Sanjeev >>>>> >>>>> -----Original Message----- >>>>> From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] >>>>> Sent: Tuesday, September 09, 2014 10:55 AM >>>>> To: dev@cloudstack.apache.org >>>>> Cc: Punith S; Francois Gaudreault >>>>> Subject: Re: S3/Swift Problem around Virtual Size >>>>> >>>>> Hi Will, >>>>> >>>>> Thanks for the input! >>>>> >>>>> I like the idea of storing the virtual size as metadata in S3 or Swift >>>>> although this could require that the end user provide this value when >>>>> uploading the template. >>>>> >>>>> However, if we have the ability to determine the virtual size of the >>>>> template after it gets downloaded to (secondary-storage) NFS and we're >>>>> able >>>>> to update the database with this info, then it would seem we would never >>>>> need to ask the user for this value. >>>>> >>>>> Either way, the tricky part might be if the template in object storage >>>>> has already been downloaded to (secondary-storage) NFS (say it was used >>>>> before). In this case, we won't need to download it to (secondary-storage) >>>>> NFS again (at least not in the same zone), so we won't have an easy >>>>> opportunity to figure out the virtual size upon download from object >>>>> storage. >>>>> >>>>> I wonder if it's too late in this process if we figured out the >>>>> virtual size before the copied template (now on (secondary-storage) NFS) >>>>> gets copied to primary storage. If we could do it at this point, then we >>>>> wouldn't have to worry about fixing the "legacy" situation because it >>>>> would >>>>> just work out naturally. We would look in the DB to see if the virtual >>>>> size >>>>> for this template is known and, if not, we could figure out the virtual >>>>> size before downloading from (secondary-storage) NFS to primary storage >>>>> each time. (Although I'm thinking this would come too late in the process >>>>> because we may have already asked the primary-storage plug-in to create >>>>> the >>>>> necessary volume.) >>>>> >>>>> By the way, I'm assuming that a template gets copied to >>>>> (secondary-storage) NFS before being copied to primary storage. I'm not >>>>> super familiar with how this process works. >>>>> >>>>> Talk to you later, >>>>> Mike >>>>> >>>>> On Mon, Sep 8, 2014 at 10:59 PM, Will Stevens <wstev...@cloudops.com> >>>>> wrote: >>>>> >>>>> > My two cents on the topic. >>>>> > >>>>> > Ideally we would save the size in the object store metadata and >>>>> > retrieve it from the metadata if it is set. If it is not set in the >>>>> > object store metadata, then when it is fetched, we have to put it on >>>>> > NFS and determine the size (then ideally save the metadata back to >>>>> the >>>>> > object store) and remove the NFS copy. >>>>> > >>>>> > This way the NFS copy approach is only ever done once and then the >>>>> > data is populated (for backwards compatibility). For all templates >>>>> > created after the patch, the metadata would be stored and retrieved >>>>> > without the need for the NFS copy. >>>>> > >>>>> > Is this feasible? >>>>> > >>>>> > Will >>>>> > >>>>> > >>>>> > *Will STEVENS* >>>>> > Lead Developer >>>>> > >>>>> > *CloudOps* *| *Cloud Solutions Experts >>>>> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* >>>>> tw >>>>> > @CloudOps_ >>>>> > >>>>> > On Tue, Sep 9, 2014 at 12:48 AM, Mike Tutkowski < >>>>> > mike.tutkow...@solidfire.com> wrote: >>>>> > >>>>> > > Hi Punith, >>>>> > > >>>>> > > Thanks for putting in time on this! >>>>> > > >>>>> > > So, option 1 occurs after we download the template to S3 or Swift. >>>>> > > We >>>>> > then >>>>> > > have to copy it to NFS so that we can determine the virtual size? >>>>> > > Relatively slow, but it would work. >>>>> > > >>>>> > > If we went this route, would we then delete that template from the >>>>> > > NFS share since its immediate purpose was so we could figure out >>>>> the >>>>> > > virtual size or would we leave it on that share? >>>>> > > >>>>> > > Option 2 is to allow the user who uploads such a template to >>>>> specify >>>>> > > the size needed for the root disk (i.e. at least the virtual size >>>>> of >>>>> > > the template...perhaps larger)? >>>>> > > >>>>> > > Does that sound like I understood the options? >>>>> > > >>>>> > > Thanks! >>>>> > > Mike >>>>> > > >>>>> > > On Mon, Sep 8, 2014 at 10:34 PM, Punith S <punit...@cloudbyte.com> >>>>> > wrote: >>>>> > > >>>>> > > > hi mike, >>>>> > > > >>>>> > > > i have figured out the issue, in NFS secondary storage the >>>>> virtual >>>>> > > > size >>>>> > > is >>>>> > > > been calculated by the VHD Processor by accessing the vhd >>>>> template. >>>>> > > > >>>>> > > > but in case of S3, cloudstack is not able to access the template >>>>> > > > but it only gets to know the physical size of the template! >>>>> > > > >>>>> > > > in order to solve this, we need to download the template from S3 >>>>> > > > to another secondary nfs storage and access the template to >>>>> > > > calculate the virtual size. >>>>> > > > >>>>> > > > if the above solution seems to be complicated, we shall have a >>>>> > > > quick >>>>> > fix >>>>> > > > where virtual size i.e the size of the SAN volume shall be >>>>> > > > created(like small-20g, medium-40g) for people using the >>>>> templates from S3. >>>>> > > > >>>>> > > > any feedback's ? >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > On Tue, Sep 9, 2014 at 6:34 AM, Mike Tutkowski < >>>>> > > > mike.tutkow...@solidfire.com> wrote: >>>>> > > > >>>>> > > >> Hi Punith, >>>>> > > >> >>>>> > > >> Have you been able to make any progress with regards to this >>>>> > > >> Swift/S3 issue? >>>>> > > >> >>>>> > > >> Thanks! >>>>> > > >> Mike >>>>> > > >> >>>>> > > >> On Wed, Aug 27, 2014 at 7:43 AM, Punith S >>>>> > > >> <punit...@cloudbyte.com> >>>>> > > wrote: >>>>> > > >> >>>>> > > >> > hi >>>>> > > >> > >>>>> > > >> > think i had a timeout problem! >>>>> > > >> > on the second try the template has been downloaded to the S3 >>>>> > > >> > bucket >>>>> > > and >>>>> > > >> > the management server shows the status as download complete >>>>> > > >> > with >>>>> > > >> template >>>>> > > >> > size as 1.6G instead of its virtual size 20G. >>>>> > > >> > >>>>> > > >> > and i see the template's status as Download Complete but it >>>>> > > >> > seems it >>>>> > > is >>>>> > > >> > not getting installed ! refer the attachment >>>>> > > >> > >>>>> > > >> > can anyone explain the "installing template" after the >>>>> download >>>>> > > >> completes ? >>>>> > > >> > >>>>> > > >> > >>>>> > > >> > On Wed, Aug 27, 2014 at 9:18 AM, Marcus <shadow...@gmail.com> >>>>> > wrote: >>>>> > > >> > >>>>> > > >> >> Per Edisons comments about not knowing the image size, can't >>>>> > > >> >> we >>>>> > just >>>>> > > >> set >>>>> > > >> >> some headers and store metadata with the template in S3 to >>>>> > > >> >> save the virtual size when the template is registered? I'm >>>>> > > >> >> assuming here that the >>>>> > SSVM >>>>> > > >> does >>>>> > > >> >> the work of pulling the template in and uploading to S3. Or >>>>> it >>>>> > could >>>>> > > be >>>>> > > >> >> stored in the template table? >>>>> > > >> >> On Aug 26, 2014 9:11 PM, "Francois Gaudreault" < >>>>> > > >> fgaudrea...@cloudops.com> >>>>> > > >> >> wrote: >>>>> > > >> >> >>>>> > > >> >> > Looks like your SSVM cannot reach Internet properly? >>>>> > > >> >> > >>>>> > > >> >> > FG >>>>> > > >> >> > >>>>> > > >> >> > On 2014-08-26, 11:14 AM, Punith S wrote: >>>>> > > >> >> > >>>>> > > >> >> >> hi francois, >>>>> > > >> >> >> >>>>> > > >> >> >> since i'm not having a swift setup, i'm using the s3 >>>>> bucket. >>>>> > > >> >> >> >>>>> > > >> >> >> and as you recommended i got the SSVM up with seeded nfs >>>>> > storage, >>>>> > > >> >> >> >>>>> > > >> >> >> post that i removed the nfs secondary storage and added >>>>> the >>>>> > > >> >> >> S3 >>>>> > > with >>>>> > > >> >> >> staging nfs store as the new sec storage, since you cannot >>>>> > > >> >> >> have >>>>> > > any >>>>> > > >> nfs >>>>> > > >> >> >> secondary storage while using the S3. >>>>> > > >> >> >> >>>>> > > >> >> >> on registering the a new template, i'm getting template >>>>> > > >> >> >> status >>>>> > > >> >> as*Unable >>>>> > > >> >> >> to execute HTTP request: No route to host* in >>>>> > > >> >> >> managementserver.log >>>>> > > >> >> >> >>>>> > > >> >> >> 2014-08-26 20:41:07,502 DEBUG [o.a.c.s.RemoteHostEndPoint] >>>>> > > >> >> >> (Timer-24:ctx-b68380cd) Sending command >>>>> > > >> org.apache.cloudstack.storage. >>>>> > > >> >> >> command.DownloadProgressCommand to host: 10 >>>>> > > >> >> >> 2014-08-26 20:41:07,507 DEBUG [c.c.a.t.Request] >>>>> > > >> (Timer-24:ctx-b68380cd) >>>>> > > >> >> >> Seq 10-5684105679694996125: Sending { Cmd , MgmtId: >>>>> > 52242179434, >>>>> > > >> via: >>>>> > > >> >> >> 10(s-142-VM), Ver: v1, Flags: 100011, >>>>> [{"org.apache.cloudstack. >>>>> > > >> >> >> >>>>> > > >> >>>>> storage.command.DownloadProgressCommand":{"jobId":"d43a17c9-3b03- >>>>> > > >> 4ff9- >>>>> > > >> >> >> 8906-e1d155981e86","request":"GET_STATUS","hvm":true," >>>>> > > >> >> >> >>>>> > > >> >>>>> description":"centext","maxDownloadSizeInBytes":53687091200,"id":209," >>>>> > > >> >> >> resourceType":"TEMPLATE","installPath":"template/tmpl/2/ >>>>> > > >> >> >> >>>>> 209/209-2-b624436c-5f37-30d4-8eaf-81582eb0d39d","_store":{" >>>>> > > >> >> >> >>>>> com.cloud.agent.api.to.S3TO":{"id":14,"uuid":"e4afd7bb-39ea >>>>> > > >> >> >> - 4128-ab93-f8a09b1d5e03","bucketName":"test-cloudstack", >>>>> > > >> >> >> "httpsFlag":false,"created":"Aug 26, 2014 8:16:24 >>>>> > > >> >> PM","enableRRS":false," >>>>> > > >> >> >> maxSingleUploadSizeInBytes":5368709120}},"url":"http:// >>>>> > > >> >> >> >>>>> download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz >>>>> > > >> >> >> 2 >>>>> > > >> >> >> >>>>> > > >> >> >>>>> > > >> >>>>> > > >>>>> > >>>>> ","format":"VHD","accountId":2,"name":"209-2-b624436c-5f37-30d4-8eaf-8 >>>>> > 1582eb0d39d","wait":0}}] >>>>> > > >> >> >> } >>>>> > > >> >> >> 2014-08-26 20:41:07,556 DEBUG [c.c.a.t.Request] >>>>> > > >> >> >> (AgentManager-Handler-10:null) Seq 10-5684105679694996125: >>>>> > > >> >> Processing: { >>>>> > > >> >> >> Ans: , MgmtId: 52242179434, via: 10, Ver: v1, Flags: 10, >>>>> > > >> >> >> [{"com.cloud.agent.api.storage.DownloadAnswer":{" >>>>> > > >> >> >> jobId":"d43a17c9-3b03-4ff9-8906-e1d155981e86"," >>>>> > > >> >> >> downloadPct":0,"errorString":"No route to >>>>> > host","downloadStatus":" >>>>> > > >> >> >> DOWNLOAD_ERROR","installPath":"template/tmpl/2/209/209-2- >>>>> > > >> >> >> b624436c-5f37-30d4-8eaf-81582eb0d39d","templateSize": >>>>> > > >> >> >> 0,"templatePhySicalSize":0,"result":true,"details":"No >>>>> > > >> >> >> route to host","wait":0}}] } >>>>> > > >> >> >> >>>>> > > >> >> >> but i don't see any logging happening in secondary storage >>>>> > > >> >> >> vm's >>>>> > > >> >> cloud.log >>>>> > > >> >> >> >>>>> > > >> >> >> not sure this error is happening due to S3! >>>>> > > >> >> >> >>>>> > > >> >> >> >>>>> > > >> >> >> thanks! >>>>> > > >> >> >> >>>>> > > >> >> > >>>>> > > >> >> > >>>>> > > >> >> > -- >>>>> > > >> >> > Francois Gaudreault >>>>> > > >> >> > Gestionnaire de Produit | Product Manager - Cloud Platform >>>>> & >>>>> > > Services >>>>> > > >> >> > t:514-629-6775 >>>>> > > >> >> > >>>>> > > >> >> > CloudOps Votre partenaire infonuagique | Cloud Solutions >>>>> > > >> >> > Experts >>>>> > > >> >> > 420 rue Guy | Montreal | Quebec | H3J 1S6 >>>>> > > >> >> > w: cloudops.com | tw: @CloudOps_ >>>>> > > >> >> > >>>>> > > >> >> > >>>>> > > >> >> >>>>> > > >> > >>>>> > > >> > >>>>> > > >> > >>>>> > > >> > -- >>>>> > > >> > regards, >>>>> > > >> > >>>>> > > >> > punith s >>>>> > > >> > cloudbyte.com >>>>> > > >> > >>>>> > > >> >>>>> > > >> >>>>> > > >> >>>>> > > >> -- >>>>> > > >> *Mike Tutkowski* >>>>> > > >> *Senior CloudStack Developer, SolidFire Inc.* >>>>> > > >> e: mike.tutkow...@solidfire.com >>>>> > > >> o: 303.746.7302 >>>>> > > >> Advancing the way the world uses the cloud >>>>> > > >> <http://solidfire.com/solution/overview/?video=play>*™* >>>>> > > >> >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > -- >>>>> > > > regards, >>>>> > > > >>>>> > > > punith s >>>>> > > > cloudbyte.com >>>>> > > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > -- >>>>> > > *Mike Tutkowski* >>>>> > > *Senior CloudStack Developer, SolidFire Inc.* >>>>> > > e: mike.tutkow...@solidfire.com >>>>> > > o: 303.746.7302 >>>>> > > Advancing the way the world uses the cloud >>>>> > > <http://solidfire.com/solution/overview/?video=play>*™* >>>>> > > >>>>> > >>>>> >>>>> >>>>> >>>>> -- >>>>> *Mike Tutkowski* >>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>> e: mike.tutkow...@solidfire.com >>>>> o: 303.746.7302 >>>>> Advancing the way the world uses the cloud >>>>> <http://solidfire.com/solution/overview/?video=play>*™* >>>>> >>>> >>>> >>>> >>>> -- >>>> *Mike Tutkowski* >>>> *Senior CloudStack Developer, SolidFire Inc.* >>>> e: mike.tutkow...@solidfire.com >>>> o: 303.746.7302 >>>> Advancing the way the world uses the cloud >>>> <http://solidfire.com/solution/overview/?video=play>*™* >>>> >>> >>> >>> >>> -- >>> *Mike Tutkowski* >>> *Senior CloudStack Developer, SolidFire Inc.* >>> e: mike.tutkow...@solidfire.com >>> o: 303.746.7302 >>> Advancing the way the world uses the cloud >>> <http://solidfire.com/solution/overview/?video=play>*™* >>> >> >> >> >> -- >> regards, >> >> punith s >> cloudbyte.com >> > > > > -- > *Mike Tutkowski* > *Senior CloudStack Developer, SolidFire Inc.* > e: mike.tutkow...@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the cloud > <http://solidfire.com/solution/overview/?video=play>*™* >
