By the way, for anyone new to this issue, this is what we're referring to here:
https://issues.apache.org/jira/browse/CLOUDSTACK-7406 On Mon, Sep 8, 2014 at 11:41 PM, Mike Tutkowski < mike.tutkow...@solidfire.com> wrote: > Great :) Then a question might be, "Is it too late in the game to > interrogate the template to discover its virtual size if we're just about > to copy the template to primary storage?" > > If it's not, this might be the place to run the logic to figure out the > virtual size. > > Really, there are three big possibilities: > > 1) Just ask the end user to provide the virtual size (not commenting here > on what happens for already-uploaded templates) > > or > > 2) Figure out the virtual size when the template is copied from object > storage to secondary storage and update the DB with this info (not sure > what happens if the template has already been copied to (secondary-storage) > NFS because it was used before) > > or > > 3) Figure out the virtual size when the template is about to be copied > from secondary storage to primary storage > > On Mon, Sep 8, 2014 at 11:35 PM, Sanjeev Neelarapu < > sanjeev.neelar...@citrix.com> wrote: > >> Mike, >> >> You are right. Template gets copied to (secondary-storage) NFS before >> being copied to primary storage >> >> -Sanjeev >> >> -----Original Message----- >> From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] >> Sent: Tuesday, September 09, 2014 10:55 AM >> To: dev@cloudstack.apache.org >> Cc: Punith S; Francois Gaudreault >> Subject: Re: S3/Swift Problem around Virtual Size >> >> Hi Will, >> >> Thanks for the input! >> >> I like the idea of storing the virtual size as metadata in S3 or Swift >> although this could require that the end user provide this value when >> uploading the template. >> >> However, if we have the ability to determine the virtual size of the >> template after it gets downloaded to (secondary-storage) NFS and we're able >> to update the database with this info, then it would seem we would never >> need to ask the user for this value. >> >> Either way, the tricky part might be if the template in object storage >> has already been downloaded to (secondary-storage) NFS (say it was used >> before). In this case, we won't need to download it to (secondary-storage) >> NFS again (at least not in the same zone), so we won't have an easy >> opportunity to figure out the virtual size upon download from object >> storage. >> >> I wonder if it's too late in this process if we figured out the virtual >> size before the copied template (now on (secondary-storage) NFS) gets >> copied to primary storage. If we could do it at this point, then we >> wouldn't have to worry about fixing the "legacy" situation because it would >> just work out naturally. We would look in the DB to see if the virtual size >> for this template is known and, if not, we could figure out the virtual >> size before downloading from (secondary-storage) NFS to primary storage >> each time. (Although I'm thinking this would come too late in the process >> because we may have already asked the primary-storage plug-in to create the >> necessary volume.) >> >> By the way, I'm assuming that a template gets copied to >> (secondary-storage) NFS before being copied to primary storage. I'm not >> super familiar with how this process works. >> >> Talk to you later, >> Mike >> >> On Mon, Sep 8, 2014 at 10:59 PM, Will Stevens <wstev...@cloudops.com> >> wrote: >> >> > My two cents on the topic. >> > >> > Ideally we would save the size in the object store metadata and >> > retrieve it from the metadata if it is set. If it is not set in the >> > object store metadata, then when it is fetched, we have to put it on >> > NFS and determine the size (then ideally save the metadata back to the >> > object store) and remove the NFS copy. >> > >> > This way the NFS copy approach is only ever done once and then the >> > data is populated (for backwards compatibility). For all templates >> > created after the patch, the metadata would be stored and retrieved >> > without the need for the NFS copy. >> > >> > Is this feasible? >> > >> > Will >> > >> > >> > *Will STEVENS* >> > Lead Developer >> > >> > *CloudOps* *| *Cloud Solutions Experts >> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw >> > @CloudOps_ >> > >> > On Tue, Sep 9, 2014 at 12:48 AM, Mike Tutkowski < >> > mike.tutkow...@solidfire.com> wrote: >> > >> > > Hi Punith, >> > > >> > > Thanks for putting in time on this! >> > > >> > > So, option 1 occurs after we download the template to S3 or Swift. >> > > We >> > then >> > > have to copy it to NFS so that we can determine the virtual size? >> > > Relatively slow, but it would work. >> > > >> > > If we went this route, would we then delete that template from the >> > > NFS share since its immediate purpose was so we could figure out the >> > > virtual size or would we leave it on that share? >> > > >> > > Option 2 is to allow the user who uploads such a template to specify >> > > the size needed for the root disk (i.e. at least the virtual size of >> > > the template...perhaps larger)? >> > > >> > > Does that sound like I understood the options? >> > > >> > > Thanks! >> > > Mike >> > > >> > > On Mon, Sep 8, 2014 at 10:34 PM, Punith S <punit...@cloudbyte.com> >> > wrote: >> > > >> > > > hi mike, >> > > > >> > > > i have figured out the issue, in NFS secondary storage the virtual >> > > > size >> > > is >> > > > been calculated by the VHD Processor by accessing the vhd template. >> > > > >> > > > but in case of S3, cloudstack is not able to access the template >> > > > but it only gets to know the physical size of the template! >> > > > >> > > > in order to solve this, we need to download the template from S3 >> > > > to another secondary nfs storage and access the template to >> > > > calculate the virtual size. >> > > > >> > > > if the above solution seems to be complicated, we shall have a >> > > > quick >> > fix >> > > > where virtual size i.e the size of the SAN volume shall be >> > > > created(like small-20g, medium-40g) for people using the templates >> from S3. >> > > > >> > > > any feedback's ? >> > > > >> > > > >> > > > >> > > > On Tue, Sep 9, 2014 at 6:34 AM, Mike Tutkowski < >> > > > mike.tutkow...@solidfire.com> wrote: >> > > > >> > > >> Hi Punith, >> > > >> >> > > >> Have you been able to make any progress with regards to this >> > > >> Swift/S3 issue? >> > > >> >> > > >> Thanks! >> > > >> Mike >> > > >> >> > > >> On Wed, Aug 27, 2014 at 7:43 AM, Punith S >> > > >> <punit...@cloudbyte.com> >> > > wrote: >> > > >> >> > > >> > hi >> > > >> > >> > > >> > think i had a timeout problem! >> > > >> > on the second try the template has been downloaded to the S3 >> > > >> > bucket >> > > and >> > > >> > the management server shows the status as download complete >> > > >> > with >> > > >> template >> > > >> > size as 1.6G instead of its virtual size 20G. >> > > >> > >> > > >> > and i see the template's status as Download Complete but it >> > > >> > seems it >> > > is >> > > >> > not getting installed ! refer the attachment >> > > >> > >> > > >> > can anyone explain the "installing template" after the download >> > > >> completes ? >> > > >> > >> > > >> > >> > > >> > On Wed, Aug 27, 2014 at 9:18 AM, Marcus <shadow...@gmail.com> >> > wrote: >> > > >> > >> > > >> >> Per Edisons comments about not knowing the image size, can't >> > > >> >> we >> > just >> > > >> set >> > > >> >> some headers and store metadata with the template in S3 to >> > > >> >> save the virtual size when the template is registered? I'm >> > > >> >> assuming here that the >> > SSVM >> > > >> does >> > > >> >> the work of pulling the template in and uploading to S3. Or it >> > could >> > > be >> > > >> >> stored in the template table? >> > > >> >> On Aug 26, 2014 9:11 PM, "Francois Gaudreault" < >> > > >> fgaudrea...@cloudops.com> >> > > >> >> wrote: >> > > >> >> >> > > >> >> > Looks like your SSVM cannot reach Internet properly? >> > > >> >> > >> > > >> >> > FG >> > > >> >> > >> > > >> >> > On 2014-08-26, 11:14 AM, Punith S wrote: >> > > >> >> > >> > > >> >> >> hi francois, >> > > >> >> >> >> > > >> >> >> since i'm not having a swift setup, i'm using the s3 bucket. >> > > >> >> >> >> > > >> >> >> and as you recommended i got the SSVM up with seeded nfs >> > storage, >> > > >> >> >> >> > > >> >> >> post that i removed the nfs secondary storage and added the >> > > >> >> >> S3 >> > > with >> > > >> >> >> staging nfs store as the new sec storage, since you cannot >> > > >> >> >> have >> > > any >> > > >> nfs >> > > >> >> >> secondary storage while using the S3. >> > > >> >> >> >> > > >> >> >> on registering the a new template, i'm getting template >> > > >> >> >> status >> > > >> >> as*Unable >> > > >> >> >> to execute HTTP request: No route to host* in >> > > >> >> >> managementserver.log >> > > >> >> >> >> > > >> >> >> 2014-08-26 20:41:07,502 DEBUG [o.a.c.s.RemoteHostEndPoint] >> > > >> >> >> (Timer-24:ctx-b68380cd) Sending command >> > > >> org.apache.cloudstack.storage. >> > > >> >> >> command.DownloadProgressCommand to host: 10 >> > > >> >> >> 2014-08-26 20:41:07,507 DEBUG [c.c.a.t.Request] >> > > >> (Timer-24:ctx-b68380cd) >> > > >> >> >> Seq 10-5684105679694996125: Sending { Cmd , MgmtId: >> > 52242179434, >> > > >> via: >> > > >> >> >> 10(s-142-VM), Ver: v1, Flags: 100011, >> [{"org.apache.cloudstack. >> > > >> >> >> >> > > >> storage.command.DownloadProgressCommand":{"jobId":"d43a17c9-3b03- >> > > >> 4ff9- >> > > >> >> >> 8906-e1d155981e86","request":"GET_STATUS","hvm":true," >> > > >> >> >> >> > > >> >> description":"centext","maxDownloadSizeInBytes":53687091200,"id":209," >> > > >> >> >> resourceType":"TEMPLATE","installPath":"template/tmpl/2/ >> > > >> >> >> 209/209-2-b624436c-5f37-30d4-8eaf-81582eb0d39d","_store":{" >> > > >> >> >> com.cloud.agent.api.to.S3TO":{"id":14,"uuid":"e4afd7bb-39ea >> > > >> >> >> - 4128-ab93-f8a09b1d5e03","bucketName":"test-cloudstack", >> > > >> >> >> "httpsFlag":false,"created":"Aug 26, 2014 8:16:24 >> > > >> >> PM","enableRRS":false," >> > > >> >> >> maxSingleUploadSizeInBytes":5368709120}},"url":"http:// >> > > >> >> >> download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz >> > > >> >> >> 2 >> > > >> >> >> >> > > >> >> >> > > >> >> > > >> > ","format":"VHD","accountId":2,"name":"209-2-b624436c-5f37-30d4-8eaf-8 >> > 1582eb0d39d","wait":0}}] >> > > >> >> >> } >> > > >> >> >> 2014-08-26 20:41:07,556 DEBUG [c.c.a.t.Request] >> > > >> >> >> (AgentManager-Handler-10:null) Seq 10-5684105679694996125: >> > > >> >> Processing: { >> > > >> >> >> Ans: , MgmtId: 52242179434, via: 10, Ver: v1, Flags: 10, >> > > >> >> >> [{"com.cloud.agent.api.storage.DownloadAnswer":{" >> > > >> >> >> jobId":"d43a17c9-3b03-4ff9-8906-e1d155981e86"," >> > > >> >> >> downloadPct":0,"errorString":"No route to >> > host","downloadStatus":" >> > > >> >> >> DOWNLOAD_ERROR","installPath":"template/tmpl/2/209/209-2- >> > > >> >> >> b624436c-5f37-30d4-8eaf-81582eb0d39d","templateSize": >> > > >> >> >> 0,"templatePhySicalSize":0,"result":true,"details":"No >> > > >> >> >> route to host","wait":0}}] } >> > > >> >> >> >> > > >> >> >> but i don't see any logging happening in secondary storage >> > > >> >> >> vm's >> > > >> >> cloud.log >> > > >> >> >> >> > > >> >> >> not sure this error is happening due to S3! >> > > >> >> >> >> > > >> >> >> >> > > >> >> >> thanks! >> > > >> >> >> >> > > >> >> > >> > > >> >> > >> > > >> >> > -- >> > > >> >> > Francois Gaudreault >> > > >> >> > Gestionnaire de Produit | Product Manager - Cloud Platform & >> > > Services >> > > >> >> > t:514-629-6775 >> > > >> >> > >> > > >> >> > CloudOps Votre partenaire infonuagique | Cloud Solutions >> > > >> >> > Experts >> > > >> >> > 420 rue Guy | Montreal | Quebec | H3J 1S6 >> > > >> >> > w: cloudops.com | tw: @CloudOps_ >> > > >> >> > >> > > >> >> > >> > > >> >> >> > > >> > >> > > >> > >> > > >> > >> > > >> > -- >> > > >> > regards, >> > > >> > >> > > >> > punith s >> > > >> > cloudbyte.com >> > > >> > >> > > >> >> > > >> >> > > >> >> > > >> -- >> > > >> *Mike Tutkowski* >> > > >> *Senior CloudStack Developer, SolidFire Inc.* >> > > >> e: mike.tutkow...@solidfire.com >> > > >> o: 303.746.7302 >> > > >> Advancing the way the world uses the cloud >> > > >> <http://solidfire.com/solution/overview/?video=play>*™* >> > > >> >> > > > >> > > > >> > > > >> > > > -- >> > > > regards, >> > > > >> > > > punith s >> > > > cloudbyte.com >> > > > >> > > >> > > >> > > >> > > -- >> > > *Mike Tutkowski* >> > > *Senior CloudStack Developer, SolidFire Inc.* >> > > e: mike.tutkow...@solidfire.com >> > > o: 303.746.7302 >> > > Advancing the way the world uses the cloud >> > > <http://solidfire.com/solution/overview/?video=play>*™* >> > > >> > >> >> >> >> -- >> *Mike Tutkowski* >> *Senior CloudStack Developer, SolidFire Inc.* >> e: mike.tutkow...@solidfire.com >> o: 303.746.7302 >> Advancing the way the world uses the cloud >> <http://solidfire.com/solution/overview/?video=play>*™* >> > > > > -- > *Mike Tutkowski* > *Senior CloudStack Developer, SolidFire Inc.* > e: mike.tutkow...@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the cloud > <http://solidfire.com/solution/overview/?video=play>*™* > -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud <http://solidfire.com/solution/overview/?video=play>*™*
