hi, i'm trying to deploy a zone with S3 alongside an empty staging nfs store as the secondary storage,
does anyone have idea about how to seed the s3 secondary storage with ssvm templates to bring up the secondary storage VM ? like in nfs we seed nfs sec store by running this script /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt \ -m /mnt/secondary \ -u http://cloudstack.apt-get.eu/systemvm/4.4/systemvm64template-4.4.0-6-xen.vhd.bz2 \ -h xenserver \ -s <optional-management-server-secret-key> \ -F should i seed the staging nfs with the above script ? thanks On Tue, Aug 26, 2014 at 4:00 PM, Francois Gaudreault < fgaudrea...@cloudops.com> wrote: > I mean we are populating the template just like we would do with normal > NFS using the UI. > > ACS takes care of pushing to swift. > > FG > On Aug 26, 2014 6:02 AM, "Francois Gaudreault" <fgaudrea...@cloudops.com> > wrote: > >> Punith, >> >> We are using Swift. We have a tmpauth proxy. >> >> FG >> On Aug 26, 2014 2:48 AM, "Punith S" <punit...@cloudbyte.com> wrote: >> >>> sure mike, >>> >>> since i don't have a S3 account, i'm getting one today. >>> >>> francois, can you brief me how you seeded your templates into S3. >>> >>> thanks! >>> >>> >>> On Mon, Aug 25, 2014 at 11:16 PM, Mike Tutkowski < >>> mike.tutkow...@solidfire.com> wrote: >>> >>>> Yes, I expect we'll see the same issue with S3, as well. >>>> >>>> Punith - Is this something you might have time to investigate? Perhaps >>>> Edison can point us in the right direction here. >>>> >>>> >>>> On Mon, Aug 25, 2014 at 5:17 AM, Francois Gaudreault < >>>> fgaudrea...@cloudops.com> wrote: >>>> >>>> > Punith, >>>> > >>>> > I highly anticipate the same issue with S3... it shares a lot of code >>>> with >>>> > swift. >>>> > >>>> > My focus would be swift, but we should fix for both :) >>>> > >>>> > FG >>>> > On Aug 25, 2014 6:33 AM, "Punith S" <punit...@cloudbyte.com> wrote: >>>> > >>>> > > thanks for opening this thread mike, >>>> > > >>>> > > since i only use nfs as my secondary storage provider, i didn't see >>>> this >>>> > > issue till date. >>>> > > >>>> > > is this issue occurring even using a S3 secondary storage with >>>> staging >>>> > nfs >>>> > > store ? >>>> > > >>>> > > if so like edison pointed we need to fetch the virtual size from >>>> the nfs >>>> > > store instead of S3 in the deploymentmanager. >>>> > > >>>> > > thanks >>>> > > >>>> > > >>>> > > On Sat, Aug 23, 2014 at 3:45 AM, Mike Tutkowski < >>>> > > mike.tutkow...@solidfire.com> wrote: >>>> > > >>>> > > > Hey Edison, >>>> > > > >>>> > > > Do you know how difficult/easy of a fix this is, who might be >>>> available >>>> > > to >>>> > > > put this fix in, and for what release (hopefully 4.4.1) this fix >>>> could >>>> > > find >>>> > > > its way in? >>>> > > > >>>> > > > Thanks! >>>> > > > Mike >>>> > > > >>>> > > > >>>> > > > On Fri, Aug 22, 2014 at 3:37 PM, Francois Gaudreault < >>>> > > > fgaudrea...@cloudops.com> wrote: >>>> > > > >>>> > > > > Min, >>>> > > > > >>>> > > > > Ok, but this is not the behavior I see. Even without requesting >>>> a VM >>>> > > > > create, the template is pushed to the staging NFS at least >>>> once. Is >>>> > it >>>> > > > > downloaded there or pushed after download, that I am not sure. >>>> I was >>>> > > > > assuming the swift upload bash script was executed after the >>>> template >>>> > > is >>>> > > > on >>>> > > > > the staging. >>>> > > > > >>>> > > > > Anyway... the focus is on the virt size, and you all know the >>>> code >>>> > > better >>>> > > > > than I do :) >>>> > > > > >>>> > > > > FG >>>> > > > > On Aug 22, 2014 5:28 PM, "Min Chen" <min.c...@citrix.com> >>>> wrote: >>>> > > > > >>>> > > > >> No. For S3/Swift, register template will directly upload >>>> templates >>>> > to >>>> > > S3 >>>> > > > >> without going through staging NFS. It will only be copied to >>>> staging >>>> > > NFS >>>> > > > >> when we first use that template to provision a VM. >>>> > > > >> >>>> > > > >> Thanks >>>> > > > >> -min >>>> > > > >> >>>> > > > >> On 8/22/14 2:25 PM, "Francois Gaudreault" < >>>> fgaudrea...@cloudops.com >>>> > > >>>> > > > >> wrote: >>>> > > > >> >>>> > > > >> >Edison, >>>> > > > >> > >>>> > > > >> >Isnt the templates downloaded to the Staging NFS first? >>>> > > > >> > >>>> > > > >> >FG >>>> > > > >> >On Aug 22, 2014 5:20 PM, "Edison Su" <edison...@citrix.com> >>>> wrote: >>>> > > > >> > >>>> > > > >> >> I know the reason why the size of template doesn¹t have >>>> correct >>>> > > > virtual >>>> > > > >> >> size if it¹s registered in S3/Swift: >>>> > > > >> >> In case of s3/swift, the template is directly stored into >>>> > s3/swift >>>> > > > >> >>through >>>> > > > >> >> swift/s3 api, there is no place for cloudstack to look into >>>> > > template, >>>> > > > >> to >>>> > > > >> >> find out the virtual size during template registration. >>>> > > > >> >> While, if secondary storage is NFS, the template is first >>>> stored >>>> > on >>>> > > > >> >> NFS(which has file system), cloudstack can unzip the >>>> template(if >>>> > > > it¹s a >>>> > > > >> >> zipped file), and read virtual size from the file, then >>>> report >>>> > back >>>> > > > to >>>> > > > >> >>mgt >>>> > > > >> >> server. >>>> > > > >> >> In order to fix it, we can add some code as: all the >>>> templates >>>> > > > >> >>registered >>>> > > > >> >> on Swift/S3, need to be downloaded to a NFS intermediate >>>> storage >>>> > > > before >>>> > > > >> >>it >>>> > > > >> >> can be consumed by primary storage. After the download >>>> finished, >>>> > > then >>>> > > > >> we >>>> > > > >> >> check virtual size of template, and report back to mgt >>>> > > server/update >>>> > > > DB >>>> > > > >> >>etc. >>>> > > > >> >> >>>> > > > >> >> From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] >>>> > > > >> >> Sent: Friday, August 22, 2014 1:38 PM >>>> > > > >> >> To: dev@cloudstack.apache.org >>>> > > > >> >> Cc: Edison Su >>>> > > > >> >> Subject: S3/Swift Problem around Virtual Size >>>> > > > >> >> >>>> > > > >> >> Hi, >>>> > > > >> >> >>>> > > > >> >> This was brought up in a different e-mail thread, but I >>>> wanted to >>>> > > > make >>>> > > > >> >>it >>>> > > > >> >> more clear that it's related to CloudStack's download code >>>> around >>>> > > > >> >>S3/Swift, >>>> > > > >> >> so I'm opening up a new thread. >>>> > > > >> >> >>>> > > > >> >> Francois (from CloudOps) noticed today that when he >>>> downloaded a >>>> > > > >> >>template >>>> > > > >> >> (VHD format) to Swift (but it looks like the same applies >>>> for S3) >>>> > > > that >>>> > > > >> >>the >>>> > > > >> >> physical and virtual sizes are set to be the same. >>>> > > > >> >> >>>> > > > >> >> This appears to have the following consequence: >>>> > > > >> >> >>>> > > > >> >> You can download a template with a physical size of, say, 3 >>>> GB >>>> > and >>>> > > a >>>> > > > >> >>root >>>> > > > >> >> disk that's supposed to be, say, 20 GB. Instead of the >>>> virtual >>>> > size >>>> > > > >> >>showing >>>> > > > >> >> as 20 GB, it shows as 3 GB. >>>> > > > >> >> >>>> > > > >> >> This is not an issue with NFS. In that situation, the two >>>> sizes >>>> > are >>>> > > > >> >> correctly accounted for. >>>> > > > >> >> >>>> > > > >> >> What later can happen is the template is downloaded from >>>> Swift >>>> > and >>>> > > > >> takes >>>> > > > >> >> up an unexpected amount of space on the XenServer storage >>>> > > repository >>>> > > > >> >>(SR). >>>> > > > >> >> >>>> > > > >> >> If there is enough space on the SR, this isn't too big of a >>>> deal. >>>> > > > >> >>However, >>>> > > > >> >> for so-called managed storage plug-ins (examples are >>>> SolidFire >>>> > and >>>> > > > >> >> CloudByte), this will lead to them dynamically creating a >>>> SAN >>>> > > volume >>>> > > > of >>>> > > > >> >>the >>>> > > > >> >> wrong size. >>>> > > > >> >> >>>> > > > >> >> Francois opened up the following ticket: >>>> > > > >> >> >>>> > > > >> >> https://issues.apache.org/jira/browse/CLOUDSTACK-7406 >>>> > > > >> >> >>>> > > > >> >> Thanks! >>>> > > > >> >> >>>> > > > >> >> -- >>>> > > > >> >> Mike Tutkowski >>>> > > > >> >> Senior CloudStack Developer, SolidFire Inc. >>>> > > > >> >> e: mike.tutkow...@solidfire.com<mailto: >>>> > > mike.tutkow...@solidfire.com> >>>> > > > >> >> o: 303.746.7302 >>>> > > > >> >> Advancing the way the world uses the cloud< >>>> > > > >> >> http://solidfire.com/solution/overview/?video=play> >>>> > > > >> >> >>>> > > > >> >>>> > > > >> >>>> > > > >>>> > > > >>>> > > > -- >>>> > > > *Mike Tutkowski* >>>> > > > *Senior CloudStack Developer, SolidFire Inc.* >>>> > > > e: mike.tutkow...@solidfire.com >>>> > > > o: 303.746.7302 >>>> > > > Advancing the way the world uses the cloud >>>> > > > <http://solidfire.com/solution/overview/?video=play>*™* >>>> > > > >>>> > > >>>> > > >>>> > > >>>> > > -- >>>> > > regards, >>>> > > >>>> > > punith s >>>> > > cloudbyte.com >>>> > > >>>> > >>>> >>>> >>>> >>>> -- >>>> *Mike Tutkowski* >>>> *Senior CloudStack Developer, SolidFire Inc.* >>>> e: mike.tutkow...@solidfire.com >>>> o: 303.746.7302 >>>> Advancing the way the world uses the cloud >>>> <http://solidfire.com/solution/overview/?video=play>*™* >>>> >>> >>> >>> >>> -- >>> regards, >>> >>> punith s >>> cloudbyte.com >>> >> -- regards, punith s cloudbyte.com
