sure mike, since i don't have a S3 account, i'm getting one today.
francois, can you brief me how you seeded your templates into S3. thanks! On Mon, Aug 25, 2014 at 11:16 PM, Mike Tutkowski < mike.tutkow...@solidfire.com> wrote: > Yes, I expect we'll see the same issue with S3, as well. > > Punith - Is this something you might have time to investigate? Perhaps > Edison can point us in the right direction here. > > > On Mon, Aug 25, 2014 at 5:17 AM, Francois Gaudreault < > fgaudrea...@cloudops.com> wrote: > > > Punith, > > > > I highly anticipate the same issue with S3... it shares a lot of code > with > > swift. > > > > My focus would be swift, but we should fix for both :) > > > > FG > > On Aug 25, 2014 6:33 AM, "Punith S" <punit...@cloudbyte.com> wrote: > > > > > thanks for opening this thread mike, > > > > > > since i only use nfs as my secondary storage provider, i didn't see > this > > > issue till date. > > > > > > is this issue occurring even using a S3 secondary storage with staging > > nfs > > > store ? > > > > > > if so like edison pointed we need to fetch the virtual size from the > nfs > > > store instead of S3 in the deploymentmanager. > > > > > > thanks > > > > > > > > > On Sat, Aug 23, 2014 at 3:45 AM, Mike Tutkowski < > > > mike.tutkow...@solidfire.com> wrote: > > > > > > > Hey Edison, > > > > > > > > Do you know how difficult/easy of a fix this is, who might be > available > > > to > > > > put this fix in, and for what release (hopefully 4.4.1) this fix > could > > > find > > > > its way in? > > > > > > > > Thanks! > > > > Mike > > > > > > > > > > > > On Fri, Aug 22, 2014 at 3:37 PM, Francois Gaudreault < > > > > fgaudrea...@cloudops.com> wrote: > > > > > > > > > Min, > > > > > > > > > > Ok, but this is not the behavior I see. Even without requesting a > VM > > > > > create, the template is pushed to the staging NFS at least once. Is > > it > > > > > downloaded there or pushed after download, that I am not sure. I > was > > > > > assuming the swift upload bash script was executed after the > template > > > is > > > > on > > > > > the staging. > > > > > > > > > > Anyway... the focus is on the virt size, and you all know the code > > > better > > > > > than I do :) > > > > > > > > > > FG > > > > > On Aug 22, 2014 5:28 PM, "Min Chen" <min.c...@citrix.com> wrote: > > > > > > > > > >> No. For S3/Swift, register template will directly upload templates > > to > > > S3 > > > > >> without going through staging NFS. It will only be copied to > staging > > > NFS > > > > >> when we first use that template to provision a VM. > > > > >> > > > > >> Thanks > > > > >> -min > > > > >> > > > > >> On 8/22/14 2:25 PM, "Francois Gaudreault" < > fgaudrea...@cloudops.com > > > > > > > >> wrote: > > > > >> > > > > >> >Edison, > > > > >> > > > > > >> >Isnt the templates downloaded to the Staging NFS first? > > > > >> > > > > > >> >FG > > > > >> >On Aug 22, 2014 5:20 PM, "Edison Su" <edison...@citrix.com> > wrote: > > > > >> > > > > > >> >> I know the reason why the size of template doesn¹t have correct > > > > virtual > > > > >> >> size if it¹s registered in S3/Swift: > > > > >> >> In case of s3/swift, the template is directly stored into > > s3/swift > > > > >> >>through > > > > >> >> swift/s3 api, there is no place for cloudstack to look into > > > template, > > > > >> to > > > > >> >> find out the virtual size during template registration. > > > > >> >> While, if secondary storage is NFS, the template is first > stored > > on > > > > >> >> NFS(which has file system), cloudstack can unzip the > template(if > > > > it¹s a > > > > >> >> zipped file), and read virtual size from the file, then report > > back > > > > to > > > > >> >>mgt > > > > >> >> server. > > > > >> >> In order to fix it, we can add some code as: all the templates > > > > >> >>registered > > > > >> >> on Swift/S3, need to be downloaded to a NFS intermediate > storage > > > > before > > > > >> >>it > > > > >> >> can be consumed by primary storage. After the download > finished, > > > then > > > > >> we > > > > >> >> check virtual size of template, and report back to mgt > > > server/update > > > > DB > > > > >> >>etc. > > > > >> >> > > > > >> >> From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] > > > > >> >> Sent: Friday, August 22, 2014 1:38 PM > > > > >> >> To: dev@cloudstack.apache.org > > > > >> >> Cc: Edison Su > > > > >> >> Subject: S3/Swift Problem around Virtual Size > > > > >> >> > > > > >> >> Hi, > > > > >> >> > > > > >> >> This was brought up in a different e-mail thread, but I wanted > to > > > > make > > > > >> >>it > > > > >> >> more clear that it's related to CloudStack's download code > around > > > > >> >>S3/Swift, > > > > >> >> so I'm opening up a new thread. > > > > >> >> > > > > >> >> Francois (from CloudOps) noticed today that when he downloaded > a > > > > >> >>template > > > > >> >> (VHD format) to Swift (but it looks like the same applies for > S3) > > > > that > > > > >> >>the > > > > >> >> physical and virtual sizes are set to be the same. > > > > >> >> > > > > >> >> This appears to have the following consequence: > > > > >> >> > > > > >> >> You can download a template with a physical size of, say, 3 GB > > and > > > a > > > > >> >>root > > > > >> >> disk that's supposed to be, say, 20 GB. Instead of the virtual > > size > > > > >> >>showing > > > > >> >> as 20 GB, it shows as 3 GB. > > > > >> >> > > > > >> >> This is not an issue with NFS. In that situation, the two sizes > > are > > > > >> >> correctly accounted for. > > > > >> >> > > > > >> >> What later can happen is the template is downloaded from Swift > > and > > > > >> takes > > > > >> >> up an unexpected amount of space on the XenServer storage > > > repository > > > > >> >>(SR). > > > > >> >> > > > > >> >> If there is enough space on the SR, this isn't too big of a > deal. > > > > >> >>However, > > > > >> >> for so-called managed storage plug-ins (examples are SolidFire > > and > > > > >> >> CloudByte), this will lead to them dynamically creating a SAN > > > volume > > > > of > > > > >> >>the > > > > >> >> wrong size. > > > > >> >> > > > > >> >> Francois opened up the following ticket: > > > > >> >> > > > > >> >> https://issues.apache.org/jira/browse/CLOUDSTACK-7406 > > > > >> >> > > > > >> >> Thanks! > > > > >> >> > > > > >> >> -- > > > > >> >> Mike Tutkowski > > > > >> >> Senior CloudStack Developer, SolidFire Inc. > > > > >> >> e: mike.tutkow...@solidfire.com<mailto: > > > mike.tutkow...@solidfire.com> > > > > >> >> o: 303.746.7302 > > > > >> >> Advancing the way the world uses the cloud< > > > > >> >> http://solidfire.com/solution/overview/?video=play> > > > > >> >> > > > > >> > > > > >> > > > > > > > > > > > > -- > > > > *Mike Tutkowski* > > > > *Senior CloudStack Developer, SolidFire Inc.* > > > > e: mike.tutkow...@solidfire.com > > > > o: 303.746.7302 > > > > Advancing the way the world uses the cloud > > > > <http://solidfire.com/solution/overview/?video=play>*™* > > > > > > > > > > > > > > > > -- > > > regards, > > > > > > punith s > > > cloudbyte.com > > > > > > > > > -- > *Mike Tutkowski* > *Senior CloudStack Developer, SolidFire Inc.* > e: mike.tutkow...@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the cloud > <http://solidfire.com/solution/overview/?video=play>*™* > -- regards, punith s cloudbyte.com
