-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 02/02/2015 05:14 PM, Logan Barfield wrote: > Hi Wido, > > I just ran a few tests with qemu-img, and it appears that it can > indeed detect the input image format. I ran exports with no > source format specified, and both -O raw and -O qcow2. I confirmed > the file sizes and formats, then re-imported with no source format, > and -O raw specified. > Ok great, but we'd have to check if that is valid for all versions of Qemu we still support. > This would indicate that qemu-img (at least the version I'm > testing with) would be fine with no source format specified. I > don't know if the QEMU Java functions require a source format or > not though. It appears that they are part of the CloudStack > project, so I would think they'd be easy enough to change if > needed. > I wrote the Qemu Java Utils for CloudStack specifically, so it's no problem to change those. Wido > Thank You, > > Logan Barfield Tranquil Hosting > > > On Sun, Feb 1, 2015 at 1:54 AM, Wido den Hollander <w...@widodh.nl> > wrote: > > > On 01/30/2015 04:43 PM, Logan Barfield wrote: >>>> Hi Wido, >>>> >>>> I didn't see the format information for snapshots stored in >>>> the database, so I'm assuming it references the associated >>>> volume format. >>>> > > Yes, it is. > >>>> I assume the cleanest way to get around that would be to have >>>> a separate format field for snapshots, but that wouldn't >>>> really make sense just for this one use case. Another way >>>> would be to just have the restore (create template from >>>> snapshot, etc.) function always assume QCOW2 for RBD >>>> snapshots, but that's dirty, and would cause problems for >>>> existing snapshots. >>>> > > That would be dirty, since I don't think we can actually know if a > snapshpot was from a RBD volume when it's on SS. > > Maybe qemu-img can detect the input format? That way we don't have > to specify it. > >>>> It seems like the best method would be to have the restore >>>> function look at the actual file to get the format >>>> information instead of pulling it from the database, though >>>> I'm not sure if that's even possible. >>>> > > I think that would be the best option indeed. > >>>> Does anyone else have any thoughts or suggestions? >>>> >>>> Thank You, >>>> >>>> Logan Barfield Tranquil Hosting >>>> >>>> >>>> On Thu, Jan 29, 2015 at 4:21 PM, Wido den Hollander >>>> <w...@widodh.nl> wrote: >>>> >>>> >>>> On 01/29/2015 10:05 PM, Logan Barfield wrote: >>>>>>> Hi Wido, >>>>>>> >>>>>>> The relevant code for this is in: >>>>>>> >>>>>>> I saw that the destination format is set to QCOW2 for >>>>>>> "createTemplateFromVolume", but in "backupSnapshot" >>>>>>> it's set to use the format of the source image >>>>>>> "destFile.setFormat(snapshotDisk.getFormat());" >>>>>>> >>>>>>> I believe just changing the "backupSnapshot" code to >>>>>>> the use QCOW2 >>>>>>> "destFile.setFormat(PhysicalDiskFormat.QCOW2);" will >>>>>>> work fine. I can submit a patch for that, but I wanted >>>>>>> to make sure there wasn't some other reason for the RAW >>>>>>> output first. >>>>>>> >>>> >>>> No, that won't work. Because the management server will then >>>> still think it's in RAW format and store that in the >>>> database. >>>> >>>> It has been a long time since I looked at this. I know I did >>>> some work in this area and made some patches. >>>> >>>> Wido >>>> >>>>>>> >>>>>>> Thank You, >>>>>>> >>>>>>> Logan Barfield Tranquil Hosting >>>>>>> >>>>>>> >>>>>>> On Thu, Jan 29, 2015 at 3:52 PM, Wido den Hollander >>>>>>> <w...@widodh.nl> wrote: >>>>>>> >>>>>>> >>>>>>> On 01/29/2015 05:16 PM, Logan Barfield wrote: >>>>>>>>>> I was doing some testing earlier this week on a >>>>>>>>>> KVM cluster, and noticed that when using S3 for >>>>>>>>>> secondary storage snapshots on RBD primary >>>>>>>>>> storage take a lot longer than they do with NFS >>>>>>>>>> primary storage. >>>>>>>>>> >>>>>>>>>> I think this is due to the NFS snapshots being >>>>>>>>>> output/uploaded in QCOW2 format, while RBD >>>>>>>>>> snapshots are output in RAW format. It appears >>>>>>>>>> that even when using sparse RAW files they are >>>>>>>>>> uploaded to S3 with the 'allocated' size instead >>>>>>>>>> of the sparse size. >>>>>>>>>> >>>>>>> >>>>>>> The problem here lies in the code in CloudStack. Since >>>>>>> the source volume is in RAW format, the snapshot also >>>>>>> becomes QCOW2. >>>>>>> >>>>>>> I remember that I wrote some patches for this, you >>>>>>> might want to look in the Git history. >>>>>>> >>>>>>>>>> My question is: Is there a good reason for RBD >>>>>>>>>> snapshots to be output as RAW instead of QCOW2? >>>>>>>>>> It appears that QCOW2 templates are >>>>>>>>>> automatically converted when deploying them as >>>>>>>>>> RBD, so that shouldn't be a problem. The only >>>>>>>>>> downside I can think of would be having to >>>>>>>>>> convert the RAW RBD data to QCOW2 as an extra >>>>>>>>>> step when creating snapshots, but even that >>>>>>>>>> seems like it would take less time than uploading >>>>>>>>>> the RAW images to S3. >>>>>>>>>> >>>>>>>>>> Thoughts? >>>>>>>>>> >>>>>>> >>>>>>> Like I said, I think I wrote patches for this. Qemu-img >>>>>>> is used to convert the RAW RBD into the snapshot, so >>>>>>> QCOW2 should be no problem as output format. But the >>>>>>> code in CloudStack seems to assume that the snapshot >>>>>>> volume is always the same format as the source. >>>>>>> >>>>>>> Wido >>>>>>> >>>>>>> P.S.: I'm on vacation in Norway at the moment, so I'm >>>>>>> pretty laggy with replying. >>>>>>> >>>>>>>>>> >>>>>>>>>> Thank You, >>>>>>>>>> >>>>>>>>>> Logan Barfield Tranquil Hosting >>>>>>>>>> -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJUz6TLAAoJEAGbWC3bPspCJ0UP/A0Y9oc0qJLhdvKSIJPujBS7 lq8we/p5NPlRGHuRFWidxvdyhf9yhkkLuZfhVyLQPm77owI65xcrVJ0w0lOwuE9d xTLn1u/Ap02cyxbkK1hd4CvhNcwEP+A+z/b71pKAMFkTD4VS/lh0UaRAuZgA1Mr4 /TaZ9IGxly6h0FCDaUzyNRIOVAd6BwdIeg+vDxQI8lt2M47YtQGPyUGEtGrUZ7mw 03u7OfZifg3zd4H468TVz3KvBs0a9cvlVNDzDvwCCNHvdFqvnYJ7dZhG5/mu2NPS cQAzmcgUiZrP/e/RYVjDQzXaLCuoyX4V9T662TtH6qGdYDWr7+SE9dfmV30yxn54 U+0BrBMIGGUyGdwjQ8RMM0i7tlWl1r+tUPcWWI5WSbnkLCJH/QuU7lNYN+ZHyhpp s3IYQnzBgWcNRLiU9ovuXYJWn9MLxhHc+kJjgRhuZFB5J4iV2mRBBOGJ4SkETN5F L8ONjWyRsIf1B+VN3t9cEpr0NQ+zCZzHFhEjTVEkRsY0q6eF7Qqw4SXznvbR0d19 /GS72zHEUDAcyTooJsbe7o1GxRckK2XAOaiK5M18liUUK/CjP/NOT+gKGctnIW8L gl5pozcxw36t/ll7yALR/n6ysa5fOt/rY7VYCV1K98v1Bz+6PBEDT0aCLifGH9bs kRdTmTg583yO/EZKljzl =mhtb -----END PGP SIGNATURE-----
