Awesome. I'm not sure what all versions of QEMU are currently supported, but that's something I can work on if there's a list somewhere.
This is something I'd like to see in the pipeline for 4.6 (though we will probably pull it into a 4.5 build). Do you think this is something we can get done in a reasonable time frame (end of Feb)? Thank You, Logan Barfield Tranquil Hosting On Mon, Feb 2, 2015 at 11:24 AM, Wido den Hollander <w...@widodh.nl> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > > On 02/02/2015 05:14 PM, Logan Barfield wrote: >> Hi Wido, >> >> I just ran a few tests with qemu-img, and it appears that it can >> indeed detect the input image format. I ran exports with no >> source format specified, and both -O raw and -O qcow2. I confirmed >> the file sizes and formats, then re-imported with no source format, >> and -O raw specified. >> > > Ok great, but we'd have to check if that is valid for all versions of > Qemu we still support. > >> This would indicate that qemu-img (at least the version I'm >> testing with) would be fine with no source format specified. I >> don't know if the QEMU Java functions require a source format or >> not though. It appears that they are part of the CloudStack >> project, so I would think they'd be easy enough to change if >> needed. >> > > I wrote the Qemu Java Utils for CloudStack specifically, so it's no > problem to change those. > > Wido > >> Thank You, >> >> Logan Barfield Tranquil Hosting >> >> >> On Sun, Feb 1, 2015 at 1:54 AM, Wido den Hollander <w...@widodh.nl> >> wrote: >> >> >> On 01/30/2015 04:43 PM, Logan Barfield wrote: >>>>> Hi Wido, >>>>> >>>>> I didn't see the format information for snapshots stored in >>>>> the database, so I'm assuming it references the associated >>>>> volume format. >>>>> >> >> Yes, it is. >> >>>>> I assume the cleanest way to get around that would be to have >>>>> a separate format field for snapshots, but that wouldn't >>>>> really make sense just for this one use case. Another way >>>>> would be to just have the restore (create template from >>>>> snapshot, etc.) function always assume QCOW2 for RBD >>>>> snapshots, but that's dirty, and would cause problems for >>>>> existing snapshots. >>>>> >> >> That would be dirty, since I don't think we can actually know if a >> snapshpot was from a RBD volume when it's on SS. >> >> Maybe qemu-img can detect the input format? That way we don't have >> to specify it. >> >>>>> It seems like the best method would be to have the restore >>>>> function look at the actual file to get the format >>>>> information instead of pulling it from the database, though >>>>> I'm not sure if that's even possible. >>>>> >> >> I think that would be the best option indeed. >> >>>>> Does anyone else have any thoughts or suggestions? >>>>> >>>>> Thank You, >>>>> >>>>> Logan Barfield Tranquil Hosting >>>>> >>>>> >>>>> On Thu, Jan 29, 2015 at 4:21 PM, Wido den Hollander >>>>> <w...@widodh.nl> wrote: >>>>> >>>>> >>>>> On 01/29/2015 10:05 PM, Logan Barfield wrote: >>>>>>>> Hi Wido, >>>>>>>> >>>>>>>> The relevant code for this is in: >>>>>>>> >>>>>>>> I saw that the destination format is set to QCOW2 for >>>>>>>> "createTemplateFromVolume", but in "backupSnapshot" >>>>>>>> it's set to use the format of the source image >>>>>>>> "destFile.setFormat(snapshotDisk.getFormat());" >>>>>>>> >>>>>>>> I believe just changing the "backupSnapshot" code to >>>>>>>> the use QCOW2 >>>>>>>> "destFile.setFormat(PhysicalDiskFormat.QCOW2);" will >>>>>>>> work fine. I can submit a patch for that, but I wanted >>>>>>>> to make sure there wasn't some other reason for the RAW >>>>>>>> output first. >>>>>>>> >>>>> >>>>> No, that won't work. Because the management server will then >>>>> still think it's in RAW format and store that in the >>>>> database. >>>>> >>>>> It has been a long time since I looked at this. I know I did >>>>> some work in this area and made some patches. >>>>> >>>>> Wido >>>>> >>>>>>>> >>>>>>>> Thank You, >>>>>>>> >>>>>>>> Logan Barfield Tranquil Hosting >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jan 29, 2015 at 3:52 PM, Wido den Hollander >>>>>>>> <w...@widodh.nl> wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 01/29/2015 05:16 PM, Logan Barfield wrote: >>>>>>>>>>> I was doing some testing earlier this week on a >>>>>>>>>>> KVM cluster, and noticed that when using S3 for >>>>>>>>>>> secondary storage snapshots on RBD primary >>>>>>>>>>> storage take a lot longer than they do with NFS >>>>>>>>>>> primary storage. >>>>>>>>>>> >>>>>>>>>>> I think this is due to the NFS snapshots being >>>>>>>>>>> output/uploaded in QCOW2 format, while RBD >>>>>>>>>>> snapshots are output in RAW format. It appears >>>>>>>>>>> that even when using sparse RAW files they are >>>>>>>>>>> uploaded to S3 with the 'allocated' size instead >>>>>>>>>>> of the sparse size. >>>>>>>>>>> >>>>>>>> >>>>>>>> The problem here lies in the code in CloudStack. Since >>>>>>>> the source volume is in RAW format, the snapshot also >>>>>>>> becomes QCOW2. >>>>>>>> >>>>>>>> I remember that I wrote some patches for this, you >>>>>>>> might want to look in the Git history. >>>>>>>> >>>>>>>>>>> My question is: Is there a good reason for RBD >>>>>>>>>>> snapshots to be output as RAW instead of QCOW2? >>>>>>>>>>> It appears that QCOW2 templates are >>>>>>>>>>> automatically converted when deploying them as >>>>>>>>>>> RBD, so that shouldn't be a problem. The only >>>>>>>>>>> downside I can think of would be having to >>>>>>>>>>> convert the RAW RBD data to QCOW2 as an extra >>>>>>>>>>> step when creating snapshots, but even that >>>>>>>>>>> seems like it would take less time than uploading >>>>>>>>>>> the RAW images to S3. >>>>>>>>>>> >>>>>>>>>>> Thoughts? >>>>>>>>>>> >>>>>>>> >>>>>>>> Like I said, I think I wrote patches for this. Qemu-img >>>>>>>> is used to convert the RAW RBD into the snapshot, so >>>>>>>> QCOW2 should be no problem as output format. But the >>>>>>>> code in CloudStack seems to assume that the snapshot >>>>>>>> volume is always the same format as the source. >>>>>>>> >>>>>>>> Wido >>>>>>>> >>>>>>>> P.S.: I'm on vacation in Norway at the moment, so I'm >>>>>>>> pretty laggy with replying. >>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thank You, >>>>>>>>>>> >>>>>>>>>>> Logan Barfield Tranquil Hosting >>>>>>>>>>> > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1 > > iQIcBAEBAgAGBQJUz6TLAAoJEAGbWC3bPspCJ0UP/A0Y9oc0qJLhdvKSIJPujBS7 > lq8we/p5NPlRGHuRFWidxvdyhf9yhkkLuZfhVyLQPm77owI65xcrVJ0w0lOwuE9d > xTLn1u/Ap02cyxbkK1hd4CvhNcwEP+A+z/b71pKAMFkTD4VS/lh0UaRAuZgA1Mr4 > /TaZ9IGxly6h0FCDaUzyNRIOVAd6BwdIeg+vDxQI8lt2M47YtQGPyUGEtGrUZ7mw > 03u7OfZifg3zd4H468TVz3KvBs0a9cvlVNDzDvwCCNHvdFqvnYJ7dZhG5/mu2NPS > cQAzmcgUiZrP/e/RYVjDQzXaLCuoyX4V9T662TtH6qGdYDWr7+SE9dfmV30yxn54 > U+0BrBMIGGUyGdwjQ8RMM0i7tlWl1r+tUPcWWI5WSbnkLCJH/QuU7lNYN+ZHyhpp > s3IYQnzBgWcNRLiU9ovuXYJWn9MLxhHc+kJjgRhuZFB5J4iV2mRBBOGJ4SkETN5F > L8ONjWyRsIf1B+VN3t9cEpr0NQ+zCZzHFhEjTVEkRsY0q6eF7Qqw4SXznvbR0d19 > /GS72zHEUDAcyTooJsbe7o1GxRckK2XAOaiK5M18liUUK/CjP/NOT+gKGctnIW8L > gl5pozcxw36t/ll7yALR/n6ysa5fOt/rY7VYCV1K98v1Bz+6PBEDT0aCLifGH9bs > kRdTmTg583yO/EZKljzl > =mhtb > -----END PGP SIGNATURE-----
