Hi Wido, I just ran a few tests with qemu-img, and it appears that it can indeed detect the input image format. I ran exports with no source format specified, and both -O raw and -O qcow2. I confirmed the file sizes and formats, then re-imported with no source format, and -O raw specified.
This would indicate that qemu-img (at least the version I'm testing with) would be fine with no source format specified. I don't know if the QEMU Java functions require a source format or not though. It appears that they are part of the CloudStack project, so I would think they'd be easy enough to change if needed. Thank You, Logan Barfield Tranquil Hosting On Sun, Feb 1, 2015 at 1:54 AM, Wido den Hollander <w...@widodh.nl> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > > On 01/30/2015 04:43 PM, Logan Barfield wrote: >> Hi Wido, >> >> I didn't see the format information for snapshots stored in the >> database, so I'm assuming it references the associated volume >> format. >> > > Yes, it is. > >> I assume the cleanest way to get around that would be to have a >> separate format field for snapshots, but that wouldn't really make >> sense just for this one use case. Another way would be to just >> have the restore (create template from snapshot, etc.) function >> always assume QCOW2 for RBD snapshots, but that's dirty, and would >> cause problems for existing snapshots. >> > > That would be dirty, since I don't think we can actually know if a > snapshpot was from a RBD volume when it's on SS. > > Maybe qemu-img can detect the input format? That way we don't have to > specify it. > >> It seems like the best method would be to have the restore >> function look at the actual file to get the format information >> instead of pulling it from the database, though I'm not sure if >> that's even possible. >> > > I think that would be the best option indeed. > >> Does anyone else have any thoughts or suggestions? >> >> Thank You, >> >> Logan Barfield Tranquil Hosting >> >> >> On Thu, Jan 29, 2015 at 4:21 PM, Wido den Hollander >> <w...@widodh.nl> wrote: >> >> >> On 01/29/2015 10:05 PM, Logan Barfield wrote: >>>>> Hi Wido, >>>>> >>>>> The relevant code for this is in: >>>>> >>>>> I saw that the destination format is set to QCOW2 for >>>>> "createTemplateFromVolume", but in "backupSnapshot" it's set >>>>> to use the format of the source image >>>>> "destFile.setFormat(snapshotDisk.getFormat());" >>>>> >>>>> I believe just changing the "backupSnapshot" code to the use >>>>> QCOW2 "destFile.setFormat(PhysicalDiskFormat.QCOW2);" will >>>>> work fine. I can submit a patch for that, but I wanted to >>>>> make sure there wasn't some other reason for the RAW output >>>>> first. >>>>> >> >> No, that won't work. Because the management server will then still >> think it's in RAW format and store that in the database. >> >> It has been a long time since I looked at this. I know I did some >> work in this area and made some patches. >> >> Wido >> >>>>> >>>>> Thank You, >>>>> >>>>> Logan Barfield Tranquil Hosting >>>>> >>>>> >>>>> On Thu, Jan 29, 2015 at 3:52 PM, Wido den Hollander >>>>> <w...@widodh.nl> wrote: >>>>> >>>>> >>>>> On 01/29/2015 05:16 PM, Logan Barfield wrote: >>>>>>>> I was doing some testing earlier this week on a KVM >>>>>>>> cluster, and noticed that when using S3 for secondary >>>>>>>> storage snapshots on RBD primary storage take a lot >>>>>>>> longer than they do with NFS primary storage. >>>>>>>> >>>>>>>> I think this is due to the NFS snapshots being >>>>>>>> output/uploaded in QCOW2 format, while RBD snapshots >>>>>>>> are output in RAW format. It appears that even when >>>>>>>> using sparse RAW files they are uploaded to S3 with the >>>>>>>> 'allocated' size instead of the sparse size. >>>>>>>> >>>>> >>>>> The problem here lies in the code in CloudStack. Since the >>>>> source volume is in RAW format, the snapshot also becomes >>>>> QCOW2. >>>>> >>>>> I remember that I wrote some patches for this, you might want >>>>> to look in the Git history. >>>>> >>>>>>>> My question is: Is there a good reason for RBD >>>>>>>> snapshots to be output as RAW instead of QCOW2? It >>>>>>>> appears that QCOW2 templates are automatically >>>>>>>> converted when deploying them as RBD, so that shouldn't >>>>>>>> be a problem. The only downside I can think of would >>>>>>>> be having to convert the RAW RBD data to QCOW2 as an >>>>>>>> extra step when creating snapshots, but even that seems >>>>>>>> like it would take less time than uploading the RAW >>>>>>>> images to S3. >>>>>>>> >>>>>>>> Thoughts? >>>>>>>> >>>>> >>>>> Like I said, I think I wrote patches for this. Qemu-img is >>>>> used to convert the RAW RBD into the snapshot, so QCOW2 >>>>> should be no problem as output format. But the code in >>>>> CloudStack seems to assume that the snapshot volume is always >>>>> the same format as the source. >>>>> >>>>> Wido >>>>> >>>>> P.S.: I'm on vacation in Norway at the moment, so I'm pretty >>>>> laggy with replying. >>>>> >>>>>>>> >>>>>>>> Thank You, >>>>>>>> >>>>>>>> Logan Barfield Tranquil Hosting >>>>>>>> > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1 > > iQIcBAEBAgAGBQJUzc2zAAoJEAGbWC3bPspCLFAP/0FuGqOTlo4DW40tP4v70zQ5 > cjF1s9rg9Afd4OGc6KmcG2OxR/A68xMuHxJCPwVZjFlcNLMEWx8goxmJd9qLQaMM > es1xHynmSO3owhA9d7bRhvfL4M9mf/w/oSn1IQZ5l9v7bJX9y5iGuPkMPlBT5yej > yFWA92AAEVgne1J/E4JvoZbDoyAYVecrDPmFD8oP/9R2I7MlUr5134aShBIhJPM/ > UgsY06dKfAWI2cW5vZxzw4haNTk68BMgjp6IBkirHqLL1wZa7R83c8/ygXsbvStJ > z5bLrxtLtwy6d3utjwJkfdtdnAfuiqZFOKdgxvhlS30Q0Y2XGLXfA3+DLVGmDBF5 > xksxiAvhpvxkJIMYZx5icFJmNVu5NiaNn6yKRTkJzA687emF4fSjn1pgTHMp+hhB > gkZcYiazrrdFU0ibFkQMNWhT80Pc9spcJEMFkX32GnXTK8EOR7hqRpGrS1LLQjnk > XrukOOOjlHnQnRS8kTP9y2Rr5zBUnB6kVwd8DYCJGYdQUPlHaMOGPC9che7HzmFg > qzNPHm/WakWotlAldYOPFzmoV16jzF+j6lpgnREZMTq3DNVk6fRH4vzjIvlWzsbF > Rrq+7KbNxYc0SJfmTBh0jV/Zpm85ed0kYzXBnekq8ryItF1e2P5t4zoVcC7XHb9n > Y2Qvi+YYYZPW6fYIiznr > =TcvA > -----END PGP SIGNATURE-----
