[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2019-08-09 Thread John Snow
Hi, there isn't really a development branch; if '-S 0' didn't help alleviate the problem there may be other problems at hand, or the APFS implementation of SEEK_DATA is causing us even more problems in new versions. You could try Yan-Jie Wang's patch that was posted in #20, but until it's posted t

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2019-07-23 Thread Sven R
Hi, I recently ran into problems and after a long time trying to find out the cause landed here, I got in trouble using a CentOs Cloud image: https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1905.qcow2.xz which extracts to a .qcow2 image with sha256 of: b376afdc0150601f15e535

Re: [Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-09-07 Thread Eric Blake
On 09/07/2018 01:04 PM, Yan-Jie Wang wrote: I have done some experiments and find out that the behavior of lseek with whence set to SEEK_DATA is different from the behavior of Linux's lseek. If the supplied offset is in the middle of a data region, it returns the start of the next data region.

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-09-07 Thread Yan-Jie Wang
I have done some experiments and find out that the behavior of lseek with whence set to SEEK_DATA is different from the behavior of Linux's lseek. If the supplied offset is in the middle of a data region, it returns the start of the next data region. There may be many data regions in a big file

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-07-25 Thread John Snow
I marked https://bugs.launchpad.net/qemu/+bug/1738840 as a duplicate of this bug, even though that bug was older, this bug has a slightly more active thread. I'm still in need of either an APFS capable machine that I can reproduce on, or another mac dev willing to help a bit, though. -- You rece

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-07-06 Thread Waldemar Kozaczuk
Bingo! Adding '-S 0' makes convert work. But it is not perfect as the end result is fully allocated image. So with qcow2 like this: image: mysql-example.qemu file format: qcow2 virtual size: 10G (10737418240 bytes) disk size: 50M cluster_size: 65536 Format specific information: compat: 1.1

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-06-21 Thread John Snow
My hunch is that we're handling zero regions incorrectly and we might be skipping data regions we ought to be copying. ...I'm looking at the `image2.qemu` file you've uploaded and it's empty! we didn't copy *anything* from your source file. Try converting with '-S 0` to disable sparse protection

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-06-21 Thread Waldemar Kozaczuk
I am not familiar with QEMU source code but I might have some cycles to look into it. Where would I look - https://github.com/qemu/qemu/blob/master/qemu-img.c? Or somewhere else? Any suggestions would be appreciated. -- You received this bug notification because you are a member of qemu- devel-ml

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-06-20 Thread John Snow
Great, thanks for the additional info! It sounds like maybe something to do with our sparse detection on APFS is broken. Maybe lseek calls are failing? This is a bad problem because it means that any read from an APFS source could break similarly. I'm still a bit tied down with other work at the

Re: [Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-06-20 Thread Waldemar Kozaczuk
I have done more tests based on your suggestion and I think it is the opposite. The problem happens is the source is APFS. Following failed: APFS -> ExFAT APFS -> Fat32 Following worked: ExFAT -> APFS FAT32 -> APFS So for now my workaround is to use USB stick formatted with FAT32 or ExFAT, copy

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-06-20 Thread John Snow
I haven't had the time to look just yet. If there's a developer out there with a Mac has the bandwidth to take a look at this I'd be grateful. (I don't have access to one presently.) I see that the target filesystem is APFS however -- I think we might have a bug in our APFS support. Do you have t

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-06-19 Thread Waldemar Kozaczuk
Have I provided all necessary data and other details? -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1776920 Title: qemu-img convert on Mac OSX creates corrupt images Status in QEMU: New Bug des

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-06-14 Thread Waldemar Kozaczuk
Raw image created by dd in steps 1 and 2. ** Attachment added: "image.img" https://bugs.launchpad.net/qemu/+bug/1776920/+attachment/5152722/+files/image.img -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.n

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-06-14 Thread Waldemar Kozaczuk
Source file 1 ** Attachment added: "lzloader.elf" https://bugs.launchpad.net/qemu/+bug/1776920/+attachment/5152720/+files/lzloader.elf -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1776920 Title

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-06-14 Thread Waldemar Kozaczuk
Also if I use the same image.qemu file and convert to vmdk format I get even smaller file which for sure is wrong as well: qemu-img convert image.qemu -O vmdk image2.vbox ll image* -rw-r--r-- 1 *** *** 6684672 Jun 14 17:17 image.img -rw-r--r-- 1 *** *** 7012352 Jun 14 17:40 image.qemu -rw-r--

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-06-14 Thread Waldemar Kozaczuk
The corrupt qcow2 image created by converting image.qemu in step 4. ** Attachment added: "image2.qemu" https://bugs.launchpad.net/qemu/+bug/1776920/+attachment/5152724/+files/image2.qemu -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-06-14 Thread Waldemar Kozaczuk
Source file 2 ** Attachment added: "boot.bin" https://bugs.launchpad.net/qemu/+bug/1776920/+attachment/5152721/+files/boot.bin -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1776920 Title: qemu

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-06-14 Thread Waldemar Kozaczuk
Original qcow2 image converted from raw image in step 3. ** Attachment added: "image.qemu" https://bugs.launchpad.net/qemu/+bug/1776920/+attachment/5152723/+files/image.qemu -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https:/

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-06-14 Thread Waldemar Kozaczuk
I believe I have distilled entire process to few repeatable steps that can be fully reproduced on my Mac. The binary source files - - boot.bin and lzloader.elf - were created on my Linux VM running in VirtualBox on same Mac but I do not think it matters as the execution completely happens on Mac.

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-06-14 Thread John Snow
Are you converting to a destination on APFS? If yes, do you have the ability to easily reproduce this by converting to a non-APFS destination and seeing if that breaks too? Thanks, --js -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU.

Re: [Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-06-14 Thread Christian Ehrhardt
> > I will provide all necessary info. Unfortunately the smallest image I can > provide is around 10M. > > What is M1/M2/M3 L1/L2/L3 file? > Just a suggestion how you could name the files Linux-at-step-1 would be L1 and similar.

Re: [Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-06-14 Thread Waldemar Kozaczuk
I will provide all necessary info. Unfortunately the smallest image I can provide is around 10M. What is M1/M2/M3 L1/L2/L3 file? Waldek On Thu, Jun 14, 2018 at 10:46 AM,  Christian Ehrhardt  < 1776...@bugs.launchpad.net> wrote: > Can this be done with like a 1M example file that you could cop

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

2018-06-14 Thread  Christian Ehrhardt 
Can this be done with like a 1M example file that you could copy off in all stages. Provide the commands you use like 1. create 2. do ?? 3. convert Then for Mac and Linux you'd have M1/M2/M3 L1/L2/L3 files that can all be attached here to be evaluated for what might be broken. IMHO In the curren