Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-07-11 Thread Alberto Garcia
On Thu 11 Jul 2019 04:32:34 PM CEST, Kevin Wolf wrote: >> - It is possible to configure very easily the number of subclusters per >> cluster. It is now hardcoded to 32 in qcow2_do_open() but any power of >> 2 would work (just change the number there if you want to test >> it). Would an optio

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-07-11 Thread Kevin Wolf
Am 11.07.2019 um 16:08 hat Alberto Garcia geschrieben: > Some questions that are still open: > > - It is possible to configure very easily the number of subclusters per > cluster. It is now hardcoded to 32 in qcow2_do_open() but any power of > 2 would work (just change the number there if you

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-07-11 Thread Alberto Garcia
On Thu 27 Jun 2019 07:08:29 PM CEST, Denis Lunev wrote: > But can we get a link to the repo with actual version of patches. Hi, I updated my code to increase the L2 entry size from 64 bits to 128 bits and thanks to this we now have 32 subclusters per cluster (32 bits for "subcluster allocated" a

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-30 Thread Kevin Wolf
Am 28.06.2019 um 17:12 hat Alberto Garcia geschrieben: > On Fri 28 Jun 2019 05:09:11 PM CEST, Kevin Wolf wrote: > > Am 28.06.2019 um 17:02 hat Alberto Garcia geschrieben: > >> On Fri 28 Jun 2019 04:57:08 PM CEST, Kevin Wolf wrote: > >> > Am 28.06.2019 um 16:43 hat Alberto Garcia geschrieben: > >> >

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
On Thu 27 Jun 2019 07:08:29 PM CEST, Denis Lunev wrote: > But can we get a link to the repo with actual version of patches. It's not in a state that can be published at the moment, but I'll try to have something available soon. Berto

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
On Fri 28 Jun 2019 04:16:28 PM CEST, Kevin Wolf wrote: >> >> >> >> I would consider 64k cluster/8k subcluster as too extreme >> >> >> >> for me. >> >> >> >> >> >> I forgot to add: this 64k/8k ratio is only with my current >> >> >> prototype. >> >> >> >> >> >> In practice if we go with the 128-bit

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Kevin Wolf
Am 28.06.2019 um 17:10 hat Alberto Garcia geschrieben: > On Fri 28 Jun 2019 05:03:13 PM CEST, Denis Lunev wrote: > > On 6/28/19 6:02 PM, Alberto Garcia wrote: > > Please note, I am not talking now about your case with COW. Here the > > allocation is performed on the sub-cluster basis, i.e.

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
On Fri 28 Jun 2019 05:09:11 PM CEST, Kevin Wolf wrote: > Am 28.06.2019 um 17:02 hat Alberto Garcia geschrieben: >> On Fri 28 Jun 2019 04:57:08 PM CEST, Kevin Wolf wrote: >> > Am 28.06.2019 um 16:43 hat Alberto Garcia geschrieben: >> >> On Thu 27 Jun 2019 06:05:55 PM CEST, Denis Lunev wrote: >> >> >

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Kevin Wolf
Am 28.06.2019 um 17:02 hat Alberto Garcia geschrieben: > On Fri 28 Jun 2019 04:57:08 PM CEST, Kevin Wolf wrote: > > Am 28.06.2019 um 16:43 hat Alberto Garcia geschrieben: > >> On Thu 27 Jun 2019 06:05:55 PM CEST, Denis Lunev wrote: > >> > Please note, I am not talking now about your case with COW.

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Denis Lunev
On 6/28/19 6:02 PM, Alberto Garcia wrote: > On Fri 28 Jun 2019 04:57:08 PM CEST, Kevin Wolf wrote: >> Am 28.06.2019 um 16:43 hat Alberto Garcia geschrieben: >>> On Thu 27 Jun 2019 06:05:55 PM CEST, Denis Lunev wrote: Please note, I am not talking now about your case with COW. Here the all

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Kevin Wolf
Am 28.06.2019 um 15:19 hat Alberto Garcia geschrieben: > On Fri 28 Jun 2019 12:04:22 PM CEST, Kevin Wolf wrote: > > Am 28.06.2019 um 11:53 hat Alberto Garcia geschrieben: > >> On Fri 28 Jun 2019 11:20:57 AM CEST, Kevin Wolf wrote: > >> >> >> I would consider 64k cluster/8k subcluster as too extreme

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
On Fri 28 Jun 2019 05:03:13 PM CEST, Denis Lunev wrote: > On 6/28/19 6:02 PM, Alberto Garcia wrote: > Please note, I am not talking now about your case with COW. Here the > allocation is performed on the sub-cluster basis, i.e. the abscence of > the sub-cluster in the image means hole o

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
On Fri 28 Jun 2019 04:57:08 PM CEST, Kevin Wolf wrote: > Am 28.06.2019 um 16:43 hat Alberto Garcia geschrieben: >> On Thu 27 Jun 2019 06:05:55 PM CEST, Denis Lunev wrote: >> > Please note, I am not talking now about your case with COW. Here the >> > allocation is performed on the sub-cluster basis,

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Kevin Wolf
Am 28.06.2019 um 16:43 hat Alberto Garcia geschrieben: > On Thu 27 Jun 2019 06:05:55 PM CEST, Denis Lunev wrote: > > Please note, I am not talking now about your case with COW. Here the > > allocation is performed on the sub-cluster basis, i.e. the abscence of > > the sub-cluster in the image means

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Denis Lunev
On 6/28/19 5:43 PM, Alberto Garcia wrote: > On Thu 27 Jun 2019 06:05:55 PM CEST, Denis Lunev wrote: Thus in respect to this patterns subclusters could give us benefits of fast random IO and good reclaim rate. >>> Exactly, but that fast random I/O would only happen when allocating >>> new

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
On Thu 27 Jun 2019 06:05:55 PM CEST, Denis Lunev wrote: >>> Thus in respect to this patterns subclusters could give us benefits >>> of fast random IO and good reclaim rate. >> Exactly, but that fast random I/O would only happen when allocating >> new clusters. Once the clusters are allocated it doe

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
On Fri 28 Jun 2019 12:04:22 PM CEST, Kevin Wolf wrote: > Am 28.06.2019 um 11:53 hat Alberto Garcia geschrieben: >> On Fri 28 Jun 2019 11:20:57 AM CEST, Kevin Wolf wrote: >> >> >> I would consider 64k cluster/8k subcluster as too extreme for me. >> >> >> >> I forgot to add: this 64k/8k ratio is onl

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
I pressed "send" too early, here's the last part of my reply: On Fri 28 Jun 2019 02:57:56 PM CEST, Alberto Garcia wrote: >> I also ran some tests on a rotating HDD drive. Here having >> subclusters doesn't make a big difference regardless of whether there >> is a backing image or not, so we can ig

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
On Thu 27 Jun 2019 06:54:34 PM CEST, Kevin Wolf wrote: >> |-++-| >> | Cluster size | subclusters=on | subclusters=off | >> |-++-| >> | 2 MB (256 KB) | 571 IOPS | 124 IOPS | >> | 1 MB

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Kevin Wolf
Am 28.06.2019 um 11:53 hat Alberto Garcia geschrieben: > On Fri 28 Jun 2019 11:20:57 AM CEST, Kevin Wolf wrote: > >> >> I would consider 64k cluster/8k subcluster as too extreme for me. > >> > >> I forgot to add: this 64k/8k ratio is only with my current prototype. > >> > >> In practice if we go

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Alberto Garcia
On Fri 28 Jun 2019 11:20:57 AM CEST, Kevin Wolf wrote: >> >> I would consider 64k cluster/8k subcluster as too extreme for me. >> >> I forgot to add: this 64k/8k ratio is only with my current prototype. >> >> In practice if we go with the 128-bit L2 entries we would have 64 >> subclusters per clu

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-28 Thread Kevin Wolf
Am 27.06.2019 um 17:42 hat Alberto Garcia geschrieben: > On Thu 27 Jun 2019 05:38:56 PM CEST, Alberto Garcia wrote: > >> I would consider 64k cluster/8k subcluster as too extreme for me. > > I forgot to add: this 64k/8k ratio is only with my current prototype. > > In practice if we go with the 12

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-27 Thread Denis Lunev
[snip] >> === >> >> And I think that's all. As you can see I didn't want to go much into >> the open technical questions (I think the on-disk format would be the >> main one), the first goal should be to decide whether this is still an >> interesting feature or not. >> >> So

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-27 Thread Kevin Wolf
Am 27.06.2019 um 15:59 hat Alberto Garcia geschrieben: > Hi all, > > a couple of years ago I came to the mailing list with a proposal to > extend the qcow2 format to add subcluster allocation. > > You can read the original message (and the discussion thread that came > afterwards) here: > >h

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-27 Thread Denis Lunev
On 6/27/19 6:38 PM, Alberto Garcia wrote: > On Thu 27 Jun 2019 04:19:25 PM CEST, Denis Lunev wrote: > >> Right now QCOW2 is not very efficient with default cluster size (64k) >> for fast performance with big disks. Nowadays ppl uses really BIG >> images and 1-2-3-8 Tb disks are really common. Unfor

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-27 Thread Alberto Garcia
On Thu 27 Jun 2019 05:38:56 PM CEST, Alberto Garcia wrote: >> I would consider 64k cluster/8k subcluster as too extreme for me. I forgot to add: this 64k/8k ratio is only with my current prototype. In practice if we go with the 128-bit L2 entries we would have 64 subclusters per cluster, or 32 if

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-27 Thread Alberto Garcia
On Thu 27 Jun 2019 04:19:25 PM CEST, Denis Lunev wrote: > Right now QCOW2 is not very efficient with default cluster size (64k) > for fast performance with big disks. Nowadays ppl uses really BIG > images and 1-2-3-8 Tb disks are really common. Unfortunately ppl want > to get random IO fast too.

Re: [Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-27 Thread Denis Lunev
On 6/27/19 4:59 PM, Alberto Garcia wrote: > Hi all, > > a couple of years ago I came to the mailing list with a proposal to > extend the qcow2 format to add subcluster allocation. > > You can read the original message (and the discussion thread that came > afterwards) here: > >https://lists.gnu

[Qemu-devel] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-06-27 Thread Alberto Garcia
Hi all, a couple of years ago I came to the mailing list with a proposal to extend the qcow2 format to add subcluster allocation. You can read the original message (and the discussion thread that came afterwards) here: https://lists.gnu.org/archive/html/qemu-block/2017-04/msg00178.html The d