On Mon, Jul 20, 2015 at 10:38:58AM -0500, Wei Huang wrote:
>
>
> On 07/20/2015 06:07 AM, Peter Maydell wrote:
> > On 16 July 2015 at 19:38, Wei Huang <w...@redhat.com> wrote:
> >> Recently we found that virtio-console devices consumes lots AArch64 guest
> >> memory, roughly 1GB with 8 devices. After debugging, it turns out that lots
> >> of factors contribute to this problem: i) guest PAGE_SIZE=64KB, ii)
> >> virtio-mmio based devices, and iii) virtio-console device. Here is the
> >> detailed analysis:
> >>
> >> 1. First, during initialization, virtio-mmio driver in guest pokes vq
> >> size by reading VIRTIO_MMIO_QUEUE_NUM_MAX (see virtio_mmio.c file).
> >> 2. QEMU returns VIRTQUEUE_MAX_SIZE (1024) to guest VM; And virtio-mmio uses
> >> it as the default vq size.
> >> 3. virtio-console driver allocates vring buffers based on this value (see
> >> add_inbuf() function of virtio_console.c file). Because PAGE_SIZE=64KB,
> >> ~64MB is allocated for each virtio-console vq.
> >>
> >> This patch addresses the problem by returning the iniatlized vring size
> >> when VM queries QEMU about VIRTIO_MMIO_QUEUE_NUM_MAX. This is similar to
> >> virtio-pci's approach. By doing this, the vq memory consumption is reduced
> >> substantially.
> >
> > I don't know if this patch is sensible to apply anyway, but from
> > this description this really sounds like a guest kernel bug.
> > QEMU tells the kernel the maximum queue size it can cope with,
> > and if the guest kernel cares about not using insane amounts of
> > RAM on queues then it should not blindly use the maximum size
> > but restrict it itself...
> Yes, this is another way of solving the problem. I think there are three
> alternatives:
>
> 1. Fix the return value of MMIO_QUEUE_NUM_MAX in QEMU (my patch);
> 2. In guest VM, virtio-mmio shouldn't query MMIO_QUEUE_NUM_MAX. Instead,
> it should behave similarly to virtio-pci driver which queries
> VIRTIO_PCI_QUEUE_NUM instead. However this approach requires
> modification of VIRTIO Specification as VIRTIO_PCI_QUEUE_NUM is
> WRITE-ONLY in virtio-mmio.
> 3. Fix virtio_console driver in guest VM. This driver currently takes in
> info->num and allocates memory based on its value. Apparently we can put
> a upper-limit on it.
>
> Your suggestion could fall in to (2) or (3). Any preference?
I tend to agree with Peter here that (3) is the correct way to fix this.
The driver should be doing something like this in its setup-vq
num = guest-preferred-queue-depth
max-num = readl(VIRTIO_MMIO_QUEUE_NUM_MAX)
if (num > max-num)
num = max-num
writel(num, VIRTIO_MMIO_QUEUE_NUM)
But I guess guest-preferred-queue-depth is currently the max, i.e.
writel(readl(VIRTIO_MMIO_QUEUE_NUM_MAX), VIRTIO_MMIO_QUEUE_NUM)
which probably isn't necessary, and a more reasonable size should be
selected.
Thanks,
drew
>
> Thanks,
> -Wei
>
>
>
> >
> > thanks
> > -- PMM
> >
>
