Re: [Qemu-devel] The status about vhost-net on kvm-arm?
Hello, Using this Qemu patchset as well as recent irqfd work, I’ve tried to make vhost-net working on Cortex-A15. Unfortunately, even if I can correctly generate irqs to the guest through irqfd, it seems to me that some pieces are still missing…. Indeed, virtio mmio interrupt status register (@ offset 0x60) is not updated by vhost thread, and reading it or writing to the peer interrupt ack register (offset 0x64) from the guest causes an VM exit … After reading older posts, I understand that vhost-net with irqfd support could only work with MSI-X support : On 01/20/2011 09:35 AM, Michael S. Tsirkin wrote: “When MSI is off, each interrupt needs to be bounced through the io thread when it's set/cleared, so vhost-net causes more context switches and higher CPU utilization than userspace virtio which handles networking in the same thread. “ Indeed, in case of MSI-X support, Virtio spec indicates that the ISR Status field is unused… I understand that Vhost does not emulate a complete virtio PCI adapter but only manage virtqueue operations. However I don’t have a clear view of what is performed by Qemu and what is performed by vhost-thread… Could someone highlight me on this point, and maybe give some clues for an implementation of Vhost with irqfd and without MSI support ??? Thanks a lot in advance. Best regards. Rémy De : kvmarm-boun...@lists.cs.columbia.edu [mailto:kvmarm-boun...@lists.cs.columbia.edu] De la part de Yingshiuan Pan Envoyé : vendredi 15 août 2014 09:25 À : Li Liu Cc : kvm...@lists.cs.columbia.edu; k...@vger.kernel.org; qemu-devel Objet : Re: [Qemu-devel] The status about vhost-net on kvm-arm? Hi, Li, It's ok, I did get those mails from mailing list. I guess it was because I did not subscribe some of mailing lists. Currently, I think I will not have any plan to renew my patcheset since I have resigned from my previous company, I do not have Cortex-A15 platform to test/verify. I'm fine with that, it would be great if you or someone can take it and improve it. Thanks. Best Regards, Yingshiuan Pan 2014-08-15 11:04 GMT+08:00 Li Liu mailto:john.li...@huawei.com>>: Hi Ying-Shiuan Pan, I don't know why for missing your mail in mailbox. Sorry about that. The results of vhost-net performance have been attached in another mail. Do you have a plan to renew your patchset to support irqfd. If not, we will try to finish it based on yours. On 2014/8/14 11:50, Li Liu wrote: > > > On 2014/8/13 19:25, Nikolay Nikolaev wrote: >> On Wed, Aug 13, 2014 at 12:10 PM, Nikolay Nikolaev >> mailto:n.nikol...@virtualopensystems.com>> >> wrote: >>> On Tue, Aug 12, 2014 at 6:47 PM, Nikolay Nikolaev >>> mailto:n.nikol...@virtualopensystems.com>> >>> wrote: Hello, On Tue, Aug 12, 2014 at 5:41 AM, Li Liu mailto:john.li...@huawei.com>> wrote: > > Hi all, > > Is anyone there can tell the current status of vhost-net on kvm-arm? > > Half a year has passed from Isa Ansharullah asked this question: > http://www.spinics.net/lists/kvm-arm/msg08152.html > > I have found two patches which have provided the kvm-arm support of > eventfd and irqfd: > > 1) [RFC PATCH 0/4] ARM: KVM: Enable the ioeventfd capability of KVM on ARM > http://lists.gnu.org/archive/html/qemu-devel/2014-01/msg01770.html > > 2) [RFC,v3] ARM: KVM: add irqfd and irq routing support > https://patches.linaro.org/32261/ > > And there's a rough patch for qemu to support eventfd from Ying-Shiuan > Pan: > > [Qemu-devel] [PATCH 0/4] ioeventfd support for virtio-mmio > https://lists.gnu.org/archive/html/qemu-devel/2014-02/msg00715.html > > But there no any comments of this patch. And I can found nothing about > qemu > to support irqfd. Do I lost the track? > > If nobody try to fix it. We have a plan to complete it about virtio-mmio > supporing irqfd and multiqueue. > > we at Virtual Open Systems did some work and tested vhost-net on ARM back in March. The setup was based on: - host kernel with our ioeventfd patches: http://www.spinics.net/lists/kvm-arm/msg08413.html - qemu with the aforementioned patches from Ying-Shiuan Pan https://lists.gnu.org/archive/html/qemu-devel/2014-02/msg00715.html The testbed was ARM Chromebook with Exynos 5250, using a 1Gbps USB3 Ethernet adapter connected to a 1Gbps switch. I can't find the actual numbers but I remember that with multiple streams the gain was clearly seen. Note that it used the minimum required ioventfd implementation and not irqfd. I guess it is feasible to think that it all can be put together and rebased + the recent irqfd work. One can achiev even better performance (because of the irqfd). >>> >>> Managed to replicate the setup with the old versions e used in March: >>> >>> Single stream from another machine to chromebook with 1Gbps USB
Re: [Qemu-devel] The status about vhost-net on kvm-arm?
Thanks for your feedback,
>static irqreturn_t vm_interrupt(int irq, void *opaque) {
> ..
>
> /* Read and acknowledge interrupts */
> /*status = readl(vm_dev->base + VIRTIO_MMIO_INTERRUPT_STATUS);
> writel(status, vm_dev->base + VIRTIO_MMIO_INTERRUPT_ACK);
>
> if (unlikely(status & VIRTIO_MMIO_INT_CONFIG)
> && vdrv && vdrv->config_changed) {
> vdrv->config_changed(&vm_dev->vdev);
> ret = IRQ_HANDLED;
> }*/
>
> //if (likely(status & VIRTIO_MMIO_INT_VRING)) {
> spin_lock_irqsave(&vm_dev->lock, flags);
> list_for_each_entry(info, &vm_dev->virtqueues, node)
> ret |= vring_interrupt(irq, info->vq);
> spin_unlock_irqrestore(&vm_dev->lock, flags);
> //}
>
> return ret;
>}
>
>This is very roughly :), and a lot of coding things need to be done.
I agree ;-)
Anyway, with this "workaround" you disable the control plane interrupt, which
is needed to bring up/down the virtio link... unless VIRTIO_NET_F_STATUS
feature is off.
I was thinking about connecting those 2 registers to an ioeventfd in order to
emulate them in Vhost and bypass Qemu... but AFAIK ioeventfd can only work with
"write" registers.
Any idea for a long term solution ?
best regards.
Rémy
-Message d'origine-
De : Li Liu [mailto:john.li...@huawei.com]
Envoyé : vendredi 17 octobre 2014 14:27
À : GAUGUEY Rémy 228890; Yingshiuan Pan
Cc : kvm...@lists.cs.columbia.edu; k...@vger.kernel.org; qemu-devel
Objet : Re: [Qemu-devel] The status about vhost-net on kvm-arm?
On 2014/10/15 22:39, GAUGUEY Rémy 228890 wrote:
> Hello,
>
> Using this Qemu patchset as well as recent irqfd work, I’ve tried to make
> vhost-net working on Cortex-A15.
> Unfortunately, even if I can correctly generate irqs to the guest through
> irqfd, it seems to me that some pieces are still missing….
> Indeed, virtio mmio interrupt status register (@ offset 0x60) is not
> updated by vhost thread, and reading it or writing to the peer
> interrupt ack register (offset 0x64) from the guest causes an VM exit
> …
>
Yeah, you are correct. But it's not far away from success if have injected irqs
to the guest through irqfd. Do below things to let guest receive packets
correctly without checking VIRTIO_MMIO_INTERRUPT_STATUS in guest virtio_mmio.c:
static irqreturn_t vm_interrupt(int irq, void *opaque) {
..
/* Read and acknowledge interrupts */
/*status = readl(vm_dev->base + VIRTIO_MMIO_INTERRUPT_STATUS);
writel(status, vm_dev->base + VIRTIO_MMIO_INTERRUPT_ACK);
if (unlikely(status & VIRTIO_MMIO_INT_CONFIG)
&& vdrv && vdrv->config_changed) {
vdrv->config_changed(&vm_dev->vdev);
ret = IRQ_HANDLED;
}*/
//if (likely(status & VIRTIO_MMIO_INT_VRING)) {
spin_lock_irqsave(&vm_dev->lock, flags);
list_for_each_entry(info, &vm_dev->virtqueues, node)
ret |= vring_interrupt(irq, info->vq);
spin_unlock_irqrestore(&vm_dev->lock, flags);
//}
return ret;
}
This is very roughly :), and a lot of coding things need to be done.
Li.
> After reading older posts, I understand that vhost-net with irqfd support
> could only work with MSI-X support :
>
> On 01/20/2011 09:35 AM, Michael S. Tsirkin wrote:
> “When MSI is off, each interrupt needs to be bounced through the io
> thread when it's set/cleared, so vhost-net causes more context switches and
> higher CPU utilization than userspace virtio which handles networking in the
> same thread.
> “
> Indeed, in case of MSI-X support, Virtio spec indicates that the ISR
> Status field is unused…
>
> I understand that Vhost does not emulate a complete virtio PCI adapter but
> only manage virtqueue operations.
> However I don’t have a clear view of what is performed by Qemu and
> what is performed by vhost-thread… Could someone highlight me on this point,
> and maybe give some clues for an implementation of Vhost with irqfd and
> without MSI support ???
>
> Thanks a lot in advance.
> Best regards.
> Rémy
>
>
>
> De : kvmarm-boun...@lists.cs.columbia.edu
> [mailto:kvmarm-boun...@lists.cs.columbia.edu] De la part de Yingshiuan
> Pan Envoyé : vendredi 15 août 2014 09:25 À : Li Liu Cc :
> kvm...@lists.cs.columbia.edu; k...@vger.kernel.org; qemu-devel Objet :
> Re: [Qemu-devel] The status about vhost-net on kvm-arm?
>
> Hi, Li,
>
> It's ok, I did get those mails from mailing list. I guess
Re: [Qemu-devel] [RFC PATCH] virtio-mmio: support for multiple irqs
Hi Shannon, >Type of backend bandwith(GBytes/sec) >virtio-net 0.66 >vhost-net 1.49 >vhost-net with irqfd2.01 > >Test cmd: ./iperf -c 192.168.0.2 -P 1 -i 10 -p 5001 -f G -t 60 Impressive results ! Could you please detail your setup ? which platform are you using and which GbE controller ? As a reference, it would be good also to have result with an iperf to the HOST to see how far we are from a native configuration... Also, I assume a pending Qemu patch is necessary to assign multiple irqs ? I'm correct ? Thanks a lot, Best regards Rémy -Message d'origine- De : Shannon Zhao [mailto:zhaoshengl...@huawei.com] Envoyé : mercredi 5 novembre 2014 09:00 À : linux-ker...@vger.kernel.org Cc : m...@redhat.com; peter.mayd...@linaro.org; john.li...@huawei.com; joel.sch...@amd.com; GAUGUEY Rémy 228890; qemu-devel@nongnu.org; n.nikol...@virtualopensystems.com; virtualizat...@lists.linux-foundation.org; peter.huangp...@huawei.com; hangaoh...@huawei.com Objet : Re: [RFC PATCH] virtio-mmio: support for multiple irqs On 2014/11/4 17:35, Shannon Zhao wrote: > As the current virtio-mmio only support single irq, so some advanced > features such as vhost-net with irqfd are not supported. And the net > performance is not the best without vhost-net and irqfd supporting. > Hi Joel, Peter, Mst, Some virtio-net with virtio-mmio performance data on ARM added as followed: Type of backend bandwith(GBytes/sec) virtio-net 0.66 vhost-net 1.49 vhost-net with irqfd2.01 Test cmd: ./iperf -c 192.168.0.2 -P 1 -i 10 -p 5001 -f G -t 60 >From this test data, irqfd has great improvement (about 30%) on performance. So maybe it's necessary to enable multiple irq support to make vhost-net with virtio-mmio on ARM be able to use irqfd. How do you guys think? Look forward for your feedback. Thanks, Shannon > This patch support virtio-mmio to request multiple irqs like > virtio-pci. With this patch and qemu assigning multiple irqs for > virtio-mmio device, it's ok to use vhost-net with irqfd on arm/arm64. >
