Re: [Question] Reduce the msix_load cost for VFIO device

[Longpeng (Mike, Cloud Infrastructure Service Product Dept.)]

在 2021/8/3 22:19, Alex Williamson 写道:
> On Tue, 3 Aug 2021 16:43:07 +0800
> "Longpeng (Mike, Cloud Infrastructure Service Product Dept.)"
> <longpe...@huawei.com> wrote:
> 
>> Hi Alex,
>>
>> We found that the msix_load() will cost 40~50ms if the VF has 60+ interrupts,
>> the following code cost too much for each interrupt:
>>
>> msix_load:
>>   for (vector = 0; vector < 60; vector++)
>>     msix_handle_mask_update
>>       vfio_msix_vector_do_use
>>         vfio_add_kvm_msi_virq
>>           kvm_irqchip_add_msi_route
>>             kvm_irqchip_commit_routes <-- cost 0.8ms each time
>>
>> In irq remapping mode, the VF interrupts are not routed through KVM irqchip
> 
> I'm not sure what this means.  Your MSIX interrupts are going through
> QEMU anyway?  Why?
> 

Um ... I made a mistake, the KVM irqchip commit operation can not be skip, it
cause the VF interrupts pass to the QEMU, it's harmful for the performance.

>> in fact, so maybe we can reduce this cost by "x-no-kvm-msix=ture", right?
>> Are there any risks if we do in this way ?
> 
> You're obviously free to configure your device this way, but the
> expectation is that any sort of startup latency is more than offset by
> improved runtime latency through the KVM route.  This option is usually
> reserved for debugging, when we want to see all interaction with the
> device in QEMU.
> 
> If there's a case where we're not detecting that a KVM route is
> ineffective, then we should figure out how to detect that and skip this
> code, but again the expectation is that the KVM route is worthwhile.
> 
> If this is specifically about kvm_irqchip_commit_routes(), maybe the
> setup path needs a way to batch multiple routes and defer the commit,
> if that's possible.  Thanks,

Agree.

I had wrote a draft that defer to commit, it works. In addition, the
vfio_disable_irqindex/vfio_enable_vectors are called N times in msix_load path,
they cost much more than kvm_irqchip_commit, it's also worth to optimize.

> 
> Alex
> 
> .
>