[Qemu-devel] QEMU ARM SMP: IPI delivery delayed until next main loop event // how to improve IPI latency?

[Alex Züpke]

Hi,

I'm benchmarking some IPI (== inter-processor-interrupt) synchronization stuff 
of my custom kernel on QEMU ARM (qemu-system-arm -M vexpress-a15 -smp 2) and 
ran into the following problem: pending IPIs are delayed until the QEMU main 
loop receives an event (for example the timer interrupt expires or I press a 
key on the console).

The following timing diagram tries to show this:

  CPU #0                       CPU #1
  ======                       ======
  ... other stuff ...          WFI (wait for interrupt, like x86 "HLT")
  send SGI in MPCore
  polls for completeness
                 <time passes ...>
  polls ...
                 <... and passes ...>
  still polls ...
                 <... and passes ...>
  still polls ...
                 <... and passes ...>

  
                 <timer interrupt expires>
                 <now QEMU switches to CPU #1>
                               receives IPI
                               signals completeness
                               WFI
                 <QEMU switches to CPU #0>
  polling done
  process timer interrupt
  ...


My timer is setup to generate an interrupt once a second, so I only get 1 IPI 
interrupt per second on QEMU. When I run the test on real hardware (i.MX6Q), I 
get millions of IPIs instead.


I tried to "fix" this by forcing QEMU back into the main loop and added a call 
to qemu_notify_event() in the IPI-sending path of the ARM interrupt controller:


diff --git a/hw/intc/arm_gic.c b/hw/intc/arm_gic.c
index c1d2e70..20dba75 100644
--- a/hw/intc/arm_gic.c
+++ b/hw/intc/arm_gic.c
@@ -21,6 +21,7 @@
 #include "hw/sysbus.h"
 #include "gic_internal.h"
 #include "qom/cpu.h"
+#include "qemu/main-loop.h"
 
 //#define DEBUG_GIC
 
@@ -898,6 +899,7 @@ static void gic_dist_writel(void *opaque, hwaddr offset,
             target_cpu = ctz32(mask);
         }
         gic_update(s);
+        qemu_notify_event();
         return;
     }
     gic_dist_writew(opaque, offset, value & 0xffff, attrs);



It works as expects (I get thousands of IPIs per second now), but it does not 
"feel right", so is there a better way to improve the responsiveness of IPI 
handling in QEMU?

Best regards
Alex