On 02/06/2017 11:08 PM, Abdulla Bubshait wrote: > Hey Thomas, > > Great analysis, thanks for sharing it all. But I had a question. You > mention setting realtime pri on the VM threads, and how it works with > NO_HZ_FULL. And I was wondering wouldn't you be able to to just use > isolcpus in cmd line to leave the VM cores to the VM as an alternative?
In this test I rely on cpuset to isolate the VM cores for exclusive use by the VM but cpuset can't completely isolate cores. There are always some kernel threads remaining that might interfere with the VM. I've never tested isolcpus but I've read that is has the same problem and can't migrate all kthreads. Cpuset can be turned on and off dynamical so it's a lot more convenient. To find out what processes run on a core try running. "perf record -e "sched:sched_switch" -C 1,2,3" where -C is the cores you want to inspect. Let it run for a while and then run "perf report --fields=sample,overhead,cpu,comm". If you have isolated the cores so only the VM may run on them you'd expect to only see the "CPU #/KVM" processes but there will probably be other stuff running as well. Swapper is the kernel thread doing busy wait so it's normal to see that. Kvm-pit is some kernel thread for handling the virtual pit so if the VM use the pit you'll see that. Besides those you'll probably also see a bunch of kworker threads. By using realtime pri the VM threads can out-preempt them. https://www.kernel.org/doc/Documentation/kernel-per-CPU-kthreads.txt This kernel doc describe a lot of tricks for disabling those kernel threads that can't be migrated. The tricks I rely on in this test was to move all interrupts to the housekeeping cpu and run the script below. # the kernel's dirty page writeback mechanism uses kthread workers. They introduce # massive arbitrary latencies when doing disk writes on the host and aren't # migrated by cset. Restrict the workqueues to use only cpu 0. echo 1 > /sys/devices/virtual/workqueue/cpumask # The CONFIG_LOCKUP_DETECTOR watchdog will wake up occasionally resulting in jitter # temporary disable it. echo 0 > /proc/sys/kernel/watchdog # The vmstat_update worker can't be disabled but it can be delayed a bit. # makes the statistics in /proc/vmstat imprecise. echo 300 > /proc/sys/vm/stat_interval # THP can allegedly result in OS jitter. Better keep it off. echo never > /sys/kernel/mm/transparent_hugepage/enabled I also compile my kernel with these options. CONFIG_RCU_NOCB_CPU_ALL Makes the RCU callbacks run in kthreads which can be migrated to housekeeping cpu CONFIG_WQ_POWER_EFFICIENT_DEFAULT: The documentation for this option suggest it makes it possible to migrate more kthreads but I haven't confirmed that. CONFIG_NO_HZ_FULL_ALL Turns off the kernel tick on cpus with only one process to make that process run uninterrupted. _______________________________________________ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
