Re: Unpredictable performance degradation in QEMU KVMs

2021-10-06 Thread Frantisek Rysanek
A couple more points: How many CPU's (sockets) does your motherboard have? Multi-socket machines are more or less in the NUMA territory. Suboptimal process scheduling / memory allocation "decisions" (on part of the CPU/process scheduler) can have "interesting effects". Think of processes migrati

Re: Unpredictable performance degradation in QEMU KVMs

2021-10-06 Thread Frantisek Rysanek
...I forgot to mention CPU temperatures and frequencies. When overheating, modern CPU's try to limit power consumption by throttling the clock rate. The temperatures can be read at the cmdline from /sys/class/hwmon, provided that you have the right drivers loaded (for Intel CPU's I'd say "modpr

Re: Unpredictable performance degradation in QEMU KVMs

2021-10-06 Thread Frantisek Rysanek
Hello Parnell, I'm just a part-time Linux admin / enthusiast, by no means a Dev-Ops professional. I have some historical experience under the hood as a HW/OS troubleshooter. So whatever I voice here is just my "two cents worth". One last note regarding the hypothetical "architecture discrepanc

Re: Unpredictable performance degradation in QEMU KVMs

2021-10-06 Thread Parnell Springmeyer
Hi Frantisek, thanks for replying. I've not checked using `latencytop`. I will do that, thanks for the suggestion. The most frustrating problem is that the degradation in performance is so far very hard to reproduce manually so we haven't really been able to determine if it's a CPU performance is

Re: Unpredictable performance degradation in QEMU KVMs

2021-10-06 Thread Parnell Springmeyer
Hi Ken, thanks for replying. 1. We use x86-64 linux hosts and x86-64 linux guests. 2. We do run multiple guest instances, what's frustrating is that we have a hard time reproducing the degradation (even running 4-5 guests at the same time). 3. We do have enough cores and we're very careful to allo

Re: Unpredictable performance degradation in QEMU KVMs

2021-10-06 Thread Frantisek Rysanek
On 5 Oct 2021 at 18:58, Parnell Springmeyer wrote: > > Hi, we use QEMU VMs for running our integration testing > infrastructure and have run into a very difficult to debug problem: > occasionally we will see a severe performance degradation in some of > our QEMU VMs. > If memory serves, QEMU

Re: Unpredictable performance degradation in QEMU KVMs

2021-10-05 Thread Ken Moffat
On Tue, Oct 05, 2021 at 06:58:51PM -0500, Parnell Springmeyer wrote: > Hi, we use QEMU VMs for running our integration testing infrastructure and > have run into a very difficult to debug problem: occasionally we will see a > severe performance degradation in some of our QEMU VMs. > > We've tried