Re: KVM lock contention on 48 core AMD machine

2011-03-21 Thread Avi Kivity
On 03/21/2011 08:48 PM, Michael Tokarev wrote: 21.03.2011 20:12, Eric Dumazet wrote: > Le lundi 21 mars 2011 à 19:02 +0200, Avi Kivity a écrit : > >> Any ideas on how to fix it? We could pre-allocate IDs and batch them in >> per-cpu caches, but it seems like a lot of work. >> > > Hmm, I dont

Re: KVM lock contention on 48 core AMD machine

2011-03-21 Thread Michael Tokarev
21.03.2011 20:12, Eric Dumazet wrote: > Le lundi 21 mars 2011 à 19:02 +0200, Avi Kivity a écrit : > >> Any ideas on how to fix it? We could pre-allocate IDs and batch them in >> per-cpu caches, but it seems like a lot of work. >> > > Hmm, I dont know what syscalls kvm do, but even a timer_getti

Re: KVM lock contention on 48 core AMD machine

2011-03-21 Thread Avi Kivity
On 03/21/2011 07:12 PM, Eric Dumazet wrote: Le lundi 21 mars 2011 à 19:02 +0200, Avi Kivity a écrit : > Any ideas on how to fix it? We could pre-allocate IDs and batch them in > per-cpu caches, but it seems like a lot of work. > Hmm, I dont know what syscalls kvm do, but even a timer_gettime

Re: KVM lock contention on 48 core AMD machine

2011-03-21 Thread Ben Nagy
On Mon, Mar 21, 2011 at 10:57 PM, Eric Dumazet wrote: > Le lundi 21 mars 2011 à 19:02 +0200, Avi Kivity a écrit : > >> Any ideas on how to fix it?  We could pre-allocate IDs and batch them in >> per-cpu caches, but it seems like a lot of work. >> > > Hmm, I dont know what syscalls kvm do, but even

Re: KVM lock contention on 48 core AMD machine

2011-03-21 Thread Eric Dumazet
Le lundi 21 mars 2011 à 19:02 +0200, Avi Kivity a écrit : > Any ideas on how to fix it? We could pre-allocate IDs and batch them in > per-cpu caches, but it seems like a lot of work. > Hmm, I dont know what syscalls kvm do, but even a timer_gettime() has to lock this idr_lock. Sounds crazy, a

Re: KVM lock contention on 48 core AMD machine

2011-03-21 Thread Avi Kivity
On 03/21/2011 06:54 PM, Eric Dumazet wrote: Le lundi 21 mars 2011 à 22:01 +0545, Ben Nagy a écrit : > >> On Mon, Mar 21, 2011 at 7:38 PM, Avi Kivity wrote: > >> > In the future, please post the binary perf.dat. > >> > >> Hi Avi, > >> > >> How do I do that? > > > > 'make nconfig'

Re: KVM lock contention on 48 core AMD machine

2011-03-21 Thread Eric Dumazet
Le lundi 21 mars 2011 à 22:01 +0545, Ben Nagy a écrit : > >> On Mon, Mar 21, 2011 at 7:38 PM, Avi Kivity wrote: > >> > In the future, please post the binary perf.dat. > >> > >> Hi Avi, > >> > >> How do I do that? > > > > 'make nconfig' and go to the kernel hacking section. > > Imprecise question

Re: KVM lock contention on 48 core AMD machine

2011-03-21 Thread Avi Kivity
On 03/21/2011 06:16 PM, Ben Nagy wrote: >> On Mon, Mar 21, 2011 at 7:38 PM, Avi Kivitywrote: >> >In the future, please post the binary perf.dat. >> >> Hi Avi, >> >> How do I do that? > > 'make nconfig' and go to the kernel hacking section. Imprecise question sorry, I meant how do I

Re: KVM lock contention on 48 core AMD machine

2011-03-21 Thread Ben Nagy
>> On Mon, Mar 21, 2011 at 7:38 PM, Avi Kivity  wrote: >> >  In the future, please post the binary perf.dat. >> >> Hi Avi, >> >> How do I do that? > > 'make nconfig' and go to the kernel hacking section. Imprecise question sorry, I meant how do I get the perf.dat not how do I disable the debugging

Re: KVM lock contention on 48 core AMD machine

2011-03-21 Thread Avi Kivity
(restored kvm@vger) On 03/21/2011 05:44 PM, Ben Nagy wrote: On Mon, Mar 21, 2011 at 7:38 PM, Avi Kivity wrote: > In the future, please post the binary perf.dat. Hi Avi, Thanks for the help! How do I do that? I'll try again on the non lock debug kernel and let you know. 'make nconfig' a

Re: KVM lock contention on 48 core AMD machine

2011-03-21 Thread Avi Kivity
On 03/21/2011 03:41 PM, Ben Nagy wrote: perf report output (huge, sorry) http://paste.ubuntu.com/583311/ In the future, please post the binary perf.dat. Anyway, I see 39.87% kvm [kernel.kallsyms] [k] delay_tsc | --- dela

Re: KVM lock contention on 48 core AMD machine

2011-03-21 Thread Ben Nagy
On Mon, Mar 21, 2011 at 5:28 PM, Ben Nagy wrote: > Bit of a status update: [...] > Since the underlying configs have changed so much, I will set up, run > all the tests again and then followup if things are still locky. Which they are. I will need to get the git sources in order to build with d

Re: KVM lock contention on 48 core AMD machine

2011-03-21 Thread Ben Nagy
On Mon, Mar 21, 2011 at 3:35 PM, Avi Kivity wrote: > It's actually not kvm code, but timer code.  Please build qemu with > ./configure --disable-strip and redo. Bit of a status update: Removing the usb tablet device helped a LOT - lowered the number of machines we could bring up until the lock %

Re: KVM lock contention on 48 core AMD machine

2011-03-21 Thread Avi Kivity
On 03/19/2011 06:45 AM, Ben Nagy wrote: On Fri, Mar 18, 2011 at 7:30 PM, Joerg Roedel wrote: > Hi Ben, > > Can you try to run > > # perf record -a -g > > for a while when your VMs are up and unresponsive? Hi Joerg, Thanks for the response. Here's that output: http://paste.ubuntu.com/58234

Re: KVM lock contention on 48 core AMD machine

2011-03-18 Thread Ben Nagy
On Fri, Mar 18, 2011 at 7:30 PM, Joerg Roedel wrote: > Hi Ben, > > Can you try to run > > # perf record -a -g > > for a while when your VMs are up and unresponsive? Hi Joerg, Thanks for the response. Here's that output: http://paste.ubuntu.com/582345/ Looks like it's spending all its time in k

Re: KVM lock contention on 48 core AMD machine

2011-03-18 Thread Stefan Hajnoczi
On Fri, Mar 18, 2011 at 12:02 PM, Ben Nagy wrote: > KVM commandline (using libvirt): > LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin > QEMU_AUDIO_DRV=none /usr/local/bin/kvm-snapshot -S -M pc-0.14 > -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name fb-0 > -u

Re: KVM lock contention on 48 core AMD machine

2011-03-18 Thread Joerg Roedel
Hi Ben, On Fri, Mar 18, 2011 at 07:02:40PM +0700, Ben Nagy wrote: > Here's some output from perf top while the system is locky: >263832.00 46.3% delay_tsc > [kernel.kallsyms] >231491.00 40.7% __ticket_spin_trylock > [kernel.kallsyms] > 14609.00 2.6% native_read