On Sun, Nov 25, 2012 at 9:57 PM, Andriy Gapon <a...@freebsd.org> wrote:
> on 24/11/2012 00:17 Alex Chistyakov said the following:
>> On Fri, Nov 23, 2012 at 9:06 PM, Andriy Gapon <a...@freebsd.org> wrote:
>>>
>>> I've cc-ed Alexander who is deeply familiar with both the scheduler and the
>>> timer
>>> code.
>>> I think that it would be nice to get ktr(4) information suitable for use
>>> with
>>> schedgraph (please google for these keywords).
>>
>> I collected two samples and put them here: http://1888.spb.ru/samples.zip
>> sched-cpu0.ktr is for a VM running on CPU #0 and sched-cpu1.ktr is for
>> a VM running on CPU #1
>> They seem to be very different.
>
> It looks like you didn't stop ktr tracing before running ktrdump or something
> like that. schedgraph can not grok the files because it believes that the
> timestamps are incorrect.
>
> # - While the workload is continuing (i.e. before it finishes), disable
> # KTR tracing by setting 'sysctl debug.ktr.mask=0'. This is necessary
> # to avoid a race condition while running ktrdump, i.e. the KTR ring buffer
> # will cycle a bit while ktrdump runs, and this confuses schedgraph because
> # the timestamps appear to go backwards at some point.
>>> Also, version of your kernel,
>>
>> kern.version: FreeBSD 9.1-PRERELEASE #4: Fri Nov 23 22:38:47 MSK 2012
>> Sources were grabbed on Nov, 16.
>>
>>> output of sysctls kern.eventtimer and kern.sched.
>>
>> kern.eventtimer.choice: LAPIC(600) HPET(550) HPET1(440) HPET2(440)
>> i8254(100) RTC(0)
>> kern.eventtimer.et.LAPIC.flags: 7
>> kern.eventtimer.et.LAPIC.frequency: 50002806
>> kern.eventtimer.et.LAPIC.quality: 600
>> kern.eventtimer.et.RTC.flags: 17
>> kern.eventtimer.et.RTC.frequency: 32768
>> kern.eventtimer.et.RTC.quality: 0
>> kern.eventtimer.et.i8254.flags: 1
>> kern.eventtimer.et.i8254.frequency: 1193182
>> kern.eventtimer.et.i8254.quality: 100
>> kern.eventtimer.et.HPET.flags: 7
>> kern.eventtimer.et.HPET.frequency: 14318180
>> kern.eventtimer.et.HPET.quality: 550
>> kern.eventtimer.et.HPET1.flags: 3
>> kern.eventtimer.et.HPET1.frequency: 14318180
>> kern.eventtimer.et.HPET1.quality: 440
>> kern.eventtimer.et.HPET2.flags: 3
>> kern.eventtimer.et.HPET2.frequency: 14318180
>> kern.eventtimer.et.HPET2.quality: 440
>> kern.eventtimer.periodic: 0
>> kern.eventtimer.timer: LAPIC
>> kern.eventtimer.activetick: 1
>> kern.eventtimer.idletick: 0
>> kern.eventtimer.singlemul: 2
>> kern.sched.cpusetsize: 8
>> kern.sched.preemption: 1
>> kern.sched.topology_spec: <groups>
>> kern.sched.steal_thresh: 2
>> kern.sched.steal_idle: 1
>> kern.sched.balance_interval: 127
>> kern.sched.balance: 1
>> kern.sched.affinity: 1
>> kern.sched.idlespinthresh: 16
>> kern.sched.idlespins: 10000
>> kern.sched.static_boost: 152
>> kern.sched.preempt_thresh: 80
>> kern.sched.interact: 30
>> kern.sched.slice: 12
>> kern.sched.quantum: 94488
>> kern.sched.name: ULE
>>
>> I tried kern.eventtimer.periodic=1 and
>> kern.timecounter.hardware=ACPI-fast but that did not help.
>
> Could you please also provide the CPU identification block from dmesg?
CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3200.18-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d
Stepping = 7
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x1fbee3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant, performance statistics
>
>>> BTW, do you use the default ULE scheduler?
>>
>> Yep.
>> I tried SCHED_4BSD and the situation became much better but not ideal.
>> %si was around 3-7% on the guest and I had to boot with noacpi and
>> disable the tickless kernel on the guest to lower it.
>> At least I was able to run a VM on CPU #0 and all cores became equal.
>
> Interesting results.
>
>>> Also, is your kernel DTrace enabled?
>
> Could you please run the following script for some seconds and report its
> output
> for both of the scenarios:
>
> profile:::profile-4001
> {
> @stacks[pid, tid, execname, stack()] = count();
> }
>
> END
> {
> trunc(@stacks, 40);
> printa(@stacks);
> }
A DTrace result for "bad" core is at https://gist.github.com/4145782
and a result for "good" core is at https://gist.github.com/4146946
Thank you!
--
SY,
Alex
_______________________________________________
freebsd-emulation@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-emulation
To unsubscribe, send any mail to "freebsd-emulation-unsubscr...@freebsd.org"
