It's likely that kernel service threads (which show up as "sched") are
preempting postgresql...at which point the runnable postgresql goes back
on the run queue to wait, at which point another idle CPU steals it so
that it can run immediately. With this happening often things can tend
to migrate around a bit....
Increased kernel service thread activity does tend to accompany heavier
I/O load...
-Eric
Kugutsumen wrote:
> On this quad core box with 8 gig of ram and a 4 gig zfs ARC, I keep seeing
> tons of cross calls and cpu migration.
>
> cross calls go up to 4000 during read only operations and up to 30000 during
> read and write operations.
>
> The only thing running is postgresql and a single threaded import script
> written in python.
> The whole import is wrapped in a transaction and constraints are deferred. I
> am using a recordsize of 8k for data and logs and fsync is off.
>
> SunOS sunrise 5.11 snv_75 i86pc i386 i86pc
>
> [EMAIL PROTECTED] / # dtrace -n 'sysinfo:::xcalls
> {
> @execs[execname] = count();
> }'
> dtrace: description 'sysinfo:::xcalls
> ' matched 2 probes
> ^C
>
> dtrace 3219
> sched 429127
>
>
> CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
> 0 5 0 1 1424 214 1110 3 49 122 0 1350 2 2 0 96
> 1 2 0 1 1270 17 1029 2 42 126 0 1226 1 2 0 97
> 2 4 0 1467 496 147 754 2 29 65 0 1150 1 7 0 92
> 3 3 0 1 744 0 973 3 18 60 0 1331 3 2 0 95
> CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
> 0 1 0 0 520 217 415 0 29 34 0 318 1 1 0 98
> 1 8 0 0 253 3 514 0 33 37 0 371 0 1 0 99
> 2 0 0 0 326 78 555 6 25 28 0 990 0 2 0 98
> 3 0 0 0 35 0 48 0 0 0 0 77 0 0 0 100
> CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
> 0 0 0 0 506 211 342 0 30 30 0 294 1 1 0 98
> 1 1 0 0 278 5 550 0 25 38 0 387 2 1 0 97
> 2 0 0 61 224 81 337 1 33 28 0 432 0 3 0 97
> 3 0 0 0 180 0 282 1 3 0 0 657 0 1 0 99
> CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
> 0 0 0 0 677 222 755 1 46 50 0 654 1 2 0 97
> 1 3 0 0 310 4 521 0 37 47 1 310 0 1 0 99
> 2 0 0 76 424 111 691 8 48 36 0 1149 1 3 0 96
> 3 0 0 0 115 0 145 1 0 1 0 313 0 0 0 100
> CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
> 0 1 0 0 680 219 759 1 59 75 0 499 2 2 0 96
> 1 1 0 1 1326 6 701 0 39 53 0 485 2 5 0 93
> 2 0 0 977 527 136 858 8 40 37 0 1587 1 6 0 93
> 3 0 0 0 64 1 111 0 6 11 0 88 0 1 0 99
> CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
> 0 0 0 0 1039 219 1259 10 122 342 0 168 0 1 0 99
> 1 1 0 0 734 4 1191 4 115 320 0 245 1 1 0 98
> 2 0 0 418 738 490 496 8 40 123 0 33 0 17 0 83
> 3 0 0 0 468 0 787 3 31 201 0 513 1 1 0 98
> CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
> 0 2 0 0 497 209 419 0 27 25 0 358 1 1 0 98
> 1 0 0 0 235 6 459 0 26 37 0 316 0 1 0 99
> 2 0 0 0 155 75 205 0 33 25 0 75 0 2 0 98
> 3 0 0 0 248 0 484 1 0 0 0 1169 0 0 0 100
> ^C
>
>
> [EMAIL PROTECTED] / # dtrace -n ' sched:::off-cpu
>
>> {
>> self->cpu = cpu;
>> }
>>
>> sched:::on-cpu /self->cpu != cpu/
>> {
>> printf("%s migrated from cpu %d to cpu %d\n",execname,self->cpu,cpu);
>> self->cpu = 0;
>> }'
>>
> dtrace: description ' sched:::off-cpu
> ' matched 6 probes
> CPU ID FUNCTION:NAME
> 0 13414 resume:on-cpu postgres migrated from cpu 1 to
> cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 2 to cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 3 to cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 2 to
> cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 2 to
> cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 1 to
> cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 2 to cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 1 to cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 1 to
> cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 2 to cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 1 to cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 2 to cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 1 to
> cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 2 to cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 2 to
> cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 1 to
> cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 2 to
> cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 2 to
> cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 2 to
> cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 1 to cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 2 to
> cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 2 to
> cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 2 to cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 2 to
> cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 2 to
> cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 2 to
> cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 2 to
> cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 1 to cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 1 to cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 1 to
> cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 1 to cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 2 to
> cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 1 to
> cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 1 to
> cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 1 to cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 1 to cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 2 to
> cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 2 to cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 2 to cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 1 to cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 2 to
> cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 1 to
> cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 2 to
> cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 2 to cpu 0
>
> 0 13414 resume:on-cpu sched migrated from cpu 3 to cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 2 to
> cpu 0
>
> 0 13414 resume:on-cpu postgres migrated from cpu 2 to
> cpu 0
>
> 1 13414 resume:on-cpu sched migrated from cpu 0 to cpu 1
>
> 1 13414 resume:on-cpu postgres migrated from cpu 0 to
> cpu 1
>
> 1 13414 resume:on-cpu sched migrated from cpu 3 to cpu 1
>
> 1 13414 resume:on-cpu sched migrated from cpu 0 to cpu 1
>
> 1 13414 resume:on-cpu sched migrated from cpu 2 to cpu 1
>
> 1 13414 resume:on-cpu postgres migrated from cpu 0 to
> cpu 1
>
> 1 13414 resume:on-cpu sched migrated from cpu 3 to cpu 1
>
> 1 13414 resume:on-cpu postgres migrated from cpu 0 to
> cpu 1
>
> 1 13414 resume:on-cpu sched migrated from cpu 2 to cpu 1
>
> 1 13414 resume:on-cpu sched migrated from cpu 0 to cpu 1
>
> 1 13414 resume:on-cpu sched migrated from cpu 0 to cpu 1
>
> 1 13414 resume:on-cpu postgres migrated from cpu 3 to
> cpu 1
>
> 1 13414 resume:on-cpu sched migrated from cpu 0 to cpu 1
>
> 1 13414 resume:on-cpu postgres migrated from cpu 0 to
> cpu 1
>
> 1 13414 resume:on-cpu sched migrated from cpu 0 to cpu 1
>
> 1 13414 resume:on-cpu postgres migrated from cpu 0 to
> cpu 1
>
> 1 13414 resume:on-cpu sched migrated from cpu 2 to cpu 1
>
> 1 13414 resume:on-cpu sched migrated from cpu 3 to cpu 1
>
> 1 13414 resume:on-cpu postgres migrated from cpu 0 to
> cpu 1
>
> 1 13414 resume:on-cpu sched migrated from cpu 0 to cpu 1
>
> 1 13414 resume:on-cpu postgres migrated from cpu 0 to
> cpu 1
>
> 1 13414 resume:on-cpu sched migrated from cpu 0 to cpu 1
>
> 1 13414 resume:on-cpu postgres migrated from cpu 0 to
> cpu 1
>
> 1 13414 resume:on-cpu sched migrated from cpu 0 to cpu 1
>
> 1 13414 resume:on-cpu postgres migrated from cpu 2 to
> cpu 1
>
> 1 13414 resume:on-cpu sched migrated from cpu 2 to cpu 1
>
> 1 13414 resume:on-cpu postgres migrated from cpu 0 to
> cpu 1
>
> 1 13414 resume:on-cpu sched migrated from cpu 0 to cpu 1
>
> 1 13414 resume:on-cpu postgres migrated from cpu 0 to
> cpu 1
>
> 1 13414 resume:on-cpu postgres migrated from cpu 0 to
> cpu 1
>
> 1 13414 resume:on-cpu postgres migrated from cpu 0 to
> cpu 1
>
> 1 13414 resume:on-cpu sched migrated from cpu 2 to cpu 1
>
>
> This message posted from opensolaris.org
> _______________________________________________
> perf-discuss mailing list
> perf-discuss@opensolaris.org
>
_______________________________________________
perf-discuss mailing list
perf-discuss@opensolaris.org
