I'm having some trouble figuring out what is causing a systemic performance
issue. By "systemic" I mean that running even seemingly trivial things
(i.e. 'ls' on a directory with only a few files in it) is accompanied by a
substantial delay before any response, say, of 15-30 seconds. Not *every*
single time, but pretty frequently. Looking at top, load average is higher
than expected on a largely idle web & smtp server, usually between .7 and
1.6. This is a dual-core opteron box (Sun Fire X2100) with 8GB RAM. (dmesg
below).
Sorting top by CPU utilization shows that the CPUs are mostly idle.
vmstat/iostat isn't making anything leap out at me as getting stuck in
paging or waiting for I/O. The disc is a softraid RAID1 volume, bioctl
shows that it's OK. This does not appear to be network bandwidth-related,
at least, I can get the expected throughput testing with iperf.
Posting something on the forum software that it's running can take 3-4
minutes to complete, and it's writing the posted info to a mysql database.
Neither httpd nor mysqld seem to wind up much load while this is occurring.
So, what should I be looking at to figure out where things are getting
stalled up?
Thanks for any ideas,
-John
dmesg:
OpenBSD 5.3 (GENERIC.MP) #0: Fri Sep 13 04:11:52 EDT 2013
j...@hytronix-gw1.hytronix.com:/usr/src/sys/arch/amd64/compile/
GENERIC.MP
real mem = 8588820480 (8190MB)
avail mem = 8337694720 (7951MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xfc0f0 (36 entries)
bios0: vendor Sun Microsystems version "S40_3A12" date 12/22/2006
bios0: Sun Microsystems Sun Fire X2100 M2
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP APIC SPCR OEMB SSDT
acpi0: wakeup devices PS2K(S4) PS2M(S4) NSMB(S4) USB0(S1) USB2(S1) NMAC(S5)
NMAD(S5) P0P1(S4) HDAC(S4) BR10(S5) BR11(S5) BR12(S5) BR1E(S5) BCM1(S5)
BR13(S5) BR14(S5) BR15(S5) SLPB(S4) PWRB(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Dual-Core AMD Opteron(tm) Processor 1210, 1809.52 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB
64b/line 16-way L2 cache
cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu0: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu0: apic clock running at 201MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Dual-Core AMD Opteron(tm) Processor 1210, 1809.28 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8
cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB
64b/line 16-way L2 cache
cpu1: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu1: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 11, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (P0P1)
acpiprt2 at acpi0: bus 8 (BR10)
acpiprt3 at acpi0: bus 7 (BR11)
acpiprt4 at acpi0: bus 5 (BR12)
acpiprt5 at acpi0: bus 6 (BR1E)
acpiprt6 at acpi0: bus 4 (BR13)
acpiprt7 at acpi0: bus 3 (BR14)
acpiprt8 at acpi0: bus 2 (BR15)
acpicpu0 at acpi0: PSS
acpicpu1 at acpi0: PSS
acpibtn0 at acpi0: SLPB
acpibtn1 at acpi0: PWRB
ipmi at mainbus0 not configured
cpu0: PowerNow! K8 1809 MHz: speeds: 1800 1000 MHz
pci0 at mainbus0 bus 0
"NVIDIA MCP55 Memory" rev 0xa2 at pci0 dev 0 function 0 not configured
pcib0 at pci0 dev 1 function 0 "NVIDIA MCP55 ISA" rev 0xa3
nviic0 at pci0 dev 1 function 1 "NVIDIA MCP55 SMBus" rev 0xa3
iic0 at nviic0
spdmem0 at iic0 addr 0x50: 2GB DDR2 SDRAM non-parity PC2-5300CL5
spdmem1 at iic0 addr 0x51: 2GB DDR2 SDRAM non-parity PC2-5300CL5
spdmem2 at iic0 addr 0x52: 2GB DDR2 SDRAM non-parity PC2-5300CL5
spdmem3 at iic0 addr 0x53: 2GB DDR2 SDRAM non-parity PC2-5300CL5
iic1 at nviic0
ohci0 at pci0 dev 2 function 0 "NVIDIA MCP55 USB" rev 0xa1: apic 2 int 15,
version 1.0, legacy support
ehci0 at pci0 dev 2 function 1 "NVIDIA MCP55 USB" rev 0xa2: apic 2 int 7
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "NVIDIA EHCI root hub" rev 2.00/1.00 addr 1
pciide0 at pci0 dev 4 function 0 "NVIDIA MCP55 IDE" rev 0xa1: DMA, channel
0 configured to compatibility, channel 1 configured to compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <Optiarc, DVD RW AD-7530A, EX30> ATAPI
5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 ignored (disabled)
pciide1 at pci0 dev 5 function 0 "NVIDIA MCP55 SATA" rev 0xa3: DMA
pciide1: using apic 2 int 10 for native-PCI interrupt
wd0 at pciide1 channel 0 drive 0: <WDC WD10EADS-00M2B0>
wd0: 16-sector PIO, LBA48, 953869MB, 1953525168 sectors
wd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 6
wd1 at pciide1 channel 1 drive 0: <WDC WD10EADS-98M2B0>
wd1: 16-sector PIO, LBA48, 953869MB, 1953525168 sectors
wd1(pciide1:1:0): using PIO mode 4, Ultra-DMA mode 6
ppb0 at pci0 dev 6 function 0 "NVIDIA MCP55 PCI-PCI" rev 0xa2
pci1 at ppb0 bus 1
vga1 at pci1 dev 5 function 0 "ASPEED Technology AST2000" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
nfe0 at pci0 dev 8 function 0 "NVIDIA MCP55 LAN" rev 0xa3: apic 2 int 11,
address 00:1b:24:5c:48:13
eephy0 at nfe0 phy 2: 88E1149 Gigabit PHY, rev. 3
nfe1 at pci0 dev 9 function 0 "NVIDIA MCP55 LAN" rev 0xa3: apic 2 int 5,
address 00:1b:24:5c:48:14
eephy1 at nfe1 phy 3: 88E1149 Gigabit PHY, rev. 3
ppb1 at pci0 dev 10 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci2 at ppb1 bus 2
ppb2 at pci0 dev 11 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci3 at ppb2 bus 3
ppb3 at pci0 dev 12 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci4 at ppb3 bus 4
ppb4 at pci0 dev 13 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci5 at ppb4 bus 5
ppb5 at pci5 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xb5
pci6 at ppb5 bus 6
bge0 at pci6 dev 4 function 0 "Broadcom BCM5715" rev 0xa3, BCM5715 A3
(0x9003): apic 2 int 15, address 00:1b:24:5c:48:11
brgphy0 at bge0 phy 1: BCM5714 10/100/1000baseT/SX PHY, rev. 0
bge1 at pci6 dev 4 function 1 "Broadcom BCM5715" rev 0xa3, BCM5715 A3
(0x9003): apic 2 int 10, address 00:1b:24:5c:48:12
brgphy1 at bge1 phy 1: BCM5714 10/100/1000baseT/SX PHY, rev. 0
ppb6 at pci0 dev 14 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci7 at ppb6 bus 7
ppb7 at pci0 dev 15 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci8 at ppb7 bus 8
pchb0 at pci0 dev 24 function 0 "AMD AMD64 0Fh HyperTransport" rev 0x00
pchb1 at pci0 dev 24 function 1 "AMD AMD64 0Fh Address Map" rev 0x00
pchb2 at pci0 dev 24 function 2 "AMD AMD64 0Fh DRAM Cfg" rev 0x00
kate0 at pci0 dev 24 function 3 "AMD AMD64 0Fh Misc Cfg" rev 0x00: core rev
JH-F3
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
wbsio0 at isa0 port 0x2e/2: W83627EHF rev 0x68
wbsio0 port 0xa10/2 not configured
usb1 at ohci0: USB revision 1.0
uhub1 at usb1 "NVIDIA OHCI root hub" rev 1.00/1.00 addr 1
mtrr: Pentium Pro MTRR support
uhidev0 at uhub1 port 1 configuration 1 interface 0 "Virtual USB MULTIMEDIA
KEYBOARD" rev 1.10/0.01 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes, country code 33
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub1 port 1 configuration 1 interface 1 "Virtual USB MULTIMEDIA
KEYBOARD" rev 1.10/0.01 addr 2
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
sd0 at scsibus2 targ 1 lun 0: <OPENBSD, SR RAID 1, 005> SCSI2 0/direct fixed
sd0: 953866MB, 512 bytes/sector, 1953519473 sectors
root on sd0a (acce12d4fecd4052.a) swap on sd0b dump on sd0b
wskbd1: disconnecting from wsdisplay0
wskbd1 detached
ukbd0 detached
uhidev0 detached
wsmouse0 detached
ums0 detached
uhidev1 detached
syncing disks...
OpenBSD 5.3 (GENERIC.MP) #0: Fri Sep 13 04:11:52 EDT 2013
j...@hytronix-gw1.hytronix.com:/usr/src/sys/arch/amd64/compile/
GENERIC.MP
real mem = 8588820480 (8190MB)
avail mem = 8337694720 (7951MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xfc0f0 (36 entries)
bios0: vendor Sun Microsystems version "S40_3A12" date 12/22/2006
bios0: Sun Microsystems Sun Fire X2100 M2
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP APIC SPCR OEMB SSDT
acpi0: wakeup devices PS2K(S4) PS2M(S4) NSMB(S4) USB0(S1) USB2(S1) NMAC(S5)
NMAD(S5) P0P1(S4) HDAC(S4) BR10(S5) BR11(S5) BR12(S5) BR1E(S5) BCM1(S5)
BR13(S5) BR14(S5) BR15(S5) SLPB(S4) PWRB(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Dual-Core AMD Opteron(tm) Processor 1210, 1809.52 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB
64b/line 16-way L2 cache
cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu0: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu0: apic clock running at 201MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Dual-Core AMD Opteron(tm) Processor 1210, 1809.28 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8
cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB
64b/line 16-way L2 cache
cpu1: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu1: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 11, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (P0P1)
acpiprt2 at acpi0: bus 8 (BR10)
acpiprt3 at acpi0: bus 7 (BR11)
acpiprt4 at acpi0: bus 5 (BR12)
acpiprt5 at acpi0: bus 6 (BR1E)
acpiprt6 at acpi0: bus 4 (BR13)
acpiprt7 at acpi0: bus 3 (BR14)
acpiprt8 at acpi0: bus 2 (BR15)
acpicpu0 at acpi0: PSS
acpicpu1 at acpi0: PSS
acpibtn0 at acpi0: SLPB
acpibtn1 at acpi0: PWRB
ipmi at mainbus0 not configured
cpu0: PowerNow! K8 1809 MHz: speeds: 1800 1000 MHz
pci0 at mainbus0 bus 0
"NVIDIA MCP55 Memory" rev 0xa2 at pci0 dev 0 function 0 not configured
pcib0 at pci0 dev 1 function 0 "NVIDIA MCP55 ISA" rev 0xa3
nviic0 at pci0 dev 1 function 1 "NVIDIA MCP55 SMBus" rev 0xa3
iic0 at nviic0
spdmem0 at iic0 addr 0x50: 2GB DDR2 SDRAM non-parity PC2-5300CL5
spdmem1 at iic0 addr 0x51: 2GB DDR2 SDRAM non-parity PC2-5300CL5
spdmem2 at iic0 addr 0x52: 2GB DDR2 SDRAM non-parity PC2-5300CL5
spdmem3 at iic0 addr 0x53: 2GB DDR2 SDRAM non-parity PC2-5300CL5
iic1 at nviic0
ohci0 at pci0 dev 2 function 0 "NVIDIA MCP55 USB" rev 0xa1: apic 2 int 15,
version 1.0, legacy support
ehci0 at pci0 dev 2 function 1 "NVIDIA MCP55 USB" rev 0xa2: apic 2 int 7
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "NVIDIA EHCI root hub" rev 2.00/1.00 addr 1
pciide0 at pci0 dev 4 function 0 "NVIDIA MCP55 IDE" rev 0xa1: DMA, channel
0 configured to compatibility, channel 1 configured to compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <Optiarc, DVD RW AD-7530A, EX30> ATAPI
5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 ignored (disabled)
pciide1 at pci0 dev 5 function 0 "NVIDIA MCP55 SATA" rev 0xa3: DMA
pciide1: using apic 2 int 10 for native-PCI interrupt
wd0 at pciide1 channel 0 drive 0: <WDC WD10EADS-00M2B0>
wd0: 16-sector PIO, LBA48, 953869MB, 1953525168 sectors
wd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 6
wd1 at pciide1 channel 1 drive 0: <WDC WD10EADS-98M2B0>
wd1: 16-sector PIO, LBA48, 953869MB, 1953525168 sectors
wd1(pciide1:1:0): using PIO mode 4, Ultra-DMA mode 6
ppb0 at pci0 dev 6 function 0 "NVIDIA MCP55 PCI-PCI" rev 0xa2
pci1 at ppb0 bus 1
vga1 at pci1 dev 5 function 0 "ASPEED Technology AST2000" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
nfe0 at pci0 dev 8 function 0 "NVIDIA MCP55 LAN" rev 0xa3: apic 2 int 11,
address 00:1b:24:5c:48:13
eephy0 at nfe0 phy 2: 88E1149 Gigabit PHY, rev. 3
nfe1 at pci0 dev 9 function 0 "NVIDIA MCP55 LAN" rev 0xa3: apic 2 int 5,
address 00:1b:24:5c:48:14
eephy1 at nfe1 phy 3: 88E1149 Gigabit PHY, rev. 3
ppb1 at pci0 dev 10 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci2 at ppb1 bus 2
ppb2 at pci0 dev 11 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci3 at ppb2 bus 3
ppb3 at pci0 dev 12 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci4 at ppb3 bus 4
ppb4 at pci0 dev 13 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci5 at ppb4 bus 5
ppb5 at pci5 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xb5
pci6 at ppb5 bus 6
bge0 at pci6 dev 4 function 0 "Broadcom BCM5715" rev 0xa3, BCM5715 A3
(0x9003): apic 2 int 15, address 00:1b:24:5c:48:11
brgphy0 at bge0 phy 1: BCM5714 10/100/1000baseT/SX PHY, rev. 0
bge1 at pci6 dev 4 function 1 "Broadcom BCM5715" rev 0xa3, BCM5715 A3
(0x9003): apic 2 int 10, address 00:1b:24:5c:48:12
brgphy1 at bge1 phy 1: BCM5714 10/100/1000baseT/SX PHY, rev. 0
ppb6 at pci0 dev 14 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci7 at ppb6 bus 7
ppb7 at pci0 dev 15 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci8 at ppb7 bus 8
pchb0 at pci0 dev 24 function 0 "AMD AMD64 0Fh HyperTransport" rev 0x00
pchb1 at pci0 dev 24 function 1 "AMD AMD64 0Fh Address Map" rev 0x00
pchb2 at pci0 dev 24 function 2 "AMD AMD64 0Fh DRAM Cfg" rev 0x00
kate0 at pci0 dev 24 function 3 "AMD AMD64 0Fh Misc Cfg" rev 0x00: core rev
JH-F3
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
wbsio0 at isa0 port 0x2e/2: W83627EHF rev 0x68
wbsio0 port 0xa10/2 not configured
usb1 at ohci0: USB revision 1.0
uhub1 at usb1 "NVIDIA OHCI root hub" rev 1.00/1.00 addr 1
mtrr: Pentium Pro MTRR support
uhidev0 at uhub1 port 1 configuration 1 interface 0 "Virtual USB MULTIMEDIA
KEYBOARD" rev 1.10/0.01 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes, country code 33
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub1 port 1 configuration 1 interface 1 "Virtual USB MULTIMEDIA
KEYBOARD" rev 1.10/0.01 addr 2
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
sd0 at scsibus2 targ 1 lun 0: <OPENBSD, SR RAID 1, 005> SCSI2 0/direct fixed
sd0: 953866MB, 512 bytes/sector, 1953519473 sectors
root on sd0a (acce12d4fecd4052.a) swap on sd0b dump on sd0b
