Hi all,
We have a web server running apache (on OpenBSD 4.4) with php and mysql
from packages.
This morning it froze and we don't know why.
The only thing we know is that the load was quite high, see output from
top below.
Can't find anything in the logs, except maillog stating it is rejecting
connections because the load avg is to high. See excerpt below.
Checking the apache accesslog shows up to 30 requests/second at the time
of the freeze, which is quite normal for this system.
Since it's a production server we could not let it rest and hope it
would catch up, so it had to be power cycled.
Any clues on what went wrong and what we can do to prevent this?
Would the server catch up if we would let it rest?
Output from top, maillog, sysctl hw and dmesg further down.
Cheers,
Fredrik
Top from just before the system froze:
load averages: 18.04, 15.31, 8.63
51 processes: 13 running, 35 idle, 3 on processor
CPU0 states: 0.0% user, 0.0% nice, 30.5% system, 0.0% interrupt,
69.5% idle
CPU1 states: 0.0% user, 0.0% nice, 23.0% system, 0.0% interrupt,
77.0% idle
CPU2 states: 0.0% user, 0.0% nice, 30.7% system, 0.0% interrupt,
69.3% idle
CPU3 states: 0.0% user, 0.0% nice, 28.1% system, 0.0% interrupt,
71.9% idle
Memory: Real: 104M/483M act/tot Free: 1525M Swap: 0K/2052M used/tot
PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
15267 www -18 0 5548K 200K run/1 - 0:14 7.13% httpd
10125 root -18 0 700K 820K run/1 - 0:10 6.74% cron
12027 www -18 0 5648K 6112K run/1 - 0:33 6.49% httpd
17558 root -18 0 204K 124K run/0 - 0:04 6.45% login
15474 www -18 0 5572K 2424K run/1 - 0:14 6.30% httpd
28059 www -18 0 5572K 2428K run/0 - 0:13 6.15% httpd
7678 www -18 0 5560K 2328K onproc/1 - 0:13 6.15% httpd
186 root -18 0 700K 84K run/1 - 0:09 6.15% cron
31714 root -18 0 5548K 9056K run/0 - 0:33 6.10% httpd
18191 root -18 0 236K 92K run/1 - 0:03 6.10% login
25968 www -18 0 6272K 9684K run/0 - 6:42 6.05% httpd
5214 www -18 0 5560K 2248K run/1 - 0:14 6.01% httpd
22310 root -18 0 516K 952K onproc/3 - 0:11 5.91% sshd
12973 _mysql -18 0 116M 68M run/0 - 104:16 5.81% mysqld
25681 root -18 0 516K 1144K run/0 - 0:11 5.81% sshd
2527 www 2 0 6344K 9764K idle netio 7:14 0.00% httpd
13090 www 2 0 6276K 9624K idle poll 6:54 0.00% httpd
Excerpt from maillog from the time before the system froze
Apr 1 09:15:02 kitty sm-mta[22701]: rejecting connections on daemon
MTA: load average: 21
Apr 1 09:15:02 kitty sm-mta[22701]: rejecting connections on daemon
MTA6: load average: 21
Apr 1 09:15:02 kitty sm-mta[22701]: rejecting connections on daemon
MSA: load average: 21
Apr 1 09:15:02 kitty sm-mta[22701]: rejecting connections on daemon
MSA6: load average: 21
Apr 1 09:15:17 kitty sm-mta[22701]: rejecting connections on daemon
MTA: load average: 21
Apr 1 09:15:17 kitty sm-mta[22701]: rejecting connections on daemon
MTA6: load average: 21
Apr 1 09:15:17 kitty sm-mta[22701]: rejecting connections on daemon
MSA: load average: 21
Apr 1 09:15:17 kitty sm-mta[22701]: rejecting connections on daemon
MSA6: load average: 21
sysctl hw
hw.machine=i386
hw.model=Intel(R) Xeon(TM) CPU 3.20GHz ("GenuineIntel" 686-class)
hw.ncpu=4
hw.byteorder=1234
hw.pagesize=4096
hw.disknames=sd0,sd1,cd0,fd0
hw.diskcount=4
hw.sensors.acpitz0.temp0=8.35 degC (zone temperature)
hw.cpuspeed=3190
hw.setperf=100
hw.vendor=HP
hw.product=ProLiant DL380 G3
hw.serialno=800JLDND3P
hw.uuid=3830304a-4c44-4e44-3350-202020202020
hw.physmem=2147028992
hw.usermem=2147004416
dmesg:
OpenBSD 4.4 (GENERIC.MP) #844: Tue Aug 12 17:24:39 MDT 2008
dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC.MP
cpu0: Intel(R) Xeon(TM) CPU 3.20GHz ("GenuineIntel" 686-class) 3.19 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID,xTPR
real mem = 2147028992 (2047MB)
avail mem = 2067607552 (1971MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 12/31/99, BIOS32 rev. 0 @ 0xf0000,
SMBIOS rev. 2.3 @ 0xec000 (49 entries)
bios0: vendor HP version "P29" date 09/15/2004
bios0: HP ProLiant DL380 G3
acpi0 at bios0: rev 0
acpi0: tables DSDT FACP APIC SPCR
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 132MHz
cpu1 at mainbus0: apid 6 (application processor)
cpu1: Intel(R) Xeon(TM) CPU 3.20GHz ("GenuineIntel" 686-class) 3.19 GHz
cpu1:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID,xTPR
cpu2 at mainbus0: apid 1 (application processor)
cpu2: Intel(R) Xeon(TM) CPU 3.20GHz ("GenuineIntel" 686-class) 3.19 GHz
cpu2:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID,xTPR
cpu3 at mainbus0: apid 7 (application processor)
cpu3: Intel(R) Xeon(TM) CPU 3.20GHz ("GenuineIntel" 686-class) 3.19 GHz
cpu3:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID,xTPR
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 11, 16 pins
ioapic1 at mainbus0: apid 3 pa 0xfec01000, version 11, 16 pins
ioapic2 at mainbus0: apid 4 pa 0xfec02000, version 11, 16 pins
ioapic3 at mainbus0: apid 5 pa 0xfec03000, version 11, 16 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PCI1)
acpiprt2 at acpi0: bus 2 (PCI2)
acpiprt3 at acpi0: bus 3 (PCI3)
acpiprt4 at acpi0: bus 6 (PCI4)
acpicpu0 at acpi0
acpicpu1 at acpi0
acpicpu2 at acpi0
acpicpu3 at acpi0
acpitz0 at acpi0: critical temperature 31 degC
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x4000 0xcc000/0x1800
0xee000/0x2000!
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "ServerWorks CNB20-HE Host (GC-LE)" rev 0x33
pchb1 at pci0 dev 0 function 1 "ServerWorks CNB20-HE Host (GC-LE)" rev 0x00
pci1 at pchb1 bus 3
pchb2 at pci0 dev 0 function 2 "ServerWorks CNB20-HE Host (GC-LE)" rev 0x00
pci2 at pchb2 bus 1
ciss0 at pci2 dev 3 function 0 "Compaq Smart Array 5i/532 rev.2" rev
0x01: apic 3 int 14 (irq 10)
ciss0: 2 LDs, HW rev 1, FW 2.76/2.76
scsibus0 at ciss0: 2 targets, initiator 2
sd0 at scsibus0 targ 0 lun 0: <COMPAQ, LOGICAL VOLUME, 2.76> SCSI2
0/direct fixed
sd0: 34703MB, 4424 cyl, 255 head, 63 sec, 512 bytes/sec, 71073600 sec total
sd1 at scsibus0 targ 1 lun 0: <COMPAQ, LOGICAL VOLUME, 2.76> SCSI2
0/direct fixed
sd1: 69413MB, 8849 cyl, 255 head, 63 sec, 512 bytes/sec, 142159185 sec total
vga1 at pci0 dev 3 function 0 "ATI Rage XL" rev 0x27
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
drm at vga1 unsupported
"Compaq iLO" rev 0x01 at pci0 dev 4 function 0 not configured
"Compaq iLO" rev 0x01 at pci0 dev 4 function 2 not configured
piixpm0 at pci0 dev 15 function 0 "ServerWorks CSB5" rev 0x93: polling
iic0 at piixpm0
spdmem0 at iic0 addr 0x51: 512MB DDR SDRAM registered ECC PC2100CL2.5
spdmem1 at iic0 addr 0x52: 512MB DDR SDRAM registered ECC PC2100CL2.5
spdmem2 at iic0 addr 0x53: 256MB DDR SDRAM registered ECC PC2300CL2.5
spdmem3 at iic0 addr 0x54: 256MB DDR SDRAM registered ECC PC2300CL2.5
spdmem4 at iic0 addr 0x55: 256MB DDR SDRAM registered ECC PC2100CL2.5
spdmem5 at iic0 addr 0x56: 256MB DDR SDRAM registered ECC PC2100CL2.5
pciide0 at pci0 dev 15 function 1 "ServerWorks CSB5 IDE" rev 0x93: DMA
atapiscsi0 at pciide0 channel 0 drive 0
scsibus1 at atapiscsi0: 2 targets, initiator 7
cd0 at scsibus1 targ 0 lun 0: <TEAC, CD-224E, 9.9A> ATAPI 5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, DMA mode 2
ohci0 at pci0 dev 15 function 2 "ServerWorks OSB4/CSB5 USB" rev 0x05:
apic 2 int 7 (irq 7), version 1.0, legacy support
pchb3 at pci0 dev 15 function 3 "ServerWorks CSB5 LPC" rev 0x00
pchb4 at pci0 dev 16 function 0 "ServerWorks CIOB-X2 PCIX" rev 0x05
pchb5 at pci0 dev 16 function 2 "ServerWorks CIOB-X2 PCIX" rev 0x05
pci3 at pchb5 bus 6
"Compaq PCI Hotplug" rev 0x14 at pci3 dev 30 function 0 not configured
pchb6 at pci0 dev 17 function 0 "ServerWorks CIOB-X2 PCIX" rev 0x05
pchb7 at pci0 dev 17 function 2 "ServerWorks CIOB-X2 PCIX" rev 0x05
pci4 at pchb7 bus 2
bge0 at pci4 dev 1 function 0 "Broadcom BCM5703X" rev 0x02, BCM5703 A2
(0x1002): apic 3 int 13 (irq 11), address 00:11:85:ba:d9:d3
brgphy0 at bge0 phy 1: BCM5703 10/100/1000baseT PHY, rev. 2
bge1 at pci4 dev 2 function 0 "Broadcom BCM5703X" rev 0x02, BCM5703 A2
(0x1002): apic 3 int 15 (irq 15), address 00:11:85:ba:d9:d2
brgphy1 at bge1 phy 1: BCM5703 10/100/1000baseT PHY, rev. 2
usb0 at ohci0: USB revision 1.0
uhub0 at usb0 "ServerWorks OHCI root hub" rev 1.00/1.00 addr 1
isa0 at mainbus0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pmsi0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pmsi0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
mtrr: Pentium Pro MTRR support
softraid0 at root
root on sd0a swap on sd0b dump on sd0b
