I am experiencing big issues with load after upgrading my Squeeze boxes to Wheezy. I have 7 app servers, all with identical hardware with identical packages and code. I upgraded one of my boxes to wheezy with packages for PHP, Python, etc... that were compiled on wheezy. My problem is that my Wheezy boxes have a load of over 3 and are not staying up during our peak time, whereas our squeeze boxes have a load of less than 1. I believe I have tracked down the issue with our load to be an interrupt issue. My interrupts on wheezy are 30% higher. CPU, I/O, Memory and Context Switches are all about the same. The one thing that stands out is that my "Rescheduling Interrupts" and "timer" are interrupting like crazy on wheezy compared to squeeze. Here is my output of total interrupts, both systems were rebooted and put into service at the same time: Thoughts? Suggestions? Here is my squeeze box interrupts: # sudo cat /proc/interrupts | awk '{ print $18, $19, $2+$3+$4+$5+$6+$7+$8+$9+$10+$11+$12+$13+$14+$15+$16+$17 }' 0 IO-APIC-edge timer 50 IO-APIC-edge i8042 2 IO-APIC-edge serial 8 IO-APIC-edge rtc0 1 IO-APIC-fasteoi acpi 0 IO-APIC-edge i8042 4 IO-APIC-fasteoi uhci_hcd:usb2 0 IO-APIC-fasteoi ehci_hcd:usb1, 2 IO-APIC-fasteoi ata_piix, 24014 IO-APIC-fasteoi uhci_hcd:usb4 48 IO-APIC-fasteoi ehci_hcd:usb3, 0 PCI-MSI-edge eth0 1 PCI-MSI-edge eth0-TxRx-0 919924 PCI-MSI-edge eth0-TxRx-1 1206377 PCI-MSI-edge eth0-TxRx-2 1208344 PCI-MSI-edge eth0-TxRx-3 817225 PCI-MSI-edge eth0-TxRx-4 1165734 PCI-MSI-edge eth0-TxRx-5 1314252 PCI-MSI-edge eth0-TxRx-6 998115 PCI-MSI-edge eth0-TxRx-7 1229384 PCI-MSI-edge eth1 1 PCI-MSI-edge eth1-TxRx-0 4776 PCI-MSI-edge eth1-TxRx-1 4444 PCI-MSI-edge eth1-TxRx-2 5557 PCI-MSI-edge eth1-TxRx-3 5344 PCI-MSI-edge eth1-TxRx-4 5827 PCI-MSI-edge eth1-TxRx-5 5060 PCI-MSI-edge eth1-TxRx-6 4078 PCI-MSI-edge eth1-TxRx-7 4317 PCI-MSI-edge ioat-msix 3 PCI-MSI-edge ioat-msix 3 PCI-MSI-edge ioat-msix 3 PCI-MSI-edge ioat-msix 3 PCI-MSI-edge ioat-msix 3 PCI-MSI-edge ioat-msix 3 PCI-MSI-edge ioat-msix 3 PCI-MSI-edge ioat-msix 3 Non-maskable interrupts 0 Local timer 3968846 Spurious interrupts 0 Performance monitoring 0 Performance pending 0 Rescheduling interrupts 67418 Function call 16404 TLB shootdowns 33915 Thermal event 0 Threshold APIC 0 Machine check 0 Machine check 128 Here is my wheezy interrupts: # sudo cat /proc/interrupts | awk '{ print $18, $19, $2+$3+$4+$5+$6+$7+$8+$9+$10+$11+$12+$13+$14+$15+$16+$17 }' IO-APIC-edge timer 504238 IO-APIC-edge i8042 3 IO-APIC-edge serial 12 IO-APIC-edge rtc0 1 IO-APIC-fasteoi acpi 0 IO-APIC-edge i8042 4 IO-APIC-fasteoi uhci_hcd:usb3 0 IO-APIC-fasteoi ehci_hcd:usb1, 2 IO-APIC-fasteoi ata_piix, 21189 IO-APIC-fasteoi uhci_hcd:usb4 47 IO-APIC-fasteoi ehci_hcd:usb2, 0 PCI-MSI-edge eth0 1 PCI-MSI-edge eth0-TxRx-0 1506134 PCI-MSI-edge eth0-TxRx-1 1102085 PCI-MSI-edge eth0-TxRx-2 1399087 PCI-MSI-edge eth0-TxRx-3 1123149 PCI-MSI-edge eth0-TxRx-4 849678 PCI-MSI-edge eth0-TxRx-5 1428705 PCI-MSI-edge eth0-TxRx-6 897420 PCI-MSI-edge eth0-TxRx-7 1321820 PCI-MSI-edge eth1 1 PCI-MSI-edge eth1-TxRx-0 4290 PCI-MSI-edge eth1-TxRx-1 4217 PCI-MSI-edge eth1-TxRx-2 3685 PCI-MSI-edge eth1-TxRx-3 4081 PCI-MSI-edge eth1-TxRx-4 5532 PCI-MSI-edge eth1-TxRx-5 6604 PCI-MSI-edge eth1-TxRx-6 3996 PCI-MSI-edge eth1-TxRx-7 4560 PCI-MSI-edge ioat-msix 3 PCI-MSI-edge ioat-msix 3 PCI-MSI-edge ioat-msix 3 PCI-MSI-edge ioat-msix 3 PCI-MSI-edge ioat-msix 3 PCI-MSI-edge ioat-msix 3 PCI-MSI-edge ioat-msix 3 PCI-MSI-edge ioat-msix 3 Non-maskable interrupts 3847 Local timer 3846061 Spurious interrupts 0 Performance monitoring 3847 IRQ work 0 Rescheduling interrupts 4109580 Function call 13442 TLB shootdowns 1745 Thermal event 0 Threshold APIC 0 Machine check 0 Machine check 128 |
