Package: linux-image Version: 2.6.26-2-amd64 Severity: important Hardware: A machine with a pair of 2.5" SATA disks, configured with three RAID1 devices (/dev/md1, /dev/md2, /dev/md3), reports a soft lockup as shown below.
The system is running Debian Linux 2.6.26-2-amd64, libc6 2.7. It has a custom card (requiring the mfb and memmgmnt drivers as shown in the log). This card shouldn't affect the disks in any way. The system contains a hardware watchdog; and the long soft lockup caused the watchdog to expire, resetting the system. On reboot, we weren't able to detect any signs of trauma on the disks, so we're left with a diagnosis of a software lockup in md1_raid1 which took the box down due to a watchdog expiry (i.e. this problem is critical for the system in question). 2011-07-03T00:57:02-07:00 testhost kernel: [2500677.138796] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) 2011-07-03T00:57:02-07:00 testhost kernel: [2500677.142796] md: delaying data-check of md2 until md1 has finished (they share one or more physical units) 2011-07-03T00:57:02-07:00 testhost kernel: [2500677.142796] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] BUG: soft lockup - CPU#0 stuck for 235s! [md1_raid1:1038] <========= 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] Modules linked in: mfb_driver(P) memmgmnt_driver(P) acpi_cpufreq cpufreq_ondemand cpufreq_conservative cpufreq_stats freq_table cpufreq_powersave cpufreq_userspace ipv6 xt_limit xt_tcpudp xt_state ipt_LOG nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack iptable_filter ip_tables x_tables coretemp dme1737 lm85 hwmon_vid i2c_i801 serio_raw i2c_core container pcspkr psmouse snd_pcm snd_timer snd soundcore snd_page_alloc button joydev evdev ext3 jbd mbcache raid1 md_mod sd_mod usbhid hid ff_memless ahci libata scsi_mod dock e1000e ehci_hcd uhci_hcd thermal processor fan thermal_sys [last unloaded: mfb_driver] 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] CPU 0: 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] Modules linked in: mfb_driver(P) memmgmnt_driver(P) acpi_cpufreq cpufreq_ondemand cpufreq_conservative cpufreq_stats freq_table cpufreq_powersave cpufreq_userspace ipv6 xt_limit xt_tcpudp xt_state ipt_LOG nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack iptable_filter ip_tables x_tables coretemp dme1737 lm85 hwmon_vid i2c_i801 serio_raw i2c_core container pcspkr psmouse snd_pcm snd_timer snd soundcore snd_page_alloc button joydev evdev ext3 jbd mbcache raid1 md_mod sd_mod usbhid hid ff_memless ahci libata scsi_mod dock e1000e ehci_hcd uhci_hcd thermal processor fan thermal_sys [last unloaded: mfb_driver] 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] Pid: 1038, comm: md1_raid1 Tainted: P 2.6.26-2-amd64 #1 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] RIP: 0010:[<ffffffffa005bb7a>] [<ffffffffa005bb7a>] :scsi_mod:scsi_request_fn+0x2ad/0x395 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] RSP: 0018:ffff81020bcbfb18 EFLAGS: 00000202 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] RAX: ffff81020bc8c050 RBX: ffff81020c9f1800 RCX: ffff81020c9f1848 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] RDX: ffff81020c9f1848 RSI: ffff81020bc023f8 RDI: ffff81020bc8c050 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] RBP: ffff81020c9f1800 R08: 0000000000000000 R09: ffff8100379f4ac0 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] R10: ffff81020bc020d8 R11: ffff81020bc020b8 R12: 00001d4c0bc020b8 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] R13: ffff81020bc020b8 R14: ffff81020bc023f8 R15: ffff81020bc023f8 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] FS: 0000000000000000(0000) GS:ffffffff8053c000(0000) knlGS:0000000000000000 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] CR2: 0000000001a8bfc0 CR3: 000000020c9a8000 CR4: 00000000000006e0 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] Call Trace: 2011-07-03T01:17:22-07:00 testhost kernel: [2502418.212841] [<ffffffffa005bb7a>] ? :scsi_mod:scsi_request_fn+0x2ad/0x395 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffff8030b8d6>] ? elv_insert+0xf2/0x220 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffff8030e179>] ? __make_request+0x3af/0x3fb 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffff8030c9ef>] ? generic_make_request+0x2fe/0x339 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffff802bddf2>] ? bio_alloc_bioset+0x89/0xd9 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffff8030ddc3>] ? submit_bio+0xdb/0xe2 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffffa00e31d2>] ? :md_mod:write_page+0x199/0x2a7 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffffa00e406d>] ? :md_mod:bitmap_unplug+0xae/0x18b 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffff8023d2f7>] ? del_timer+0x56/0x5f 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffffa00efbc5>] ? :raid1:flush_pending_writes+0x56/0x8d 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffffa00f0119>] ? :raid1:raid1d+0x6d/0xc8f 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffff80428e9f>] ? thread_return+0x6b/0xac 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffff8023cf95>] ? lock_timer_base+0x26/0x4b 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffff8023d00b>] ? try_to_del_timer_sync+0x51/0x5a 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffff8023d020>] ? del_timer_sync+0xc/0x16 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffff8042916a>] ? schedule_timeout+0x92/0xad 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffff8023cc88>] ? process_timeout+0x0/0x5 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffff8042915d>] ? schedule_timeout+0x85/0xad 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffffa00e1bc1>] ? :md_mod:md_thread+0xd7/0xed 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffff80246221>] ? autoremove_wake_function+0x0/0x2e 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffffa00e1aea>] ? :md_mod:md_thread+0x0/0xed 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffff802460fb>] ? kthread+0x47/0x74 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffff802301e9>] ? schedule_tail+0x27/0x5c 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffff8020cf28>] ? child_rip+0xa/0x12 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffff802460b4>] ? kthread+0x0/0x74 2011-07-03T01:17:23-07:00 testhost kernel: [2502418.212841] [<ffffffff8020cf1e>] ? child_rip+0x0/0x12
