Package: linux-2.6 Version: 2.6.32-30 Severity: important I'm getting a repeatable BUG from ext4, which seems to be caused by Postfix processing its mail queue. The specific filesystem block device that has the problem seems to be "dm-13", which on this boot is the logical volume containing the "/var/spool/postfix" chroot.
This is a completely standard Debian installation running on an Amazon EC2 instance (x86_64). The filesystem is mounted in "data=journal" mode. This crash is *very* repeatable. It occurs almost every reboot when there are more than 1 or 2 queued emails. I will try re-mounting the filesystem in "data=ordered" mode momentarily. The relevant filesystems are: /dev/mapper/system-root / ext4 rw,noatime,barrier=1,data=ordered 0 0 /dev/mapper/system-var /var ext4 rw,noatime,barrier=1,nodelalloc,data=journal 0 0 /dev/mapper/system-log /var/log ext4 rw,nosuid,nodev,noatime,barrier=1,data=ordered 0 0 /dev/xvda1 /boot ext3 rw,noatime,user_xattr,acl,data=journal 0 0 /dev/mapper/db-mail /var/mail ext4 rw,nosuid,nodev,noatime,barrier=1,data=ordered 0 0 /dev/mapper/db-postfix /var/spool/postfix ext4 rw,nosuid,nodev,noatime,barrier=1,nodelalloc,data=journal 0 0 /dev/mapper/db-smtp /var/lib/postfix ext4 rw,nosuid,nodev,noatime,barrier=1,nodelalloc,data=journal 0 0 /dev/mapper/db-smtpcfg /etc/postfix ext4 rw,nosuid,nodev,noatime,barrier=1,nodelalloc,data=journal 0 0 In particular, I note that there was a previous report of a BUG at fs/jbd2/commit.c:533 which never seemed to get isolated: http://www.kerneltrap.com/mailarchive/linux-ext4/2009/9/2/6373283 I need to get this system operational again right now, but I'm going to take a consistent snapshot of it so I can debug it later. NOTE: For followers on the linux-ext4 mailing list, this particular system is running the Debian "squeeze" kernel (based on 2.6.32), so it's theoretically possible this bug has been fixed upstream since then. I didn't have any luck finding such a fix on Google, though. -- Package-specific info: ** Version: Linux version 2.6.32-5-xen-amd64 (Debian 2.6.32-30) (b...@decadent.org.uk) (gcc version 4.3.5 (Debian 4.3.5-4) ) #1 SMP Wed Jan 12 05:46:49 UTC 2011 ** Command line: root=/dev/mapper/system-root ro ** Tainted: D (128) * Kernel has oopsed before. ** Kernel log: [ 118.525038] alloc irq_desc for 526 on node -1 [ 118.525040] alloc kstat_irqs on node -1 [ 118.700415] device-mapper: uevent: version 1.0.3 [ 118.700890] device-mapper: ioctl: 4.15.0-ioctl (2009-04-01) initialised: dm-de...@redhat.com [ 118.925563] EXT4-fs (dm-0): INFO: recovery required on readonly filesystem [ 118.925580] EXT4-fs (dm-0): write access will be enabled during recovery [ 118.968700] EXT4-fs (dm-0): orphan cleanup on readonly fs [ 118.968716] EXT4-fs (dm-0): ext4_orphan_cleanup: deleting unreferenced inode 790044 [ 118.968761] EXT4-fs (dm-0): ext4_orphan_cleanup: deleting unreferenced inode 790012 [ 118.968768] EXT4-fs (dm-0): ext4_orphan_cleanup: deleting unreferenced inode 790011 [ 118.968775] EXT4-fs (dm-0): ext4_orphan_cleanup: deleting unreferenced inode 790010 [ 118.968782] EXT4-fs (dm-0): ext4_orphan_cleanup: deleting unreferenced inode 790009 [ 118.968788] EXT4-fs (dm-0): 5 orphan inodes deleted [ 118.968794] EXT4-fs (dm-0): recovery complete [ 118.979150] EXT4-fs (dm-0): mounted filesystem with ordered data mode [ 119.293543] udev[204]: starting version 164 [ 119.366778] input: PC Speaker as /devices/platform/pcspkr/input/input1 [ 119.436417] Error: Driver 'pcspkr' is already registered, aborting... [ 124.153241] Adding 4194296k swap on /dev/xvdb1. Priority:-1 extents:1 across:4194296k SS [ 125.156599] loop: module loaded [ 138.650657] EXT4-fs (dm-21): Ignoring delalloc option - requested data journaling mode [ 138.650959] EXT4-fs (dm-21): mounted filesystem with journalled data mode [ 138.660092] EXT4-fs (dm-22): mounted filesystem with ordered data mode [ 138.674436] kjournald starting. Commit interval 5 seconds [ 138.675145] EXT3 FS on xvda1, internal journal [ 138.675155] EXT3-fs: mounted filesystem with journal data mode. [ 138.728462] EXT4-fs (xvdc): mounted filesystem without journal [ 138.745406] EXT4-fs (dm-17): mounted filesystem with ordered data mode [ 138.748531] EXT4-fs (dm-18): mounted filesystem with ordered data mode [ 138.774667] EXT4-fs (dm-19): mounted filesystem with ordered data mode [ 138.780834] EXT4-fs (dm-2): Ignoring delalloc option - requested data journaling mode [ 138.781400] EXT4-fs (dm-2): mounted filesystem with journalled data mode [ 138.784700] EXT4-fs (dm-1): Ignoring delalloc option - requested data journaling mode [ 138.784773] EXT4-fs (dm-1): mounted filesystem with journalled data mode [ 138.790320] EXT4-fs (dm-4): Ignoring delalloc option - requested data journaling mode [ 138.790871] EXT4-fs (dm-4): mounted filesystem with journalled data mode [ 138.794955] EXT4-fs (dm-3): Ignoring delalloc option - requested data journaling mode [ 138.795064] EXT4-fs (dm-3): mounted filesystem with journalled data mode [ 138.800441] EXT4-fs (dm-14): Ignoring delalloc option - requested data journaling mode [ 138.801015] EXT4-fs (dm-14): mounted filesystem with journalled data mode [ 138.804188] EXT4-fs (dm-13): Ignoring delalloc option - requested data journaling mode [ 138.804263] EXT4-fs (dm-13): mounted filesystem with journalled data mode [ 138.807812] EXT4-fs (dm-12): Ignoring delalloc option - requested data journaling mode [ 138.807890] EXT4-fs (dm-12): mounted filesystem with journalled data mode [ 138.814241] EXT4-fs (dm-16): mounted filesystem with ordered data mode [ 138.817864] EXT4-fs (dm-15): mounted filesystem with ordered data mode [ 138.821251] EXT4-fs (dm-9): mounted filesystem with ordered data mode [ 138.824801] EXT4-fs (dm-7): Ignoring delalloc option - requested data journaling mode [ 138.824877] EXT4-fs (dm-7): mounted filesystem with journalled data mode [ 138.828423] EXT4-fs (dm-6): Ignoring delalloc option - requested data journaling mode [ 138.828495] EXT4-fs (dm-6): mounted filesystem with journalled data mode [ 138.832295] EXT4-fs (dm-5): Ignoring delalloc option - requested data journaling mode [ 138.832868] EXT4-fs (dm-5): mounted filesystem with journalled data mode [ 138.836422] EXT4-fs (dm-8): Ignoring delalloc option - requested data journaling mode [ 138.836495] EXT4-fs (dm-8): mounted filesystem with journalled data mode [ 138.840309] EXT4-fs (dm-11): mounted filesystem with ordered data mode [ 138.844664] EXT4-fs (dm-10): Ignoring delalloc option - requested data journaling mode [ 138.844735] EXT4-fs (dm-10): mounted filesystem with journalled data mode [ 138.859715] EXT4-fs (dm-20): mounted filesystem with ordered data mode [ 150.205516] eth0: no IPv6 routers present [ 157.443694] ------------[ cut here ]------------ [ 157.443708] kernel BUG at /build/buildd-linux-2.6_2.6.32-30-amd64-d4MbNM/linux-2.6-2.6.32/debian/build/source_amd64_xen/fs/jbd2/commit.c:534! [ 157.443717] invalid opcode: 0000 [#1] SMP [ 157.443722] last sysfs file: /sys/devices/virtual/vc/vcsa6/uevent [ 157.443727] CPU 0 [ 157.443730] Modules linked in: ext3 jbd loop snd_pcm snd_timer snd soundcore evdev snd_page_alloc pcspkr ext4 mbcache jbd2 crc16 dm_mod xen_netfront xen_blkfront [ 157.443759] Pid: 748, comm: jbd2/dm-13-8 Not tainted 2.6.32-5-xen-amd64 #1 [ 157.443764] RIP: e030:[<ffffffffa00213bc>] [<ffffffffa00213bc>] jbd2_journal_commit_transaction+0x58b/0x115b [jbd2] [ 157.443778] RSP: e02b:ffff8801dc2adcf0 EFLAGS: 00010202 [ 157.443782] RAX: 0000000000000009 RBX: ffff8801d7c26870 RCX: 00000000ffff74c8 [ 157.443788] RDX: 00000000ffff74c8 RSI: ffff8801d7c26800 RDI: ffff8801d87a9024 [ 157.443793] RBP: ffff8801d7c26850 R08: ffff8801d87a9058 R09: ffff88000b1df780 [ 157.443798] R10: ffff8801d7aa0048 R11: ffff8801dba1d880 R12: ffff8801d87a9000 [ 157.443803] R13: ffff8801dc2add88 R14: ffff8801d7c26800 R15: 0000000000000000 [ 157.443812] FS: 00007f3afa7d3700(0000) GS:ffff88000b1ca000(0000) knlGS:0000000000000000 [ 157.443818] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b [ 157.443822] CR2: 00007f3afc878268 CR3: 00000001d7b05000 CR4: 0000000000002660 [ 157.443828] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 157.443833] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000 [ 157.443839] Process jbd2/dm-13-8 (pid: 748, threadinfo ffff8801dc2ac000, task ffff8801db02e2e0) [ 157.443844] Stack: [ 157.443847] ffff8801db02e318 ffff8801dc2add70 ffff8801db02e2e0 ffff8801d7aa38c8 [ 157.443854] <0> 00000024a85fc12e ffff8801d87a9024 ffff8801d87a939c ffff88000b1df780 [ 157.443865] <0> ffff8801d72f3000 ffff8801db02e2e0 0000000800000000 00000441000116e0 [ 157.443877] Call Trace: [ 157.443886] [<ffffffff8130b79f>] ? thread_return+0x79/0xe0 [ 157.443893] [<ffffffff8100e609>] ? __raw_callee_save_xen_irq_disable+0x11/0x1e [ 157.443901] [<ffffffff8130cb3a>] ? _spin_unlock_irqrestore+0xd/0xe [ 157.443908] [<ffffffff8105b748>] ? try_to_del_timer_sync+0x63/0x6c [ 157.443917] [<ffffffffa002788f>] ? kjournald2+0xbe/0x206 [jbd2] [ 157.443923] [<ffffffff81065d4a>] ? autoremove_wake_function+0x0/0x2e [ 157.443930] [<ffffffffa00277d1>] ? kjournald2+0x0/0x206 [jbd2] [ 157.443936] [<ffffffff81065a7d>] ? kthread+0x79/0x81 [ 157.443942] [<ffffffff81012baa>] ? child_rip+0xa/0x20 [ 157.443947] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6 [ 157.443952] [<ffffffff81012ba0>] ? child_rip+0x0/0x20 [ 157.443956] Code: 00 48 89 4c 24 38 45 31 ed c7 44 24 4c 00 00 00 00 c7 44 24 50 00 00 00 00 c7 44 24 58 ff ff ff ff 48 89 44 24 18 e9 f0 02 00 00 <0f> 0b eb fe 41 f6 04 24 02 74 40 48 8b 13 48 8d 42 02 f0 80 62 [ 157.444009] RIP [<ffffffffa00213bc>] jbd2_journal_commit_transaction+0x58b/0x115b [jbd2] [ 157.444017] RSP <ffff8801dc2adcf0> [ 157.444035] ---[ end trace 1277bd032d9ae98f ]--- ** Model information Amazon EC2 instance ** Loaded modules: Module Size Used by ext3 106502 1 jbd 37085 1 ext3 loop 11799 0 snd_pcm 60551 0 snd_timer 15582 1 snd_pcm snd 46446 2 snd_pcm,snd_timer soundcore 4598 1 snd evdev 7352 0 snd_page_alloc 6249 1 snd_pcm pcspkr 1699 0 ext4 289033 24 mbcache 5050 2 ext3,ext4 jbd2 67015 1 ext4 crc16 1319 1 ext4 dm_mod 53754 69 xen_netfront 16257 0 xen_blkfront 9435 8 ** Network interface configuration: ### ## WARNING: ## DO NOT MESS WITH THIS FILE, AND ESPECIALLY DO NOT RUN IFDOWN OR IFUP!!! ## THIS MACHINE IS AN AMAZON EC2 VM WITH NO CONSOLE ACCESS!!! ### auto lo iface lo inet loopback dns-search border.exmeritus.com exmeritus.com auto eth0 iface eth0 inet dhcp ** Network status: *** IP interfaces and addresses: 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 12:31:3b:0a:94:dc brd ff:ff:ff:ff:ff:ff inet 10.204.151.42/23 brd 10.204.151.255 scope global eth0 inet6 fe80::1031:3bff:fe0a:94dc/64 scope link valid_lft forever preferred_lft forever *** Device statistics: Inter-| Receive | Transmit face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed lo: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth0: 185543 1684 0 0 0 0 0 0 239161 1212 0 0 0 0 0 0 *** Protocol statistics: Ip: 1653 total packets received 1 with invalid addresses 0 forwarded 0 incoming packets discarded 1652 incoming packets delivered 1203 requests sent out Icmp: 9 ICMP messages received 0 input ICMP message failed. ICMP input histogram: echo requests: 9 9 ICMP messages sent 0 ICMP messages failed ICMP output histogram: echo replies: 9 IcmpMsg: InType8: 9 OutType0: 9 Tcp: 26 active connections openings 6 passive connection openings 7 failed connection attempts 7 connection resets received 6 connections established 1477 segments received 1008 segments send out 20 segments retransmited 0 bad segments received. 12 resets sent Udp: 166 packets received 0 packets to unknown port received. 0 packet receive errors 166 packets sent UdpLite: TcpExt: 2 TCP sockets finished time wait in fast timer 22 delayed acks sent 4 packets directly queued to recvmsg prequeue. 563 packet headers predicted 84 acknowledgments not containing data payload received 740 predicted acknowledgments 6 congestion windows recovered without slow start after partial ack 13 other TCP timeouts 1 connections reset due to unexpected data 2 connections reset due to early user close 7 connections aborted due to timeout IpExt: InOctets: 184459 OutOctets: 221425 *** Device features: eth0: 0x50003 lo: 0x13865 ** PCI devices: ** USB devices: not available -- System Information: Debian Release: 6.0 APT prefers stable APT policy: (990, 'stable'), (700, 'testing'), (600, 'unstable'), (500, 'squeeze-updates'), (500, 'experimental') Architecture: amd64 (x86_64) Kernel: Linux 2.6.32-5-xen-amd64 (SMP w/2 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Versions of packages linux-image-2.6.32-5-xen-amd64 depends on: ii debconf [debconf-2.0] 1.5.36.1 Debian configuration management sy ii initramfs-tools 0.98.8 tools for generating an initramfs ii linux-base 2.6.32-30 Linux image base package ii module-init-tools 3.12-1 tools for managing Linux kernel mo Versions of packages linux-image-2.6.32-5-xen-amd64 recommends: pn firmware-linux-free <none> (no description available) Versions of packages linux-image-2.6.32-5-xen-amd64 suggests: ii grub-legacy [grub] 0.97-64 GRand Unified Bootloader (Legacy v pn linux-doc-2.6.32 <none> (no description available) Versions of packages linux-image-2.6.32-5-xen-amd64 is related to: pn firmware-bnx2 <none> (no description available) pn firmware-bnx2x <none> (no description available) pn firmware-ipw2x00 <none> (no description available) pn firmware-ivtv <none> (no description available) pn firmware-iwlwifi <none> (no description available) pn firmware-linux <none> (no description available) pn firmware-linux-nonfree <none> (no description available) pn firmware-qlogic <none> (no description available) pn firmware-ralink <none> (no description available) pn xen-hypervisor <none> (no description available) -- debconf information: linux-image-2.6.32-5-xen-amd64/postinst/ignoring-do-bootloader-2.6.32-5-xen-amd64: linux-image-2.6.32-5-xen-amd64/prerm/removing-running-kernel-2.6.32-5-xen-amd64: true linux-image-2.6.32-5-xen-amd64/postinst/depmod-error-initrd-2.6.32-5-xen-amd64: false linux-image-2.6.32-5-xen-amd64/postinst/missing-firmware-2.6.32-5-xen-amd64: -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
