On Jun 8, 2010, at 12:08 PM, Dianne Yumul wrote:
> Hello,
>
> I'm getting the same thing on one of our servers since upgrading to CentOS
> 5.5:
>
> INFO: task pdflush:21249 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> pdflush D 00001EE1 3540 21249 11 21226 (L-TLB)
> f0b59f24 00000046 af2b024f 00001ee1 c04ec500 00000000 c041e314 0000000a
> f76ae550 af2b2566 00001ee1 00002317 00000001 f76ae65c c180dc44 c198e200
> 00000000 c180e5e4 f0b59fa8 f76ae550 c180dc44 c061bbc0 f76ae550 ffffffff
> Call Trace:
> [<c04ec500>] __next_cpu+0x12/0x21
> [<c041e314>] find_busiest_group+0x177/0x462
> [<c061bbc0>] schedule+0xbc/0xa55
> [<c061d981>] rwsem_down_read_failed+0x128/0x143
> [<c04389ef>] .text.lock.rwsem+0x35/0x3a
> [<c047c014>] sync_supers+0x2f/0xb8
> [<c045df9c>] wb_kupdate+0x36/0x10f
> [<c045e431>] pdflush+0x0/0x1a3
> [<c045e53c>] pdflush+0x10b/0x1a3
> [<c045df66>] wb_kupdate+0x0/0x10f
> [<c0435f43>] kthread+0xc0/0xed
> [<c0435e83>] kthread+0x0/0xed
> [<c0405c53>] kernel_thread_helper+0x7/0x10
>
>> From the bugs already filed, it seems to happen to many (or any?) processes
>> and some notice hangups and performance drops. But our system seems okay,
>> probably because it has low traffic and is mostly idle. But I'll still
>> reboot to the previous kernel version tonight.
>
> dianne
>
> On Jun 8, 2010, at 1:04 AM, Ireneusz Piasecki wrote:
>
>> W dniu 2010-06-08 09:54, Tsuyoshi Nagata pisze:
>>> Hi
>>> (2010/06/08 5:12), Steve Brooks wrote:
>>>> Jun 7 19:45:21 sraid3 kernel: [<ffffffff800ec2a2>] inode_wait+0x0/0xd
>>>> Jun 7 19:45:21 sraid3 kernel: [<ffffffff80063ab0>]
>>>> out_of_line_wait_on_bit+0x6c/0x78
>>>> Jun 7 19:45:21 sraid3 kernel: [<ffffffff800a0aec>]
>>>> wake_bit_function+0x0/0x23
>>>> Jun 7 19:45:21 sraid3 kernel: [<ffffffff8003dbbf>] ifind_fast+0x6e/0x83
>>> This message was created at Linux/fs/inode.c:ifind_fast()
>>> The source code was bellows,
>>>
>>> Linux/fs/inode.c:
>>> 912 static struct inode *ifind_fast(struct super_block *sb,
>>> 913 struct hlist_head *head, unsigned long ino)
>>> 914 {
>>> 915 struct inode *inode;
>>> 916
>>> 917 *LOCK* spin_lock(&inode_lock);<= This takes
>>> 918 inode = find_inode_fast(sb, head, ino);<= more 120s.
>>> 919 if (inode) {
>>> 920 __iget(inode);
>>> 921 *UNLOCK* spin_unlock(&inode_lock);
>>> 922 wait_on_inode(inode);
>>> 923 return inode;
>>> 924 }
>>> 925 spin_unlock(&inode_lock);
>>> 926 return NULL;
>>> 927 }
>>> 928
>>>
>>> I guess your your file system has a trouble with i-node(file number)
>>> resources.
>>> CAUSES:
>>> Hard Disk trouble (bit error/raid trouble.)
>>> i-node trouble (overflow. etc.)
>>> Memory/CPU trouble(&inode_lock)
>>>
>>> Buy Fresh Hard disks& rebuild them is convenience way.
>>> Or memtest86 can finds DIMM trouble.(or CPU, mother board)
>>> Or ext4 bug in 194.3.1 kernel, back to ext3!
>>>
>> Ok, then i will test all of my centos 5.5 32 nodes: cpu, ram, disks etc.
>> This came with the kernel of Centos 5.5. Before there was'nt such
>> errors/warrning. Redhat bugizilla:
>> https://bugzilla.redhat.com/show_bug.cgi?id=573106
>>
>> I.Piasecki
>>
>>> -tsuyoshi
>>> _______________________________________________
>>> CentOS mailing list
>>> CentOS@centos.org
>>> http://lists.centos.org/mailman/listinfo/centos
>>>
>>
>> _______________________________________________
>> CentOS mailing list
>> CentOS@centos.org
>> http://lists.centos.org/mailman/listinfo/centos
>>
>
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
Oh crud, I did a top post! So sorry, won't happen again.
dianne
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
