Hi,
One of the osd in my cluster downs w no reason, I saw the error message in the
log below, I restarted osd, but after several hours, the problem come back
again. Could you help?
"Too many open files not handled on operation 24 (541468.0.1, or op 1, counting
from 0)
-96> 2014-05-14 22:12:24.281185 7f617b33e700 5 -- op tracker -- , seq:
788808, time: 2014-05-14 22:12:24.281164, event: reached_pg, request:
osd_op(client.21276.0:3884815 rb.0.31c7.238e1f 29.000000003c15 [write
2273280~65536] 4.110fcf4 e12271) v4
-95> 2014-05-14 22:12:24.281192 7f618556d700 0
filestore(/var/lib/ceph/osd/ceph-3) unexpected error code
-94> 2014-05-14 22:12:24.281197 7f6181b4b700 5 -- op tracker -- , seq:
788843, time: 2014-05-14 22:12:24.281011, event: header_read, request:
osd_op(client.21276.0:3884929 rb.0.31c7.238e1 f29.000000005614 [write
3137536~65536] 4.63e147e e12271) v4
> 2014-05-14 22:12:24.289987 7f6185d6e700 -1 os/FileStore.cc: In function
> 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t,
> int, ThreadPool::TPHandle*)' thre ad 7f6185d6e700 time 2014-05-14
> 22:12:24.282488
os/FileStore.cc: 2448: FAILED assert(0 == "unexpected error")
ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)
1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int,
ThreadPool::TPHandle*)+0x11c3) [0x723a43]
2: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*,
std::allocator<ObjectStore::Transaction*> >&, unsigned long,
ThreadPool::TPHandle*)+0x74) [0x72a4d4]
3: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x29a)
[0x72a78a]
4: (ThreadPool::worker(ThreadPool::WorkThread*)+0x551) [0x988f21]
5: (ThreadPool::WorkThread::entry()+0x10) [0x98bf50]
6: /lib64/libpthread.so.0() [0x3a7ce079d1]
7: (clone()+0x6d) [0x3a7cae8b6d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this."........
#iostat
avg-cpu: %user %nice %system %iowait %steal %idle
0.44 0.00 0.14 0.41 0.00 99.01
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sdb 1.23 0.10 35.72 12738 4762008
sdc 5.25 214.25 1288.81 28564314 171824232
sdd 4.16 139.98 1021.69 18662490 136211888
sde 4.61 207.50 1039.20 27663258 138545960
sdf 7.94 203.24 2530.63 27095930 337383704
sdg 4.77 0.57 1459.29 75330 194553064
sdh 4.38 0.37 1287.42 48954 171638304
sdi 85.80 132.13 8157.53 17616004 1087562272
sdj 8.77 10.99 1701.90 1465844 226897024
sda 4.55 0.60 1331.50 80010 177516216
Wei Cao (Buddy)
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
