Sage, does firefly require to manually set "ulimit -n" while add a new storage node with 16 osds(500G disks)?
Wei Cao (Buddy) -----Original Message----- From: Sage Weil [mailto:s...@inktank.com] Sent: Thursday, May 15, 2014 10:49 PM To: Cao, Buddy Cc: ceph-us...@ceph.com Subject: Re: [ceph-users] osd down/autoout problem On Thu, 15 May 2014, Cao, Buddy wrote: > ?Too many open files not handled on operation 24 (541468.0.1, or op 1, > counting from 0) You need to increase the 'ulimit -n' max open files limit. You can do this in ceph.conf with 'max open files' if it's sysvinit or manually in /etc/init/ceph-osd.conf if its upstart. sage > > -96> 2014-05-14 22:12:24.281185 7f617b33e700 5 -- op tracker -- , seq: > 788808, time: 2014-05-14 22:12:24.281164, event: reached_pg, request: > osd_op(client.21276.0:3884815 rb.0.31c7.238e1f 29.000000003c15 [write > 2273280~65536] 4.110fcf4 e12271) v4 > > -95> 2014-05-14 22:12:24.281192 7f618556d700 0 > filestore(/var/lib/ceph/osd/ceph-3) unexpected error code > > -94> 2014-05-14 22:12:24.281197 7f6181b4b700 5 -- op tracker -- , seq: > 788843, time: 2014-05-14 22:12:24.281011, event: header_read, request: > > osd_op(client.21276.0:3884929 rb.0.31c7.238e1 f29.000000005614 [write > 3137536~65536] 4.63e147e e12271) v4 > > > 2014-05-14 22:12:24.289987 7f6185d6e700 -1 os/FileStore.cc: In > > function > 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, > uint64_t, int, ThreadPool::TPHandle*)' thre ad 7f6185d6e700 time > 2014-05-14 > 22:12:24.282488 > > os/FileStore.cc: 2448: FAILED assert(0 == "unexpected error") > > ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60) > > 1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned > long, int, ThreadPool::TPHandle*)+0x11c3) [0x723a43] > > 2: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, > std::allocator<ObjectStore::Transaction*> >&, unsigned long, > ThreadPool::TPHandle*)+0x74) [0x72a4d4] > > 3: (FileStore::_do_op(FileStore::OpSequencer*, > ThreadPool::TPHandle&)+0x29a) [0x72a78a] > > 4: (ThreadPool::worker(ThreadPool::WorkThread*)+0x551) [0x988f21] > > 5: (ThreadPool::WorkThread::entry()+0x10) [0x98bf50] > > 6: /lib64/libpthread.so.0() [0x3a7ce079d1] > > 7: (clone()+0x6d) [0x3a7cae8b6d] > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this.???.. > > > > > > #iostat > > avg-cpu: %user %nice %system %iowait %steal %idle > > 0.44 0.00 0.14 0.41 0.00 99.01 > > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn > > sdb 1.23 0.10 35.72 12738 4762008 > > sdc 5.25 214.25 1288.81 28564314 171824232 > > sdd 4.16 139.98 1021.69 18662490 136211888 > > sde 4.61 207.50 1039.20 27663258 138545960 > > sdf 7.94 203.24 2530.63 27095930 337383704 > > sdg 4.77 0.57 1459.29 75330 194553064 > > sdh 4.38 0.37 1287.42 48954 171638304 > > sdi 85.80 132.13 8157.53 17616004 1087562272 > > sdj 8.77 10.99 1701.90 1465844 226897024 > > sda 4.55 0.60 1331.50 80010 177516216 > > > > > > Wei Cao (Buddy) > > > > > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
