Re: [ceph-users] High 0.94.5 OSD memory use at 8GB RAM/TB raw disk during recovery

2015-12-01 Thread Ross Annetts
Hi, May be a bit late now, but if you are adding additional capacity I would recommend adding one OSD at a time. If you want even more control add the OSD with a weight of 0 and gradually increase. This limits the amount of data that needs to be moved and resources required. Regards, Ross

Re: [ceph-users] High 0.94.5 OSD memory use at 8GB RAM/TB raw disk during recovery

2015-12-01 Thread Laurent GUERBY
On Tue, 2015-12-01 at 13:51 -0600, Ryan Tokarek wrote: > > On Nov 30, 2015, at 6:52 PM, Laurent GUERBY wrote: > > > > Hi, > > > > We lost a disk today in our ceph cluster so we added a new machine with > > 4 disks to replace the capacity and we activated straw1 tunable too > > (we also tried str

Re: [ceph-users] High 0.94.5 OSD memory use at 8GB RAM/TB raw disk during recovery

2015-12-01 Thread Ryan Tokarek
> On Nov 30, 2015, at 6:52 PM, Laurent GUERBY wrote: > > Hi, > > We lost a disk today in our ceph cluster so we added a new machine with > 4 disks to replace the capacity and we activated straw1 tunable too > (we also tried straw2 but we quickly backed up this change). > > During recovery OSD

Re: [ceph-users] High 0.94.5 OSD memory use at 8GB RAM/TB raw disk during recovery

2015-12-01 Thread Laurent GUERBY
On Mon, 2015-11-30 at 18:58 -0600, Mark Nelson wrote: > Hi Laurent, > > Wow, that's excessive! I'd see if anyone else has any tricks first, but > if nothing else helps, running an OSD through valgrind with massif will > probably help pinpoint what's going on. Have you tweaked the recovery > t

Re: [ceph-users] High 0.94.5 OSD memory use at 8GB RAM/TB raw disk during recovery

2015-11-30 Thread Mark Nelson
Oh, forgot to ask, any core dumps? Mark On 11/30/2015 06:58 PM, Mark Nelson wrote: Hi Laurent, Wow, that's excessive! I'd see if anyone else has any tricks first, but if nothing else helps, running an OSD through valgrind with massif will probably help pinpoint what's going on. Have you twea

Re: [ceph-users] High 0.94.5 OSD memory use at 8GB RAM/TB raw disk during recovery

2015-11-30 Thread Mark Nelson
Hi Laurent, Wow, that's excessive! I'd see if anyone else has any tricks first, but if nothing else helps, running an OSD through valgrind with massif will probably help pinpoint what's going on. Have you tweaked the recovery tunables at all? Mark On 11/30/2015 06:52 PM, Laurent GUERBY wr

[ceph-users] High 0.94.5 OSD memory use at 8GB RAM/TB raw disk during recovery

2015-11-30 Thread Laurent GUERBY
Hi, We lost a disk today in our ceph cluster so we added a new machine with 4 disks to replace the capacity and we activated straw1 tunable too (we also tried straw2 but we quickly backed up this change). During recovery OSD started crashing on all of our machines the issue being OSD RAM usage th