> Op 25 augustus 2017 om 15:00 schreef Matthew Vernon <m...@sanger.ac.uk>: > > > Hi, > > We have a medium-sized (2520 osds, 42 hosts, 88832 pgs, 15PB raw > capacity) Jewel cluster (on Ubuntu), and in normal operation, our mon > store size is around the 1.2G mark. I've noticed, though, that when > doing larger rebalances, they can grow really very large (up to nearly > 70G, which is nearly all the rootfs on our mons); when recently adding > more osds to the cluster (and consequently increasing pg number by > increasing the size of an existing pool) we nearly ran out of disk space > on the mons due to their mon stores getting very big. > > Since I imagine having your mons run out of disk mid-rebalance is A Bad > Idea, is there a way to estimate how big a mon store might get? >
As long as there is at least one PG not active+clean there MONs will keep a history of all OSDMaps, this can grow large. With Kraken en Luminous this has changed, but I haven't seen very large clusters yet with that version running, they are all still on Jewel. With Jewel I've seen MON stores of >200GB when doing a multi-day rebalance with 2000 OSDs in the system. Wido > Thanks, > > Matthew > > > -- > The Wellcome Trust Sanger Institute is operated by Genome Research > Limited, a charity registered in England with number 1021457 and a > company registered in England with number 2742969, whose registered > office is 215 Euston Road, London, NW1 2BE. > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
