> Op 7 november 2016 om 13:17 schreef Wido den Hollander <w...@42on.com>: > > > > > Op 4 november 2016 om 2:05 schreef Joao Eduardo Luis <j...@suse.de>: > > > > > > On 11/03/2016 06:18 PM, w...@42on.com wrote: > > > > > >> Personally, I don't like this solution one bit, but I can't see any > > >> other way without a patched monitor, or maybe ceph_monstore_tool. > > >> > > >> If you are willing to wait till tomorrow, I'll be happy to kludge a > > >> sanitation feature onto ceph_monstore_tool that will clean those > > >> versions for you (latency being due to coding + testing + building). > > >> > > > > > > Yes, no rush. I will wait. Thanks! > > > > Hey Wido, > > > > I just pushed 'wip-monstore-tool-sanitize-paxos.hammer' to upstream's > > ceph.git. It should be built by morning. > > > > This is a half-assed feature to ceph-monstore-tool, so you should test > > it first on a copy of your monitors, just to make sure I didn't forget > > any corner case. > > > > The command you want is > > > > ceph-monstore-tool <path-to-monstore> sanitize-paxos -- --dry-run > > > > and, if you're happy with whatever it spits out, then drop --dry-run. > > > > Should this work for you, next step will be to merge this onto master > > and, hopefully, backport to previous versions. We'll see. > > > > I tried, but it doesn't seem to work. >
To wrap this up, after having a off the list discussion with Joao this was fixed. The fix is in this branch: https://github.com/jecluis/ceph/tree/wip-monstore-tool-copy.hammer Using 'store-copy-conservative' from the old to new path the store got trimmed and went down from 36GB to ~100M. Ran this on all three Monitors one by one and it fixed it. Joao will work on getting this back in upstream. Wido > # time ./ceph-monstore-tool /var/lib/ceph/mon/ceph-mon1 sanitize-paxos -- > > It ran for an hour (without dry-run) and outputted: > > paxos [175000202 .. 175000831] > found surplus paxos versions that ought to be removed: > - [ 225964 .. 226320] > - [ 260032 .. 260401] > - [ 891292 .. 891642] > ... > ... > ... > - [ 35210293 .. 35210571] > - [ 35213765 .. 35213766] > removing paxos versions [225964 .. 226320] > removing paxos versions [260032 .. 260401] > ... > ... > ... > removing paxos versions [35188867 .. 35189115] > removing paxos versions [35210293 .. 35210571] > removing paxos versions [35213765 .. 35213766] > all done! > > real 57m18.904s > user 56m26.928s > sys 0m9.884s > > > I ran it on the Monitor which will become primary again after it syncs. After > running the tool the store is still 36GB in size, grows to 50GB during the > sync and then trims down again to 36GB. > > Wido > > > HTH, > > > > -Joao > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
