Is there already a tracker issue? I'm seeing the same problem here. Started deletion of a bucket with a few hundred million objects a week ago or so and I've now noticed that it's also leaking memory and probably going to crash. Going to investigate this further...
Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Tue, Jul 9, 2019 at 1:26 PM Matt Benjamin <mbenj...@redhat.com> wrote: > Hi Harald, > > Please file a tracker issue, yes. (Deletes do tend to be slower, > presumably due to rocksdb compaction.) > > Matt > > On Tue, Jul 9, 2019 at 7:12 AM Harald Staub <harald.st...@switch.ch> > wrote: > > > > Currently removing a bucket with a lot of objects: > > radosgw-admin bucket rm --bucket=$BUCKET --bypass-gc --purge-objects > > > > This process was killed by the out-of-memory killer. Then looking at the > > graphs, we see a continuous increase of memory usage for this process, > > about +24 GB per day. Removal rate is about 3 M objects per day. > > > > It is not the fastest hardware, and this index pool is still without > > SSDs. The bucket is sharded, 1024 shards. We are on Nautilus 14.2.1, now > > about 500 OSDs. > > > > So with this bucket with 60 M objects, we would need about 480 GB of RAM > > to come through. Or is there a workaround? Should I open a tracker issue? > > > > The killed remove command can just be called again, but it will be > > killed again before it finishes. Also, it has to run some time until it > > continues to actually remove objects. This "wait time" is also > > increasing. Last time, after about 16 M objects already removed, the > > wait time was nearly 9 hours. Also during this time, there is a memory > > ramp, but not so steep. > > > > BTW it feels strange that the removal of objects is slower (about 3 > > times) than adding objects. > > > > Harry > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > -- > > Matt Benjamin > Red Hat, Inc. > 315 West Huron Street, Suite 140A > Ann Arbor, Michigan 48103 > > http://www.redhat.com/en/technologies/storage > > tel. 734-821-5101 > fax. 734-769-8938 > cel. 734-216-5309 > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
