On 09/16/2014 04:35 PM, Gregory Farnum wrote:
I don't really know; Joao has handled all these cases. I *think* they've
been tied to a few bad versions of LevelDB, but I'm not certain. (There
were a number of discussions about it on the public mailing lists.)
-Greg
On Tuesday, September 16, 2014, Florian Haas <flor...@hastexo.com
<mailto:flor...@hastexo.com>> wrote:
Hi Greg,
just picked up this one from the archive while researching a different
issue and thought I'd follow up.
On Tue, Aug 19, 2014 at 6:24 PM, Gregory Farnum <g...@inktank.com
<javascript:;>> wrote:
> The sst files are files used by leveldb to store its data; you cannot
> remove them. Are you running on a very small VM? How much space are
> the files taking up in aggregate?
> Speaking generally, I think you should see something less than a GB
> worth of data there, but some versions of leveldb under some
scenarios
> are known to misbehave and grow pretty large.
Can you elaborate on the scenarios where leveldb is misbehaving? I've
also seen reports of this before, with .sst files growing to several
GB in size. Is this a cause for concern (for example, would you expect
mons to slow down) and if so, how would you recover? Would you
essentially nuke the mon and replace it with another?
Forcing the monitor to compact on start and restarting the mon is the
current workaround for overgrown ssts. This happens on a regular basis
with some clusters and I've not been able to track down the source. It
seems that leveldb keeps hold of previous, useless data and will only
relinquish it upon being closed.
When this happens monitors do not slow down per se but they tend to
misbehave: hanging at times, spurious elections, flapping quorum. This
is mostly because, up until recently, the monitors would wait on updates
to be written to leveldb. Leveldb would in turn misbehave as (afaict)
it's busy dealing with clutter. Sage pushed patches to master to have
the monitor performing async writes to leveldb so to prevent the monitor
hanging when leveldb hangs, and this should help quite a bit with all
the weirdness.
So to recap: current workaround is add 'mon compact on start = true' on
your ceph.conf and restart the monitor.
-Joao
Cheers,
Florian
--
Software Engineer #42 @ http://inktank.com | http://ceph.com
--
Joao Eduardo Luis
Software Engineer | http://inktank.com | http://ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
