We had a similar issue in Firefly, where we had a very large number
(about 1.500.000) of buckets for a single RGW user. We observed a
number of slow requests in day-to-day use, but did not think much of it
at the time.
At one point the primary OSD managing the list of buckets for that user
crashed and could not restart, because processing the tremendous amount
of buckets on startup - which also seemed to be single-threaded,
judging by to 100% CPU usage we could see - took longer than the
suicide-timeout. That lead to this OSD crashing again, and again.
Eventually, it would be marked out and the secondary tried to process
the list with the same result, leading to a cascading failure.
While I am quite certain it is a different code path in your case (you
speak about a handful of buckets), it certainly sounds like the a very
similar issue. Do you have lots of objects in those few buckets, or are
they few, but large in size to reach the 30TB? Worst case you might be
in for a similar procedure as we had to take: Take load off the
cluster, increase the timeouts to ridiculous levels and copy the data
over into a more evenly distributed set of buckets (users in our case).
Fortunately as long as we did not try to write to the problematic
buckets, we could still read from them.
Please notice that this is only a guess, I could be completely wrong.
Daniel
On 2015-11-03 13:33:19 +0000, Gerd Jakobovitsch said:
Dear all,
I have a cluster running hammer (0.94.5), with 5 nodes. The main usage
is for S3-compatible object storage.
I am getting to a very troublesome problem at a ceph cluster. A single
object in the .rgw.buckets.index is not responding to request and takes
a very long time while recovering after an osd restart. During this
time, the OSDs where this object is mapped got heavily loaded, with
high cpu as well as memory usage. At the same time, the directory
/var/lib/ceph/osd/ceph-XX/current/omap gets a large number of entries (
> 10000), that won't decrease.
Very frequently, I get >100 blocked requests for this object, and the
main OSD that stores it ends up accepting no other requests. Very
frequently the OSD ends up crashing due to filestore timeout, and
getting it up again is very troublesome - it usually has to run alone
in the node for a long time, until the object gets recovered, somehow.
At the OSD logs, there are several entries like these:
-7051> 2015-11-03 10:46:08.339283 7f776974f700 10 log_client logged
2015-11-03 10:46:02.942023 osd.63 10.17.0.9:6857/2002 41 : cluster
[WRN] slow re
quest 120.003081 seconds old, received at 2015-11-03 10:43:56.472825:
osd_repop(osd.53.236531:7 34.7
8a7482ff/.dir.default.198764998.1/head//34 v 2369
84'22) currently commit_sent
2015-11-03 10:28:32.405265 7f0035982700 0 log_channel(cluster) log
[WRN] : 97 slow requests, 1 included below; oldest blocked for >
2046.502848 secs
2015-11-03 10:28:32.405269 7f0035982700 0 log_channel(cluster) log
[WRN] : slow request 1920.676998 seconds old, received at 2015-11-03
09:56:31.7282
24: osd_op(client.210508702.0:14696798 .dir.default.198764998.1 [call
rgw.bucket_prepare_op] 15.8a7482ff ondisk+write+known_if_redirected
e236956) cur
rently waiting for blocked object
Is there any way to go deeper into this problem, or to rebuild the .rgw
index without loosing data? I currently have 30 TB of data in the
cluster - most of it concentrated in a handful of buckets - that I
can't loose.
Regards.
--
--
As informações contidas nesta mensagem são CONFIDENCIAIS, protegidas
pelo sigilo legal e por direitos autorais. A divulgação, distribuição,
reprodução ou qualquer forma de utilização do teor deste documento
depende de autorização do emissor, sujeitando-se o infrator às sanções
legais. Caso esta comunicação tenha sido recebida por engano, favor
avisar imediatamente, respondendo esta mensagem.
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
--
Daniel Schneller
Principal Cloud Engineer
CenterDevice GmbH
https://www.centerdevice.de
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
