Thanks a lot Sebastian, i'm going to try that, also i'm having an
issue while trying to test a rbd creation, i've install in the deploy
server the ceph-client:
ceph@ceph-deploy01:/etc/ceph$ sudo rbd -n client.ceph-test -k
/home/ceph/ceph-cluster/ceph.client.admin.keyring create --size 10240
cephdata
2013-11-29 15:20:25.683930 7fcd9979c780 0 librados:
client.ceph-openstack authentication error (1) Operation not permitted
rbd: couldn't connect to the cluster!
Anyone know what could be the issue here? maybe it has something to
do with keys or maybe not...
Thanks in advance,
Best regards,
German Anders
--- Original message ---
Asunto: Re: [ceph-users] radosgw daemon stalls on download of some
files
De: Sebastian <webmas...@mailz.de>
Para: ceph-users <ceph-users@lists.ceph.com>
Fecha: Friday, 29/11/2013 16:18
Hi Yehuda,
It's interesting, the responses are received but seems that they
aren't being handled (hence the following pings). There are a few
things that you could look at. First, try to connect to the admin
socket and see if you get any useful information from there. This
could include in-flight requests, look for other requests that have
not completed. Also see if there's indication for requests throttling.
Do you refer to the methods mentioned here?
http://ceph.com/docs/dumpling/radosgw/troubleshooting/?
Unfortunately the socket file is not present. Do i have to activate it
in the config somehow? I could not find any reference to that in the
docs. Is it already included in my radosgw version?
radosgw -v
ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)
Another thing to look at would be at the seemingly unrelated timeout
messages. These should not happen and might indicate that there's
something that is holding you up that shouldn't. Try searching for the
same thread id that is specified in these messages (omit the 0x
prefix), and see what's the last thing that it's doing.
I checked that:
http://pastebin.com/Z23PWwjt
i do not see anything unusual before the messages happen, but maybe
you see something odd.
You could also try turning on also 'debug objecter = 20', see if it
provides more info (it's very verbose though).
Did that, but that is way to verbose for me ;) I uploaded it here:
http://pastebin.com/VBPAVP6z
There might be some requests mixed into it, but the one for
cdn/52974400c6dd6ca719000004/source.avi is the one that stalled.
How much are you loading the gateway before that happens? We've seen a
similar issue in the past that was related to the fcgi library that is
dynamically linked with the radosgw process (that is, not the apache
mod_fastcgi module). This, however, would only happen when there's
heavy load and the fd numbers handled by the radosgw surpassed 1024
(buggy library that was using select() instead of poll()).
There are not that many requests on the Storage, maybe 10-20 req/min.
The cluster serves as a source for a CDN, so once the resource is
fetched it should not be fetched again soon. I checked for the open
files, and there are only about 10-20 open file handles for the
radosgw process. So this probably is not the issue.
Sebastian
Yehuda
On Fri, Nov 29, 2013 at 7:28 AM, Sebastian <webmas...@mailz.de> wrote:
Hi,
thanks for the hint. I tried this again and noticed that the time out
message does seem to be unrelated. Here is the log file for a stalling
request with debug turned on:
http://pastebin.com/DcQuc9wP
I really cannot really find a real "error" in the log. The download
stalls at about 500kb at that point though. Restarting radosgw fixes
it for 1 download only, the next one is broken again. But as i said
this does not happen for all files.
Sebastian
On 27.11.2013, at 21:53, Yehuda Sadeh wrote:
On Wed, Nov 27, 2013 at 4:46 AM, Sebastian <webmas...@mailz.de> wrote:
Hi,
we have a setup of 4 Servers running ceph and radosgw. We use it as an
internal S3 service for our files. The Servers run Debian Squeeze with
Ceph 0.67.4.
The cluster has been running smoothly for quite a while, but we are
currently experiencing issues with the radosgw. For some files the
HTTP Download just stalls at around 500kb.
The Apache error log just says:
[error] [client ] FastCGI: comm with server "/var/www/s3gw.fcgi"
aborted: idle timeout (30 sec)
[error] [client ] Handler for fastcgi-script returned invalid result
code 1
radosgw logging:
7f00bc66a700 1 heartbeat_map is_healthy 'RGWProcess::m_tp thread
0x7f00934bb700' had timed out after 600
7f00bc66a700 1 heartbeat_map is_healthy 'RGWProcess::m_tp thread
0x7f00ab4eb700' had timed out after 600
The interesting thing is that the cluster health is fine an only some
files are not working properly. Most of them just work fine. A restart
of radosgw fixes the issue. The other ceph logs are also clean.
Any idea why this happens?
No, but you can turn on 'debug ms = 1' on your gateway ceph.conf, and
that might give some better indication.
Yehuda
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
