I've just tried setting up the radosgw on centos6 according to
http://ceph.com/docs/master/radosgw/config/
There didn't seem to be an init script in the rpm I installed, so I
copied the one from here:
https://raw.githubusercontent.com/ceph/ceph/31b0823deb53a8300856db3c104c0e16d05e79f7/src/init-radosgw.sysv
That launches one process.
While I can run the admin commands just fine to create users etc.,
making a simple wget request to the domain I set up returns a 500 due to
a timeout. Every request I make results in another radosgw process
being created, which seems to start even more processes itself. I only
have to make a few requests to have about 60 radosgw processes.
I am at a bit of a loss to tell what is going on. I've included what I
assume to be the error below. I see some filures to aquire locks, and
it complaining that another process has already created the unix socket,
but don't know how relevant those are. If anyone could point me in the
right direction, I would appreciate it.
2014-06-15 20:21:03.814081 7f8c94cf9700 1 -- 10.30.83.29:0/1028955 -->
10.30.85.60:6806/18981 -- ping v1 -- ?+0 0x7f8c98068e50 con 0x1e73630
2014-06-15 20:21:03.814097 7f8c94cf9700 1 -- 10.30.83.29:0/1028955 -->
10.30.85.60:6813/20066 -- ping v1 -- ?+0 0x7f8c980693e0 con 0x1e76460
2014-06-15 20:21:03.814108 7f8c94cf9700 1 -- 10.30.83.29:0/1028955 -->
10.30.85.60:6810/19519 -- ping v1 -- ?+0 0x7f8c98069600 con 0x1e79b90
2014-06-15 20:21:03.815286 7fd779f84820 0 framework: fastcgi
2014-06-15 20:21:03.815305 7fd779f84820 0 starting handler: fastcgi
2014-06-15 20:21:03.816726 7fd74e6f8700 1 -- 10.30.83.29:0/1008789 -->
10.30.85.60:6800/18433 -- osd_op(client.6329.0:24 [pgls start_epoch 0]
7.0 ack+read e290) v4 -- ?+0 0x7fd75800a630 con 0x1505210
2014-06-15 20:21:03.817584 7fd770adb700 1 -- 10.30.83.29:0/1008789 <==
osd.1 10.30.85.60:6800/18433 9 ==== osd_op_reply(24 [pgls start_epoch
0] v0'0 uv0 ondisk = 1) v6 ==== 167+0+44 (1244889235 0 139081063)
0x7fd754000ce0 con 0x1505210
2014-06-15 20:21:03.817665 7fd770adb700 1 -- 10.30.83.29:0/1008789 -->
10.30.85.60:6806/18981 -- osd_op(client.6329.0:25 [pgls start_epoch 0]
7.1 ack+read e290) v4 -- ?+0 0x7fd75800aae0 con 0x15043c0
2014-06-15 20:21:03.819356 7fd770adb700 1 -- 10.30.83.29:0/1008789 <==
osd.2 10.30.85.60:6806/18981 5 ==== osd_op_reply(25 [pgls start_epoch
0] v0'0 uv0 ondisk = 1) v6 ==== 167+0+44 (3347405639 0 139081063)
0x1512950 con 0x15043c0
2014-06-15 20:21:03.819509 7fd770adb700 1 -- 10.30.83.29:0/1008789 -->
10.30.85.61:6800/28605 -- osd_op(client.6329.0:26 [pgls start_epoch 0]
7.2 ack+read e290) v4 -- ?+0 0x7fd75800c3e0 con 0x7fd75800ac80
2014-06-15 20:21:03.819635 7fd74fafa700 2 garbage collection: start
2014-06-15 20:21:03.819798 7fd74fafa700 1 -- 10.30.83.29:0/1008789 -->
10.30.85.60:6806/18981 -- osd_op(client.6329.0:27 gc.21 [call lock.lock]
6.6dc01772 ondisk+write e290) v4 -- ?+0 0x7fd7600022b0 con 0x15043c0
2014-06-15 20:21:03.823164 7fd770adb700 1 -- 10.30.83.29:0/1008789 <==
osd.2 10.30.85.60:6806/18981 6 ==== osd_op_reply(27 gc.21 [call] v0'0
uv0 ondisk = -16 ((16) Device or resource busy)) v6 ==== 172+0+0
(3774749926 0 0) 0x1512950 con 0x15043c0
2014-06-15 20:21:03.823309 7fd74fafa700 0 RGWGC::process() failed to
acquire lock on gc.21
2014-06-15 20:21:03.823457 7fd74fafa700 1 -- 10.30.83.29:0/1008789 -->
10.30.85.60:6810/19519 -- osd_op(client.6329.0:28 gc.22 [call lock.lock]
6.97748d0d ondisk+write e290) v4 -- ?+0 0x7fd760002bc0 con 0x150b280
2014-06-15 20:21:03.821327 7fd74d2f6700 -1 common/Thread.cc: In function
'void Thread::create(size_t)' thread 7fd74d2f6700 time 2014-06-15
20:21:03.819948
common/Thread.cc: 110: FAILED assert(ret == 0)
ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
1: (Thread::create(unsigned long)+0x8a) [0x7fd77900363a]
2: (ThreadPool::start_threads()+0x12e) [0x7fd778fe770e]
3: (ThreadPool::start()+0x7a) [0x7fd778feaa9a]
4: (RGWFCGXProcess::run()+0x195) [0x4ae305]
5: /usr/bin/radosgw() [0x4b3bbe]
6: (()+0x79d1) [0x7fd7772929d1]
7: (clone()+0x6d) [0x7fd776fdfb5d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- begin dump of recent events ---
-125> 2014-06-15 20:21:03.205715 7fd779f84820 5 asok(0x14c3070)
register_command perfcounters_dump hook 0x14c4a80
-124> 2014-06-15 20:21:03.205761 7fd779f84820 5 asok(0x14c3070)
register_command 1 hook 0x14c4a80
-123> 2014-06-15 20:21:03.205766 7fd779f84820 5 asok(0x14c3070)
register_command perf dump hook 0x14c4a80
-122> 2014-06-15 20:21:03.205773 7fd779f84820 5 asok(0x14c3070)
register_command perfcounters_schema hook 0x14c4a80
-121> 2014-06-15 20:21:03.205780 7fd779f84820 5 asok(0x14c3070)
register_command 2 hook 0x14c4a80
-120> 2014-06-15 20:21:03.205784 7fd779f84820 5 asok(0x14c3070)
register_command perf schema hook 0x14c4a80
-119> 2014-06-15 20:21:03.205787 7fd779f84820 5 asok(0x14c3070)
register_command config show hook 0x14c4a80
-118> 2014-06-15 20:21:03.205791 7fd779f84820 5 asok(0x14c3070)
register_command config set hook 0x14c4a80
-117> 2014-06-15 20:21:03.205798 7fd779f84820 5 asok(0x14c3070)
register_command config get hook 0x14c4a80
-116> 2014-06-15 20:21:03.205801 7fd779f84820 5 asok(0x14c3070)
register_command log flush hook 0x14c4a80
-115> 2014-06-15 20:21:03.205805 7fd779f84820 5 asok(0x14c3070)
register_command log dump hook 0x14c4a80
-114> 2014-06-15 20:21:03.205809 7fd779f84820 5 asok(0x14c3070)
register_command log reopen hook 0x14c4a80
-113> 2014-06-15 20:21:03.211474 7fd779f84820 0 ceph version 0.80.1
(a38fe1169b6d2ac98b427334c12d7cf81f809b74), process radosgw, pid 8787
-112> 2014-06-15 20:21:03.211487 7fd779f84820 -1 WARNING: libcurl
doesn't support curl_multi_wait()
-111> 2014-06-15 20:21:03.211491 7fd779f84820 -1 WARNING: cross zone /
region transfer performance may be affected
-110> 2014-06-15 20:21:03.213562 7fd779f84820 1 finished
global_init_daemonize
-109> 2014-06-15 20:21:03.213886 7fd779f84820 5 asok(0x14c3070) init
/var/run/ceph/ceph-client.radosgw.cephadmin.asok
-108> 2014-06-15 20:21:03.213923 7fd779f84820 5 asok(0x14c3070)
bind_and_listen /var/run/ceph/ceph-client.radosgw.cephadmin.asok
-107> 2014-06-15 20:21:03.215425 7fd779f84820 -1 asok(0x14c3070)
AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed
to bind the UNIX domain socket to
'/var/run/ceph/ceph-client.radosgw.cephadmin.asok': (17) File exists
-106> 2014-06-15 20:21:03.217405 7fd779f84820 10 monclient(hunting):
build_initial_monmap
-105> 2014-06-15 20:21:03.217514 7fd779f84820 1 librados: starting
msgr at :/0
-104> 2014-06-15 20:21:03.217530 7fd779f84820 1 librados: starting
objecter
-103> 2014-06-15 20:21:03.217583 7fd779f84820 1 -- :/0
messenger.start
-102> 2014-06-15 20:21:03.217619 7fd779f84820 1 librados: setting
wanted keys
-101> 2014-06-15 20:21:03.217624 7fd779f84820 1 librados: calling
monclient init
-100> 2014-06-15 20:21:03.217627 7fd779f84820 10 monclient(hunting):
init
-99> 2014-06-15 20:21:03.217634 7fd779f84820 5 adding auth protocol:
none
-98> 2014-06-15 20:21:03.217638 7fd779f84820 10 monclient(hunting):
auth_supported 1 method none
-97> 2014-06-15 20:21:03.217712 7fd779f84820 10 monclient(hunting):
_reopen_session rank -1 name
-96> 2014-06-15 20:21:03.217777 7fd779f84820 10 monclient(hunting):
picked mon.noname-a con 0x14d2310 addr 10.30.83.29:6789/0
-95> 2014-06-15 20:21:03.218043 7fd779f84820 10 monclient(hunting):
_send_mon_message to mon.noname-a at 10.30.83.29:6789/0
-94> 2014-06-15 20:21:03.218058 7fd779f84820 1 -- :/1008789 -->
10.30.83.29:6789/0 -- auth(proto 0 42 bytes epoch 0) v1 -- ?+0 0x14d27b0
con 0x14d2310
-93> 2014-06-15 20:21:03.218076 7fd779f84820 10 monclient(hunting):
renew_subs
-92> 2014-06-15 20:21:03.218082 7fd779f84820 10 monclient(hunting):
authenticate will time out at 2014-06-15 20:26:03.218082
-91> 2014-06-15 20:21:03.218834 7fd779f7c700 1 --
10.30.83.29:0/1008789 learned my addr 10.30.83.29:0/1008789
-90> 2014-06-15 20:21:03.220748 7fd770adb700 1 --
10.30.83.29:0/1008789 <== mon.0 10.30.83.29:6789/0 1 ==== mon_map v1
==== 199+0+0 (2303777899 0 0) 0x7fd764000ab0 con 0x14d2310
-89> 2014-06-15 20:21:03.220801 7fd770adb700 10 monclient(hunting):
handle_monmap mon_map v1
-88> 2014-06-15 20:21:03.220813 7fd770adb700 10 monclient(hunting):
got monmap 1, mon.noname-a is now rank -1
-87> 2014-06-15 20:21:03.220815 7fd770adb700 10 monclient(hunting):
dump:
epoch 1
fsid b7ab70e4-bbce-437d-8ab1-a8ae24ad1824
last_changed 2014-06-06 00:37:40.461800
created 2014-06-06 00:37:40.461800
0: 10.30.83.29:6789/0 mon.cephadmin
-86> 2014-06-15 20:21:03.221014 7fd770adb700 1 --
10.30.83.29:0/1008789 <== mon.0 10.30.83.29:6789/0 2 ====
auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (473450819 0 0)
0x7fd764000e50 con 0x14d2310
-85> 2014-06-15 20:21:03.221159 7fd770adb700 10 monclient(hunting):
my global_id is 6329
-84> 2014-06-15 20:21:03.221164 7fd770adb700 1 monclient(hunting):
found mon.cephadmin
-83> 2014-06-15 20:21:03.221167 7fd770adb700 10 monclient:
_send_mon_message to mon.cephadmin at 10.30.83.29:6789/0
-82> 2014-06-15 20:21:03.221173 7fd770adb700 1 --
10.30.83.29:0/1008789 --> 10.30.83.29:6789/0 --
mon_subscribe({monmap=0+}) v2 -- ?+0 0x14d2dc0 con 0x14d2310
-81> 2014-06-15 20:21:03.221730 7fd779f84820 5 monclient:
authenticate success, global_id 6329
-80> 2014-06-15 20:21:03.221756 7fd779f84820 5 asok(0x14c3070)
register_command objecter_requests hook 0x14d2c90
-79> 2014-06-15 20:21:03.221822 7fd779f84820 10 monclient:
renew_subs
-78> 2014-06-15 20:21:03.221827 7fd779f84820 10 monclient:
_send_mon_message to mon.cephadmin at 10.30.83.29:6789/0
-77> 2014-06-15 20:21:03.221833 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.83.29:6789/0 --
mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x14d3af0 con 0x14d2310
-76> 2014-06-15 20:21:03.221841 7fd779f84820 10 monclient:
renew_subs
-75> 2014-06-15 20:21:03.221843 7fd779f84820 10 monclient:
_send_mon_message to mon.cephadmin at 10.30.83.29:6789/0
-74> 2014-06-15 20:21:03.221845 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.83.29:6789/0 --
mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x14d41b0 con 0x14d2310
-73> 2014-06-15 20:21:03.221879 7fd779f84820 1 librados: init
done
-72> 2014-06-15 20:21:03.223484 7fd770adb700 1 --
10.30.83.29:0/1008789 <== mon.0 10.30.83.29:6789/0 3 ==== mon_map v1
==== 199+0+0 (2303777899 0 0) 0x7fd764000ab0 con 0x14d2310
-71> 2014-06-15 20:21:03.223501 7fd770adb700 10 monclient:
handle_monmap mon_map v1
-70> 2014-06-15 20:21:03.223508 7fd770adb700 10 monclient: got
monmap 1, mon.cephadmin is now rank 0
-69> 2014-06-15 20:21:03.223510 7fd770adb700 10 monclient: dump:
epoch 1
fsid b7ab70e4-bbce-437d-8ab1-a8ae24ad1824
last_changed 2014-06-06 00:37:40.461800
created 2014-06-06 00:37:40.461800
0: 10.30.83.29:6789/0 mon.cephadmin
2014-06-15 20:21:03.839153 7f4fe0dfa700 1 -- 10.30.83.29:0/1029095 -->
10.30.85.60:6800/18433 -- ping v1 -- ?+0 0x7f4fe40669c0 con 0xa2fa30
2014-06-15 20:21:03.839194 7f4fe0dfa700 1 -- 10.30.83.29:0/1029095 -->
10.30.85.60:6806/18981 -- ping v1 -- ?+0 0x7f4fe4068050 con 0xa32630
2014-06-15 20:21:03.839206 7f4fe0dfa700 1 -- 10.30.83.29:0/1029095 -->
10.30.85.60:6813/20066 -- ping v1 -- ?+0 0x7f4fe40685e0 con 0xa35460
2014-06-15 20:21:03.839216 7f4fe0dfa700 1 -- 10.30.83.29:0/1029095 -->
10.30.85.60:6810/19519 -- ping v1 -- ?+0 0x7f4fe4068800 con 0xa38b90
-68> 2014-06-15 20:21:03.223636 7fd770adb700 1 --
10.30.83.29:0/1008789 <== mon.0 10.30.83.29:6789/0 4 ====
mon_subscribe_ack(300s) v1 ==== 20+0+0 (549634071 0 0) 0x7fd764000ab0
con 0x14d2310
-67> 2014-06-15 20:21:03.223657 7fd770adb700 10 monclient:
handle_subscribe_ack sent 2014-06-15 20:21:03.218077 renew after
2014-06-15 20:23:33.218077
-66> 2014-06-15 20:21:03.223797 7fd770adb700 1 --
10.30.83.29:0/1008789 <== mon.0 10.30.83.29:6789/0 5 ====
osd_map(290..290 src has 1..290) v3 ==== 8602+0+0 (484351910 0 0)
0x7fd764002bf0 con 0x14d2310
-65> 2014-06-15 20:21:03.224990 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6800/18433 --
osd_op(client.6329.0:1 default.region [getxattrs,stat] 8.9a566808
ack+read e290) v4 -- ?+0 0x15055b0 con 0x1505210
-64> 2014-06-15 20:21:03.225330 7fd770adb700 1 --
10.30.83.29:0/1008789 <== mon.0 10.30.83.29:6789/0 6 ====
mon_subscribe_ack(300s) v1 ==== 20+0+0 (549634071 0 0) 0x7fd764000a30
con 0x14d2310
-63> 2014-06-15 20:21:03.225347 7fd770adb700 10 monclient:
handle_subscribe_ack sent 0.000000, ignoring
-62> 2014-06-15 20:21:03.225457 7fd770adb700 1 --
10.30.83.29:0/1008789 <== mon.0 10.30.83.29:6789/0 7 ====
osd_map(290..290 src has 1..290) v3 ==== 8602+0+0 (484351910 0 0)
0x7fd764002bc0 con 0x14d2310
-61> 2014-06-15 20:21:03.225567 7fd770adb700 1 --
10.30.83.29:0/1008789 <== mon.0 10.30.83.29:6789/0 8 ====
mon_subscribe_ack(300s) v1 ==== 20+0+0 (549634071 0 0) 0x7fd764000a30
con 0x14d2310
-60> 2014-06-15 20:21:03.225581 7fd770adb700 10 monclient:
handle_subscribe_ack sent 0.000000, ignoring
-59> 2014-06-15 20:21:03.225655 7fd76cdd4700 2
RGWDataChangesLog::ChangesRenewThread: start
-58> 2014-06-15 20:21:03.229355 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.1 10.30.85.60:6800/18433 1 ====
osd_op_reply(1 default.region [getxattrs,stat] v0'0 uv3 ondisk = 0) v6
==== 223+0+20 (3277333364 0 919087199) 0x7fd754000c40 con 0x1505210
-57> 2014-06-15 20:21:03.229570 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6800/18433 --
osd_op(client.6329.0:2 default.region [read 0~524288] 8.9a566808
ack+read e290) v4 -- ?+0 0x1504210 con 0x1505210
-56> 2014-06-15 20:21:03.232697 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.1 10.30.85.60:6800/18433 2 ====
osd_op_reply(2 default.region [read 0~17] v0'0 uv3 ondisk = 0) v6 ====
181+0+17 (291799883 0 2082767461) 0x7fd754000c40 con 0x1505210
-55> 2014-06-15 20:21:03.232927 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6806/18981 --
osd_op(client.6329.0:3 region_info.default [getxattrs,stat] 8.43611588
ack+read e290) v4 -- ?+0 0x15073e0 con 0x15043c0
-54> 2014-06-15 20:21:03.235726 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.2 10.30.85.60:6806/18981 1 ====
osd_op_reply(3 region_info.default [getxattrs,stat] v0'0 uv1 ondisk = 0)
v6 ==== 228+0+20 (880048796 0 2281270695) 0x1507980 con 0x15043c0
-53> 2014-06-15 20:21:03.235853 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6806/18981 --
osd_op(client.6329.0:4 region_info.default [read 0~524288] 8.43611588
ack+read e290) v4 -- ?+0 0x15073e0 con 0x15043c0
-52> 2014-06-15 20:21:03.237509 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.2 10.30.85.60:6806/18981 2 ====
osd_op_reply(4 region_info.default [read 0~145] v0'0 uv1 ondisk = 0) v6
==== 186+0+145 (3252816400 0 4143625220) 0x1507a10 con 0x15043c0
-51> 2014-06-15 20:21:03.237756 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6813/20066 --
osd_op(client.6329.0:5 zone_info.default [getxattrs,stat] 8.e01f9ae
ack+read e290) v4 -- ?+0 0x15092d0 con 0x1509010
-50> 2014-06-15 20:21:03.240799 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.0 10.30.85.60:6813/20066 1 ====
osd_op_reply(5 zone_info.default [getxattrs,stat] v0'0 uv3 ondisk = 0)
v6 ==== 226+0+20 (1392771561 0 1459014944) 0x7fd754000c40 con 0x1509010
-49> 2014-06-15 20:21:03.240990 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6813/20066 --
osd_op(client.6329.0:6 zone_info.default [read 0~524288] 8.e01f9ae
ack+read e290) v4 -- ?+0 0x15092d0 con 0x1509010
-48> 2014-06-15 20:21:03.242462 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.0 10.30.85.60:6813/20066 2 ====
osd_op_reply(6 zone_info.default [read 0~660] v0'0 uv3 ondisk = 0) v6
==== 184+0+660 (3906808835 0 1432975907) 0x7fd754000c40 con 0x1509010
-47> 2014-06-15 20:21:03.242623 7fd779f84820 2 zone default is
master
-46> 2014-06-15 20:21:03.242696 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6813/20066 --
osd_op(client.6329.0:7 region_map [getxattrs,stat] 8.bd31b503 ack+read
e290) v4 -- ?+0 0x150b1f0 con 0x1509010
-45> 2014-06-15 20:21:03.244047 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.0 10.30.85.60:6813/20066 3 ====
osd_op_reply(7 region_map [getxattrs,stat] v0'0 uv0 ack = -2 ((2) No
such file or directory)) v6 ==== 219+0+0 (2976023010 0 0) 0x7fd754000c00
con 0x1509010
-44> 2014-06-15 20:21:03.244378 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6813/20066 --
osd_op(client.6329.0:8 notify.0 [create 0~0] 4.4322fa9f ondisk+write
e290) v4 -- ?+0 0x1504890 con 0x1509010
-43> 2014-06-15 20:21:03.285522 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.0 10.30.85.60:6813/20066 4 ====
osd_op_reply(8 notify.0 [create 0~0] v290'4026 uv4026 ondisk = 0) v6
==== 175+0+0 (3806777381 0 0) 0x7fd754000c00 con 0x1509010
-42> 2014-06-15 20:21:03.285709 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6813/20066 --
osd_op(client.6329.0:9 notify.0 [watch add cookie 1 ver 0] 4.4322fa9f
ondisk+read+write e290) v4 -- ?+0 0x150b1f0 con 0x1509010
-41> 2014-06-15 20:21:03.319254 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.0 10.30.85.60:6813/20066 5 ====
osd_op_reply(9 notify.0 [watch add cookie 1 ver 0] v290'4027 uv4026
ondisk = 0) v6 ==== 175+0+0 (1566327965 0 0) 0x7fd754000c00 con
0x1509010
-40> 2014-06-15 20:21:03.320027 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6810/19519 --
osd_op(client.6329.0:10 notify.1 [create 0~0] 4.16dafda0 ondisk+write
e290) v4 -- ?+0 0x150b7d0 con 0x150b280
-39> 2014-06-15 20:21:03.367842 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.3 10.30.85.60:6810/19519 1 ====
osd_op_reply(10 notify.1 [create 0~0] v290'4026 uv4026 ondisk = 0) v6
==== 175+0+0 (2269688623 0 0) 0x7fd75c000b70 con 0x150b280
-38> 2014-06-15 20:21:03.368010 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6810/19519 --
osd_op(client.6329.0:11 notify.1 [watch add cookie 2 ver 0] 4.16dafda0
ondisk+read+write e290) v4 -- ?+0 0x150be80 con 0x150b280
-37> 2014-06-15 20:21:03.394700 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.3 10.30.85.60:6810/19519 2 ====
osd_op_reply(11 notify.1 [watch add cookie 2 ver 0] v290'4027 uv4026
ondisk = 0) v6 ==== 175+0+0 (2271015729 0 0) 0x7fd75c000b70 con
0x150b280
-36> 2014-06-15 20:21:03.394870 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6813/20066 --
osd_op(client.6329.0:12 notify.2 [create 0~0] 4.88aa5c95 ondisk+write
e290) v4 -- ?+0 0x150be80 con 0x1509010
-35> 2014-06-15 20:21:03.427602 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.0 10.30.85.60:6813/20066 6 ====
osd_op_reply(12 notify.2 [create 0~0] v290'4026 uv4026 ondisk = 0) v6
==== 175+0+0 (4160736164 0 0) 0x7fd754000c00 con 0x1509010
-34> 2014-06-15 20:21:03.427717 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6813/20066 --
osd_op(client.6329.0:13 notify.2 [watch add cookie 3 ver 0] 4.88aa5c95
ondisk+read+write e290) v4 -- ?+0 0x150c600 con 0x1509010
-33> 2014-06-15 20:21:03.469298 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.0 10.30.85.60:6813/20066 7 ====
osd_op_reply(13 notify.2 [watch add cookie 3 ver 0] v290'4027 uv4026
ondisk = 0) v6 ==== 175+0+0 (1641409143 0 0) 0x7fd754000c00 con
0x1509010
-32> 2014-06-15 20:21:03.469448 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6800/18433 --
osd_op(client.6329.0:14 notify.3 [create 0~0] 4.f8c99aee ondisk+write
e290) v4 -- ?+0 0x150c600 con 0x1505210
-31> 2014-06-15 20:21:03.502959 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.1 10.30.85.60:6800/18433 3 ====
osd_op_reply(14 notify.3 [create 0~0] v290'4023 uv4023 ondisk = 0) v6
==== 175+0+0 (2083021438 0 0) 0x7fd754000c00 con 0x1505210
-30> 2014-06-15 20:21:03.503140 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6800/18433 --
osd_op(client.6329.0:15 notify.3 [watch add cookie 4 ver 0] 4.f8c99aee
ondisk+read+write e290) v4 -- ?+0 0x150ce70 con 0x1505210
-29> 2014-06-15 20:21:03.544955 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.1 10.30.85.60:6800/18433 4 ====
osd_op_reply(15 notify.3 [watch add cookie 4 ver 0] v290'4024 uv4023
ondisk = 0) v6 ==== 175+0+0 (2920416415 0 0) 0x7fd754000c00 con
0x1505210
-28> 2014-06-15 20:21:03.545133 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6813/20066 --
osd_op(client.6329.0:16 notify.4 [create 0~0] 4.a204812d ondisk+write
e290) v4 -- ?+0 0x150cd80 con 0x1509010
-27> 2014-06-15 20:21:03.593755 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.0 10.30.85.60:6813/20066 8 ====
osd_op_reply(16 notify.4 [create 0~0] v290'4024 uv4024 ondisk = 0) v6
==== 175+0+0 (1091987275 0 0) 0x7fd754000c00 con 0x1509010
-26> 2014-06-15 20:21:03.593933 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6813/20066 --
osd_op(client.6329.0:17 notify.4 [watch add cookie 5 ver 0] 4.a204812d
ondisk+read+write e290) v4 -- ?+0 0x150d5f0 con 0x1509010
-25> 2014-06-15 20:21:03.619749 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.0 10.30.85.60:6813/20066 9 ====
osd_op_reply(17 notify.4 [watch add cookie 5 ver 0] v290'4025 uv4024
ondisk = 0) v6 ==== 175+0+0 (2911738149 0 0) 0x7fd754000c00 con
0x1509010
-24> 2014-06-15 20:21:03.619923 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6806/18981 --
osd_op(client.6329.0:18 notify.5 [create 0~0] 4.31099063 ondisk+write
e290) v4 -- ?+0 0x150d5f0 con 0x15043c0
-23> 2014-06-15 20:21:03.643621 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.2 10.30.85.60:6806/18981 3 ====
osd_op_reply(18 notify.5 [create 0~0] v290'4024 uv4024 ondisk = 0) v6
==== 175+0+0 (2183452954 0 0) 0x150db90 con 0x15043c0
-22> 2014-06-15 20:21:03.643803 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6806/18981 --
osd_op(client.6329.0:19 notify.5 [watch add cookie 6 ver 0] 4.31099063
ondisk+read+write e290) v4 -- ?+0 0x150e030 con 0x15043c0
-21> 2014-06-15 20:21:03.685620 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.2 10.30.85.60:6806/18981 4 ====
osd_op_reply(19 notify.5 [watch add cookie 6 ver 0] v290'4025 uv4024
ondisk = 0) v6 ==== 175+0+0 (3507614674 0 0) 0x150e430 con 0x15043c0
-20> 2014-06-15 20:21:03.685795 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6800/18433 --
osd_op(client.6329.0:20 notify.6 [create 0~0] 4.97c520d4 ondisk+write
e290) v4 -- ?+0 0x150dd90 con 0x1505210
-19> 2014-06-15 20:21:03.736082 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.1 10.30.85.60:6800/18433 5 ====
osd_op_reply(20 notify.6 [create 0~0] v290'4025 uv4025 ondisk = 0) v6
==== 175+0+0 (864969722 0 0) 0x7fd754000c00 con 0x1505210
-18> 2014-06-15 20:21:03.736264 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6800/18433 --
osd_op(client.6329.0:21 notify.6 [watch add cookie 7 ver 0] 4.97c520d4
ondisk+read+write e290) v4 -- ?+0 0x150e610 con 0x1505210
-17> 2014-06-15 20:21:03.770558 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.1 10.30.85.60:6800/18433 6 ====
osd_op_reply(21 notify.6 [watch add cookie 7 ver 0] v290'4026 uv4025
ondisk = 0) v6 ==== 175+0+0 (3468124297 0 0) 0x7fd754000c00 con
0x1505210
-16> 2014-06-15 20:21:03.770737 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6800/18433 --
osd_op(client.6329.0:22 notify.7 [create 0~0] 4.84ada7c9 ondisk+write
e290) v4 -- ?+0 0x150e430 con 0x1505210
-15> 2014-06-15 20:21:03.791459 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.1 10.30.85.60:6800/18433 7 ====
osd_op_reply(22 notify.7 [create 0~0] v290'4025 uv4025 ondisk = 0) v6
==== 175+0+0 (662567101 0 0) 0x7fd754000c00 con 0x1505210
-14> 2014-06-15 20:21:03.791613 7fd779f84820 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6800/18433 --
osd_op(client.6329.0:23 notify.7 [watch add cookie 8 ver 0] 4.84ada7c9
ondisk+read+write e290) v4 -- ?+0 0x150eca0 con 0x1505210
-13> 2014-06-15 20:21:03.813703 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.1 10.30.85.60:6800/18433 8 ====
osd_op_reply(23 notify.7 [watch add cookie 8 ver 0] v290'4026 uv4025
ondisk = 0) v6 ==== 175+0+0 (2424934418 0 0) 0x7fd754000c00 con
0x1505210
-12> 2014-06-15 20:21:03.815286 7fd779f84820 0 framework: fastcgi
-11> 2014-06-15 20:21:03.815305 7fd779f84820 0 starting handler:
fastcgi
-10> 2014-06-15 20:21:03.816726 7fd74e6f8700 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6800/18433 --
osd_op(client.6329.0:24 [pgls start_epoch 0] 7.0 ack+read e290) v4 --
?+0 0x7fd75800a630 con 0x1505210
-9> 2014-06-15 20:21:03.817584 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.1 10.30.85.60:6800/18433 9 ====
osd_op_reply(24 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v6 ====
167+0+44 (1244889235 0 139081063) 0x7fd754000ce0 con 0x1505210
-8> 2014-06-15 20:21:03.817665 7fd770adb700 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6806/18981 --
osd_op(client.6329.0:25 [pgls start_epoch 0] 7.1 ack+read e290) v4 --
?+0 0x7fd75800aae0 con 0x15043c0
-7> 2014-06-15 20:21:03.819356 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.2 10.30.85.60:6806/18981 5 ====
osd_op_reply(25 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v6 ====
167+0+44 (3347405639 0 139081063) 0x1512950 con 0x15043c0
-6> 2014-06-15 20:21:03.819509 7fd770adb700 1 --
10.30.83.29:0/1008789 --> 10.30.85.61:6800/28605 --
osd_op(client.6329.0:26 [pgls start_epoch 0] 7.2 ack+read e290) v4 --
?+0 0x7fd75800c3e0 con 0x7fd75800ac80
-5> 2014-06-15 20:21:03.819635 7fd74fafa700 2 garbage collection:
start
-4> 2014-06-15 20:21:03.819798 7fd74fafa700 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6806/18981 --
osd_op(client.6329.0:27 gc.21 [call lock.lock] 6.6dc01772 ondisk+write
e290) v4 -- ?+0 0x7fd7600022b0 con 0x15043c0
-3> 2014-06-15 20:21:03.823164 7fd770adb700 1 --
10.30.83.29:0/1008789 <== osd.2 10.30.85.60:6806/18981 6 ====
osd_op_reply(27 gc.21 [call] v0'0 uv0 ondisk = -16 ((16) Device or
resource busy)) v6 ==== 172+0+0 (3774749926 0 0) 0x1512950 con 0x15043c0
-2> 2014-06-15 20:21:03.823309 7fd74fafa700 0 RGWGC::process()
failed to acquire lock on gc.21
-1> 2014-06-15 20:21:03.823457 7fd74fafa700 1 --
10.30.83.29:0/1008789 --> 10.30.85.60:6810/19519 --
osd_op(client.6329.0:28 gc.22 [call lock.lock] 6.97748d0d ondisk+write
e290) v4 -- ?+0 0x7fd760002bc0 con 0x150b280
0> 2014-06-15 20:21:03.821327 7fd74d2f6700 -1 common/Thread.cc: In
function 'void Thread::create(size_t)' thread 7fd74d2f6700 time
2014-06-15 20:21:03.819948
common/Thread.cc: 110: FAILED assert(ret == 0)
ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
1: (Thread::create(unsigned long)+0x8a) [0x7fd77900363a]
2: (ThreadPool::start_threads()+0x12e) [0x7fd778fe770e]
3: (ThreadPool::start()+0x7a) [0x7fd778feaa9a]
4: (RGWFCGXProcess::run()+0x195) [0x4ae305]
5: /usr/bin/radosgw() [0x4b3bbe]
6: (()+0x79d1) [0x7fd7772929d1]
7: (clone()+0x6d) [0x7fd776fdfb5d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
1/ 1 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
5/ 5 rgw
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/httpd/client.radosgw.cephadmin.log
--- end dump of recent events ---
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
