Hi,
I'm wondering why slow requests are being reported mainly when the request
has been put into the queue for processing by its PG (queued_for_pg ,
http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/#debugging-slow-request
).
Could it be due too low pg_num/pgp_num ?
It looks that slow requests are mainly addressed to
default.rgw.buckets.data (pool id 20) , volumes (pool id 3) and
default.rgw.buckets.index (pool id 14)
2018-01-31 12:06:55.899557 osd.59 osd.59 10.212.32.22:6806/4413 38 :
cluster [WRN] slow request 30.125793 seconds old, received at 2018-01-31
12:06:25.773675: osd_op(client.857003.0:126171692 3.a4fec1ad 3.a4fec1ad
(undecoded) ack+ondisk+write+known_if_redirected e5722) currently
queued_for_pg
Btw how can I get more human-friendly client information from log entry
like above ?
Current pg_num/pgp_num
pool 3 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool
stripe_width 0 application rbd
removed_snaps [1~3]
pool 14 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule
0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 20 'default.rgw.buckets.data' erasure size 9 min_size 6 crush_rule 1
object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags
hashpspool stripe_width 4224 application rgw
Usage
GLOBAL:
SIZE AVAIL RAW USED %RAW USED OBJECTS
385T 144T 241T 62.54 31023k
POOLS:
NAME ID QUOTA OBJECTS QUOTA BYTES
USED %USED MAX AVAIL OBJECTS DIRTY READ
WRITE RAW USED
volumes 3 N/A N/A
40351G 70.91 16557G 10352314 10109k 2130M
2520M 118T
default.rgw.buckets.index 14 N/A N/A
0 0 16557G 205 205 160M
27945k 0
default.rgw.buckets.data 20 N/A N/A
79190G 70.51 33115G 20865953 20376k 122M
113M 116T
# ceph osd pool ls detail
pool 0 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins
pg_num 64 pgp_num 64 last_change 4502 flags hashpspool stripe_width 0
application rbd
pool 1 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins
pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool stripe_width 0
application rbd
pool 2 'images' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 512 pgp_num 512 last_change 5175 flags hashpspool
stripe_width 0 application rbd
removed_snaps [1~7,14~2]
pool 3 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool
stripe_width 0 application rbd
removed_snaps [1~3]
pool 4 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool stripe_width
0 application rgw
pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 6 'default.rgw.data.root' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 7 'default.rgw.gc' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 8 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 9 'default.rgw.users.uid' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 10 'default.rgw.usage' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 11 'default.rgw.users.email' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 owner
18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 12 'default.rgw.users.keys' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 owner
18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 13 'default.rgw.users.swift' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 14 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule
0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 15 'default.rgw.buckets.data.old' replicated size 3 min_size 2
crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 4502
flags hashpspool stripe_width 0 application rgw
pool 16 'default.rgw.buckets.non-ec' replicated size 3 min_size 2
crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags
hashpspool stripe_width 0 application rgw
pool 17 'default.rgw.buckets.extra' replicated size 3 min_size 2 crush_rule
0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 18 '.rgw.buckets.extra' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 20 'default.rgw.buckets.data' erasure size 9 min_size 6 crush_rule 1
object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags
hashpspool stripe_width 4224 application rgw
pool 21 'benchmark_replicated' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 4550 flags
hashpspool stripe_width 0 application rbd
removed_snaps [1~3]
pool 22 'benchmark_erasure_coded' erasure size 9 min_size 7 crush_rule 1
object_hash rjenkins pg_num 32 pgp_num 32 last_change 4552 flags hashpspool
stripe_width 24576 application rbd
removed_snaps [1~3]
# ceph df detail
GLOBAL:
SIZE AVAIL RAW USED %RAW USED OBJECTS
385T 144T 241T 62.54 31023k
POOLS:
NAME ID QUOTA OBJECTS QUOTA BYTES
USED %USED MAX AVAIL OBJECTS DIRTY READ
WRITE RAW USED
rbd 0 N/A N/A
0 0 16557G 0 0 1
134k 0
vms 1 N/A N/A
0 0 16557G 0 0 0
0 0
images 2 N/A N/A
7659M 0.05 16557G 1022 1022 51247
5668 22977M
volumes 3 N/A N/A
40351G 70.91 16557G 10352314 10109k 2130M
2520M 118T
.rgw.root 4 N/A N/A
1588 0 16557G 4 4 90
4 4764
default.rgw.control 5 N/A N/A
0 0 16557G 8 8 0
0 0
default.rgw.data.root 6 N/A N/A
93943 0 16557G 336 336 239k
6393 275k
default.rgw.gc 7 N/A N/A
0 0 16557G 32 32 1773M
5281k 0
default.rgw.log 8 N/A N/A
0 0 16557G 185 185 22404k
14936k 0
default.rgw.users.uid 9 N/A N/A
3815 0 16557G 15 15 187k
53303 11445
default.rgw.usage 10 N/A N/A
0 0 16557G 7 7 278k
556k 0
default.rgw.users.email 11 N/A N/A
58 0 16557G 3 3 0
3 174
default.rgw.users.keys 12 N/A N/A
177 0 16557G 10 10 262
22 531
default.rgw.users.swift 13 N/A N/A
40 0 16557G 3 3 0
3 120
default.rgw.buckets.index 14 N/A N/A
0 0 16557G 205 205 160M
27945k 0
default.rgw.buckets.data.old 15 N/A N/A
668G 3.88 16557G 180867 176k 707k
2318k 2004G
default.rgw.buckets.non-ec 16 N/A N/A
0 0 16557G 114 114 17960
12024 0
default.rgw.buckets.extra 17 N/A N/A
0 0 16557G 0 0 0
0 0
.rgw.buckets.extra 18 N/A N/A
0 0 16557G 0 0 0
0 0
default.rgw.buckets.data 20 N/A N/A
79190G 70.51 33115G 20865953 20376k 122M
113M 116T
benchmark_replicated 21 N/A N/A
1415G 7.88 16557G 363800 355k 1338k
1251k 4247G
benchmark_erasure_coded 22 N/A N/A
11057M 0.03 33115G 2761 2761 398
5520 16586M
Thanks
Jakub
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
