Hello,
I currently have 5 servers within my CEPH Cluster
2 x (10 * 8TB Disks)
3 x (10 * 4TB Disks)
Currently seeing a larger difference in OSD use across the two separate server
types, as well as within the server itself.
For example on one 4TB server I have an OSD at 64% and one at 84%, where on the
8TB servers the OSD range from 49% to 64%, where the highest used OSD's are on
the 4TB.
Each drive has a weight set correctly for the drive size and each server has
the correct weight set, below is my crush map. Apart from running the command
to adjust the re-weight is there anything I am doing wrong or should change for
better spread of data, not looking for near perfect but where the 8TB drives
are sitting at 64% max and 4TB are sitting at 80%'s causes a big inbalance.
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54
# buckets
host sn1 {
id -2 # do not change unnecessarily
# weight 72.800
alg straw2
hash 0 # rjenkins1
item osd.0 weight 7.280
item osd.1 weight 7.280
item osd.3 weight 7.280
item osd.4 weight 7.280
item osd.2 weight 7.280
item osd.5 weight 7.280
item osd.6 weight 7.280
item osd.7 weight 7.280
item osd.8 weight 7.280
item osd.9 weight 7.280
}
host sn3 {
id -6 # do not change unnecessarily
# weight 72.800
alg straw2
hash 0 # rjenkins1
item osd.10 weight 7.280
item osd.11 weight 7.280
item osd.12 weight 7.280
item osd.13 weight 7.280
item osd.14 weight 7.280
item osd.15 weight 7.280
item osd.16 weight 7.280
item osd.17 weight 7.280
item osd.18 weight 7.280
item osd.19 weight 7.280
}
host sn4 {
id -7 # do not change unnecessarily
# weight 36.060
alg straw2
hash 0 # rjenkins1
item osd.20 weight 3.640
item osd.21 weight 3.640
item osd.22 weight 3.640
item osd.23 weight 3.640
item osd.24 weight 3.640
item osd.25 weight 3.640
item osd.26 weight 3.640
item osd.27 weight 3.640
item osd.28 weight 3.640
item osd.29 weight 3.300
}
host sn5 {
id -8 # do not change unnecessarily
# weight 36.060
alg straw2
hash 0 # rjenkins1
item osd.30 weight 3.640
item osd.31 weight 3.640
item osd.32 weight 3.640
item osd.33 weight 3.640
item osd.34 weight 3.640
item osd.35 weight 3.640
item osd.36 weight 3.640
item osd.37 weight 3.640
item osd.38 weight 3.640
item osd.39 weight 3.640
}
host sn6 {
id -9 # do not change unnecessarily
# weight 36.060
alg straw2
hash 0 # rjenkins1
item osd.40 weight 3.640
item osd.41 weight 3.640
item osd.42 weight 3.640
item osd.43 weight 3.640
item osd.44 weight 3.640
item osd.45 weight 3.640
item osd.46 weight 3.640
item osd.47 weight 3.640
item osd.48 weight 3.640
item osd.49 weight 3.640
}
root default {
id -1 # do not change unnecessarily
# weight 253.780
alg straw2
hash 0 # rjenkins1
item sn1 weight 72.800
item sn3 weight 72.800
item sn4 weight 36.060
item sn5 weight 36.060
item sn6 weight 36.060
}
# rules
rule replicated_ruleset {
ruleset 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
Thanks,
Ashley
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
