Hi, We have a large 1PB ceph cluster. We recently added 6 nodes with 16 2TB disks each to the cluster. All the 5 nodes rebalanced well without any issues and the sixth/last node OSDs started acting weird as I increase weight of one osd the utilization doesn't change but a different osd on the same node utilization is getting increased. Rebalance complete fine but utilization is not right.
Increased weight of OSD 610 to 0.2 from 0.0 but utilization of OSD 611 started increasing but its weight is 0.0. If I increase weight of OSD 611 to 0.2 then its overall utilization is growing to what if its weight is 0.4. So if I increase weight of 610 and 615 to their full weight then utilization on OSD 610 is 1% and on OSD 611 is inching towards 100% where I had to stop and downsize the OSD's crush weight back to 0.0 to avoid any implications on ceph cluster. Its not just one osd but different OSD's on that one node. The only correlation I found out is 610 and 611 OSD Journal partitions are on the same SSD drive and all the OSDs are SAS drives. Any help on how to debug or resolve this will be helpful. Attached the screenshot. with shows 610, 612 and 620 osd crush weight is increased to 0.2 but OSDs 611, 615 and 623 utilization increased but has 0 crush weight. Thanks, Pardhiv K
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
