We've encountered this problem a lot. As far as I know the best practice should
be making the distribution of PG across OSDs as even as you can after you
create the pool and before you write any data.
1. the disk utilization = (PGs per OSD) * (files per PG). Ceph is good at
making (files per PG
I think these numbers are about what is expected. You could try a couple
things to improve it, but neither of them are common:
1) increase the number of PGs (and pgp_num) a lot more. I you decide to
experiment with this, watch your CPU and memory numbers carefully.
2) try to correct for the inequ