Are these dual-socket machines? Perhaps NUMA is involved? On Thu., Jun. 27, 2019, 4:56 a.m. Lars Täuber, <taeu...@bbaw.de> wrote:
> Hi! > > In our cluster I ran some benchmarks. > The results are always similar but strange to me. > I don't know what the results mean. > The cluster consists of 7 (nearly) identical hosts for osds. Two of them > have one an additional hdd. > The hdds are from identical type. The ssds for the journal and wal are of > identical type. The configuration is identical (ssd-db-lv-size) for each > osd. > The hosts are connected the same way to the same switches. > This nautilus cluster was set up with ceph-ansible 4.0 on debian buster. > > This are the results of > # ceph --format plain tell osd.* bench > > osd.0: bench: wrote 1 GiB in blocks of 4 MiB in 15.0133 sec at 68 MiB/sec > 17 IOPS > osd.1: bench: wrote 1 GiB in blocks of 4 MiB in 6.98357 sec at 147 MiB/sec > 36 IOPS > osd.2: bench: wrote 1 GiB in blocks of 4 MiB in 6.80336 sec at 151 MiB/sec > 37 IOPS > osd.3: bench: wrote 1 GiB in blocks of 4 MiB in 12.0813 sec at 85 MiB/sec > 21 IOPS > osd.4: bench: wrote 1 GiB in blocks of 4 MiB in 8.51311 sec at 120 MiB/sec > 30 IOPS > osd.5: bench: wrote 1 GiB in blocks of 4 MiB in 6.61376 sec at 155 MiB/sec > 38 IOPS > osd.6: bench: wrote 1 GiB in blocks of 4 MiB in 14.7478 sec at 69 MiB/sec > 17 IOPS > osd.7: bench: wrote 1 GiB in blocks of 4 MiB in 12.9266 sec at 79 MiB/sec > 19 IOPS > osd.8: bench: wrote 1 GiB in blocks of 4 MiB in 15.2513 sec at 67 MiB/sec > 16 IOPS > osd.9: bench: wrote 1 GiB in blocks of 4 MiB in 9.26225 sec at 111 MiB/sec > 27 IOPS > osd.10: bench: wrote 1 GiB in blocks of 4 MiB in 13.6641 sec at 75 MiB/sec > 18 IOPS > osd.11: bench: wrote 1 GiB in blocks of 4 MiB in 13.8943 sec at 74 MiB/sec > 18 IOPS > osd.12: bench: wrote 1 GiB in blocks of 4 MiB in 13.235 sec at 77 MiB/sec > 19 IOPS > osd.13: bench: wrote 1 GiB in blocks of 4 MiB in 10.4559 sec at 98 MiB/sec > 24 IOPS > osd.14: bench: wrote 1 GiB in blocks of 4 MiB in 12.469 sec at 82 MiB/sec > 20 IOPS > osd.15: bench: wrote 1 GiB in blocks of 4 MiB in 17.434 sec at 59 MiB/sec > 14 IOPS > osd.16: bench: wrote 1 GiB in blocks of 4 MiB in 11.7184 sec at 87 MiB/sec > 21 IOPS > osd.17: bench: wrote 1 GiB in blocks of 4 MiB in 12.8702 sec at 80 MiB/sec > 19 IOPS > osd.18: bench: wrote 1 GiB in blocks of 4 MiB in 20.1894 sec at 51 MiB/sec > 12 IOPS > osd.19: bench: wrote 1 GiB in blocks of 4 MiB in 9.60049 sec at 107 > MiB/sec 26 IOPS > osd.20: bench: wrote 1 GiB in blocks of 4 MiB in 15.0613 sec at 68 MiB/sec > 16 IOPS > osd.21: bench: wrote 1 GiB in blocks of 4 MiB in 17.6074 sec at 58 MiB/sec > 14 IOPS > osd.22: bench: wrote 1 GiB in blocks of 4 MiB in 16.39 sec at 62 MiB/sec > 15 IOPS > osd.23: bench: wrote 1 GiB in blocks of 4 MiB in 15.2747 sec at 67 MiB/sec > 16 IOPS > osd.24: bench: wrote 1 GiB in blocks of 4 MiB in 10.2462 sec at 100 > MiB/sec 24 IOPS > osd.25: bench: wrote 1 GiB in blocks of 4 MiB in 13.5297 sec at 76 MiB/sec > 18 IOPS > osd.26: bench: wrote 1 GiB in blocks of 4 MiB in 7.46824 sec at 137 > MiB/sec 34 IOPS > osd.27: bench: wrote 1 GiB in blocks of 4 MiB in 11.2216 sec at 91 MiB/sec > 22 IOPS > osd.28: bench: wrote 1 GiB in blocks of 4 MiB in 16.6205 sec at 62 MiB/sec > 15 IOPS > osd.29: bench: wrote 1 GiB in blocks of 4 MiB in 10.1477 sec at 101 > MiB/sec 25 IOPS > > > The different runs differ by ±1 IOPS. > Why are the osds 1,2,4,5,9,19,26 faster than the others? > > Restarting an osd did change the result. > > Could someone give me hint where to look further to find the reason? > > Thanks > Lars > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
