rep3datapool pg num is 512, Average number of PG replicas per OSD is 139
scrubs, balancer and pg autoscaler was disabled
RAM is 128G, swap is 0
From: Anthony D'Atri
Date: 2024-10-30 12:03
To: Louisa
CC: ceph-users
Subject: Re: [ceph-users] why performance difference between 'rados bench seq'
and 'rados bench rand' quite significant
The good Mr. Nelson and others may have more to contribute, but a few thoughts:
* Running for 60 or 120 seconds isn’t quantitative: rados bench typically
exhibits a clear ramp-up; watch the per-second stats.
* Suggest running for 10 minutes, three times in a row and averaging the results
* How many PGs in rep3datapool? Average number of PG replicas per OSD shown by
`ceph osd df` ? I would shoot for 150 - 200 in your case.
* Disable scrubs, the balancer, and the pg autoscaler during benching.
* If you have OS swap configured, disable it and reboot. How much RAM?
> On Oct 29, 2024, at 11:43 PM, Louisa <lushash...@126.com> wrote:
>
> Hi all,
> We used 'rados bench' to test 4k object read and write operations.
> Our cluster is pacific, one node, 11 bluestore osd ,db and wal share the
> block device. Block device is HDD.
>
> 1. testing 4k write with command 'rados bench 120 write -t 16 -b 4K -p
> rep3datapool --run-name 4kreadwrite --no-cleanup'
>
> 2. Before tesing 4k reads, we restarted all OSD daemons. The perfomance of
> 'rados bench 120 seq -t 16 -p rep3datapool --run-name 4kreadwrite' was very
> good, which Average IOPS: 17735;
> using 'ceph daemon osd.1 perf dump rocksdb' , we found the
> rocksdb:get_latency avgcount: 15189, avgtime: 0.000012947 (12.9us)
>
> 3. Before tesing 4k rand reads, we restarted all OSD daemons. 'rados bench
> 60 rand -t 16 -p rep3datapool --run-name 4kreadwrite' average IOPS: 2071
> rocksdb:get_latency avgcount: 8756, avgtime: 0.001761293 (1.7ms)
>
> Q1: Why performance difference between 'rados bench seq' and 'rados bench
> rand' quite significant? How to explain the rocksdb get_latency perfomance
> between this two scenario?
>
> 4. We write 40w 4k object to the pool, restarted all OSD daemons. running
> 'rados bench 120 seq -t 16 -p rep3datapool --run-name 4kreadwrite' again.
> Average IOPS~= 2000.
> rocsdb:get_latency avgtime also reached milliseconds level
> Q2: Why 'rados bench seq' performance decresing extremly after writing some
> more 4k object to the pool?
>
> Q3: Is there any methods and suggestions to optimized the read performance of
> this scenario under this hardware configuration.
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
