On 6/11/20 11:30 AM, Stephan wrote:
Hi Mark,
thanks for your comprehensive response!
Our tests are basically matching the linked results (we are testing with 2
OSDs/NVMe and fio/librbd too, but having a much smaller setup). Sometimes we
see smaller or higher improvements from Nautilus to Octupus but it is similar.
Only the random write iops are the other way round, namely a lot slower in our
setup …
Meanwhile we have gone through some more testing:
@1) Increasing osd_memory_target from the default (which ist 4GB as far as we
know) to 16GB doesn't change the results.
Ok, not likely due to onode cache misses then!
@2/3) The CPUs are configured for high performance in the BIOS and we also
ensured that it is set in the kernel as well (governor performance). Each node
in our test-setup has one Intel E2690-v3 with 12/24 cores/threads running
constantly at 3,1GHz.
Ok, that's good. FWIW this typically makes a fairly substantial
difference with high performance NVMe drives so it might be worth just
verifying that you are seeing an improvement vs letting the CPUs drop
into low power C states.
@4) Yes, we have tested bluefs_buffered_io without success. We did some
profiling using gdbpmp, collecting 100 samples shows that 0.5%-1% of the time
is spent in io_submit. There is an extrem performance impact when profiling
(reducing iops to several hundreds operations/second), therefore we are
uncertain if this is a relevant information. Can we improve the profiling (we
used gdbpmp.py -p … -n 100 -m bstore_kv_sync,bstore_kv_final -o … like in the
example on github)? We gladly provide the sample data collected if this could
be helpful. Furthermore we checked iostats, which seems to be okay (w_await
most times below 1).
Yeah, gdbpmp will have a a big effect on performance which in some cases
could affect the results (especially if you are profiling the client and
the osd at the same time). Having said that, the way it works is that
it periodically stops the process and looks at the sample of what it was
doing when it was stopped. Between those pauses it runs normally. So if
you end up with a bunch of samples all in lock contention there's
probably a decent chance that the OSD really is spending a lot of time
in lock contention during normal execution too. You can adjust how long
it sleeps between sample collection with the -s paramater if you want to
maximize the time between pauses (though it will make it take even
longer to gather samples).
@5) We have set noscrub and norebalance as well as disabled the automated
scaling of the pg count during all our tests.
As the results are reproducible when switching between Nautilus and Octopus,
there must clearly be something going on in Octopus. Maybe this only affects
very small setups like ours? As far as we see you have been testing with 8
nodes/64 NVMe total, where our setup only consists of 3 nodes with one NVMe
each.
It's possible, though even in small setups while we were testing master
post-nautilus we were seeing quite a bit better throughput. If you
could try a 1000 sample gdbpmp profile of one of your OSDs on octopus
(and even better another one on nautilus) that would be most helpful!
Please also include the benchmark command-line that was run if possible.
Thanks,
Mark
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
