On Tue, 4 Jul 2023 at 10:00, Matthew Booth <mbo...@redhat.com> wrote: > > On Mon, 3 Jul 2023 at 18:33, Ilya Dryomov <idryo...@gmail.com> wrote: > > > > On Mon, Jul 3, 2023 at 6:58 PM Mark Nelson <mark.nel...@clyso.com> wrote: > > > > > > > > > On 7/3/23 04:53, Matthew Booth wrote: > > > > On Thu, 29 Jun 2023 at 14:11, Mark Nelson <mark.nel...@clyso.com> wrote: > > > >>>>> This container runs: > > > >>>>> fio --rw=write --ioengine=sync --fdatasync=1 > > > >>>>> --directory=/var/lib/etcd --size=100m --bs=8000 --name=etcd_perf > > > >>>>> --output-format=json --runtime=60 --time_based=1 > > > >>>>> > > > >>>>> And extracts sync.lat_ns.percentile["99.000000"] > > > >>>> Matthew, do you have the rest of the fio output captured? It would > > > >>>> be interesting to see if it's just the 99th percentile that is bad > > > >>>> or the PWL cache is worse in general. > > > >>> Sure. > > > >>> > > > >>> With PWL cache: https://paste.openstack.org/show/820504/ > > > >>> Without PWL cache: > > > >>> https://paste.openstack.org/show/b35e71zAwtYR2hjmSRtR/ > > > >>> With PWL cache, 'rbd_cache'=false: > > > >>> https://paste.openstack.org/show/byp8ZITPzb3r9bb06cPf/ > > > >> > > > >> Also, how's the CPU usage client side? I would be very curious to see > > > >> if unwindpmp shows anything useful (especially lock contention): > > > >> > > > >> > > > >> https://github.com/markhpc/uwpmp > > > >> > > > >> > > > >> Just attach it to the client-side process and start out with something > > > >> like 100 samples (more are better but take longer). You can run it > > > >> like: > > > >> > > > >> > > > >> ./unwindpmp -n 100 -p <pid> > > > > I've included the output in this gist: > > > > https://gist.github.com/mdbooth/2d68b7e081a37e27b78fe396d771427d > > > > > > > > That gist contains 4 runs: 2 with PWL enabled and 2 without, and also > > > > a markdown file explaining the collection method. > > > > > > > > Matt > > > > > > > > > Thanks Matt! I looked through the output. Looks like the symbols might > > > have gotten mangled. I'm not an expert on the RBD client, but I don't > > > think we would really be calling into > > > rbd_group_snap_rollback_with_progress from > > > librbd::cache::pwl::ssd::WriteLogEntry::writeback_bl. Was it possible > > > you used the libdw backend for unwindpmp? libdw sometimes gives > > > strange/mangled callgraphs, but I haven't seen it before with > > > libunwind. Hopefully Congmin Yin or Ilya can confirm if it's garbage. > > > > > > > > So with that said, assuming we can trust these callgraphs at all, it > > > looks like it might be worth looking at the latency of the > > > AbstractWriteLog, librbd::cache::pwl::ssd::WriteLogEntry::writeback_bl, > > > and possibly usage of librados::v14_2_0::IoCtx::object_list. On the > > > > Hi Mark, > > > > Both rbd_group_snap_rollback_with_progress and > > librados::v14_2_0::IoCtx::object_list entries don't make sense to me, > > so I'd say it's garbage. > > Unfortunately I'm not at all familiar with this tool. Do you know how > it obtains its symbols? I didn't install any debuginfo packages, so I > was a bit surprised to see any symbols at all.
I installed the following debuginfo packages and re-ran the tests: elfutils-debuginfod-client-0.189-2.fc38.x86_64 elfutils-debuginfod-client-devel-0.189-2.fc38.x86_64 ceph-debuginfo-17.2.6-3.fc38.x86_64 librbd1-debuginfo-17.2.6-3.fc38.x86_64 librados2-debuginfo-17.2.6-3.fc38.x86_64 qemu-debuginfo-7.2.1-2.fc38.x86_64 qemu-system-x86-core-debuginfo-7.2.1-2.fc38.x86_64 boost-debuginfo-1.78.0-11.fc38.x86_64 Note that unwindpmp now runs considerably slower (because it re-reads debug symbols for each sample?), so I had to reduce the number of samples to 500. I have updated the gist with the new results: https://gist.github.com/mdbooth/2d68b7e081a37e27b78fe396d771427d Thanks, Matt -- Matthew Booth _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
