From: Omar Sandoval <osan...@fb.com> This is v2 of my patch series optimizing debuginfod for kernel debuginfo. v1 is here [7].
The main change from v1 is reworking the database changes to be backward compatible and therefore not require reindexing. Patch 1 is a preparatory refactor. Patch 2 makes the schema changes. Patch 3 implements the seekable xz extraction. Patch 4 populates the table of seekable entries at scan time. Patch 5 does it for pre-existing files at request time. Here is the background copied and pasted from v1: drgn [1] currently uses debuginfod with great success for debugging userspace processes. However, for debugging the Linux kernel (drgn's main use case), we have had some performance issues with debuginfod, so we intentionally avoid using it. Specifically, it sometimes takes over a minute for debuginfod to respond to queries for vmlinux and kernel modules (not including the actual download time). The reason for the slowness is that Linux kernel debuginfo packages are very large and contain lots of files. To respond to a query for a Linux kernel debuginfo file, debuginfod has to decompress and iterate through the whole package until it finds that file. If the file is towards the end of the package, this can take a very long time. This was previously reported for vdso files [2][3], which debuginfod was able to mitigate with improved caching and prefetching. However, kernel modules are far greater in number, vary drastically by hardware and workload, and can be spread all over the package, so in practice I've still been seeing long delays. This was also discussed on the drgn issue tracker [4]. The fundamental limitation is that Linux packages, which are essentially compressed archives with extra metadata headers, don't support random access to specific files. However, the multi-threaded xz compression format does actually support random access. And, luckily, the kernel debuginfo packages on Fedora, Debian, and Ubuntu all happen to use multi-threaded xz compression! debuginfod can take advantage of this: when it scans a package, if it is a seekable xz archive, it can save the uncompressed offset and size of each file. Then, when it needs a file, it can seek to that offset and extract it from there. This requires some understanding of the xz format and low-level liblzma code, but the speedup is massive: where the worst case was previously about 50 seconds just to find a file in a kernel debuginfo package, with this change the worst case is 0.25 seconds, a ~200x improvement! This works for both .rpm and .deb files. I tested this by requesting and verifying the digest of every file from a few kernel debuginfo rpms and debs [5]. P.S. The biggest downside of this change is that it depends on a very specific compression format that is only used by kernel packages incidentally. I think this is something we should formalize with Linux distributions: large debuginfo packages should use a seekable format. Currently, xz in multi-threaded mode is the only option, but Zstandard also has an experimental seekable format that is worth looking into [6]. Thanks, Omar 1: https://github.com/osandov/drgn 2: https://sourceware.org/bugzilla/show_bug.cgi?id=29478 3: https://bugzilla.redhat.com/show_bug.cgi?id=1970578 4: https://github.com/osandov/drgn/pull/380 5: https://gist.github.com/osandov/89d521fdc6c9a07aa8bb0ebf91974346 6: https://github.com/facebook/zstd/tree/dev/contrib/seekable_format 7: https://sourceware.org/pipermail/elfutils-devel/2024q3/007191.html Omar Sandoval (5): debuginfod: factor out common code for responding from an archive debugifod: add new table and views for seekable archives debuginfod: optimize extraction from seekable xz archives debuginfod: populate _r_seekable on scan debuginfod: populate _r_seekable on request configure.ac | 5 + debuginfod/Makefile.am | 2 +- debuginfod/debuginfod.cxx | 921 ++++++++++++++++++++++++++++++++------ 3 files changed, 788 insertions(+), 140 deletions(-) -- 2.45.2
