Re: [PATCH 0/3] debuginfod: speed up extraction from kernel debuginfo packages by 200x
Hi, Omar - Thanks. I wish this sort of amazing kludge weren't necessary, but given that it helps, so be it. I'd like to commend you on the effort needed to match your code up with the stylistic idiosyncracies of the debuginfod c++ code. It looks just like the other code. My only reservation is the schema change. Reindexing some of our large repos takes WEEKS. Here's a possible way to avoid that: - Preserve the current BUILDID schema id and tables as is. - Add a new table for the intra-archive coordinates. Think of it like a cache. Index it with archive-file-name and content-file-name (source0, source1 IIRC). - During a fetch out of the archive-file-name, check whether the new table has a record for that file. If yes, cache hit, go through to the xz extraction stuff, winner! - If not, try the is_seekable() check on the archive. If it is true, we have an archive that should be seekable, but we don't have it in the intra-archive cache. So take this opportunity to index that archive (only), populate the cache table, as the archive is being extracted. (No need to use the new cache data then, since we've just paid the effort of decompressing/reading the whole thing already.) - Need to confirm that during grooming, a disappeared archive-file-name would also drop the corresponding intra-archive rows. - Heck, during grooming or scanning, maybe the tool could preemptively do the intra-archive coordinate cache thing if it's not already done, just to defeat the latency of doing it on demand. What do you think? - FChE
[PATCH] libdwfl: Make dwfl_report_offline_memory work with ELF_C_READ_MMAP
elf_memory open mode recently changed from ELF_C_READ to
ELF_C_READ_MMAP. This broken dwfl_report_offline_memory that changes
mode to ELF_C_READ_MMAP_PRIVATE to be compatible with subsequent
elf_begin on embedded ELF files.
The proper implementation of dwfl_report_offline_memory doesn't change
open mode and subsequent elf_begin invocations simply use cmd from the
reference Elf*.
Add tests to exercise Elf* to trigger the bug caused by incorrect cmd
set to Elf*.
* libdwfl/offline.c (process_archive): Use archive->cmd
instead of hardcoded ELF_C_READ_MMAP_PRIVATE.
* libdwfl/open.c (libdw_open_elf): Use elf->cmd instead of
hardcoded ELF_C_READ_MMAP_PRIVATE.
(__libdw_open_elf_memory): Don't override (*elfp)->cmd.
* tests/Makefile.am (dwfl_report_offline_memory): Add libelf
as dependency.
* tests/dwfl-report-offline-memory.c: Add count_sections to
exercise Elf* from dwfl_report_offline_memory.
* tests/run-dwfl-report-offline-memory.sh: Add expected number
of sections to test invocations.
Signed-off-by: Aleksei Vetrov
---
libdwfl/offline.c | 5 ++--
libdwfl/open.c | 4 +---
tests/Makefile.am | 2 +-
tests/dwfl-report-offline-memory.c | 32 ++---
tests/run-dwfl-report-offline-memory.sh | 6 ++---
5 files changed, 37 insertions(+), 12 deletions(-)
diff --git a/libdwfl/offline.c b/libdwfl/offline.c
index e9ab0cc1..24e9e180 100644
--- a/libdwfl/offline.c
+++ b/libdwfl/offline.c
@@ -32,6 +32,7 @@
# include
#endif
+#include "libelfP.h"
#include "libdwflP.h"
#include
@@ -254,7 +255,7 @@ process_archive (Dwfl *dwfl, const char *name, const char
*file_name, int fd,
{
Dwfl_Module *mod = NULL;
/* elf_begin supports opening archives even with fd == -1 passed. */
- Elf *member = elf_begin (fd, ELF_C_READ_MMAP_PRIVATE, archive);
+ Elf *member = elf_begin (fd, archive->cmd, archive);
if (unlikely (member == NULL)) /* Empty archive. */
{
__libdwfl_seterrno (DWFL_E_BADELF);
@@ -263,7 +264,7 @@ process_archive (Dwfl *dwfl, const char *name, const char
*file_name, int fd,
while (process_archive_member (dwfl, name, file_name, predicate,
fd, member, &mod) != ELF_C_NULL)
-member = elf_begin (fd, ELF_C_READ_MMAP_PRIVATE, archive);
+member = elf_begin (fd, archive->cmd, archive);
/* We can drop the archive Elf handle even if we're still using members
in live modules. When the last module's elf_end on a member returns
diff --git a/libdwfl/open.c b/libdwfl/open.c
index d0f357ed..43b29fa9 100644
--- a/libdwfl/open.c
+++ b/libdwfl/open.c
@@ -151,7 +151,7 @@ libdw_open_elf (int *fdp, Elf **elfp, bool close_on_fail,
bool archive_ok,
elf->state.ar.elf_ar_hdr.ar_name = "libdwfl is faking you out";
elf->state.ar.elf_ar_hdr.ar_size = elf->maximum_size - offset;
elf->state.ar.offset = offset - sizeof (struct ar_hdr);
- Elf *subelf = elf_begin (-1, ELF_C_READ_MMAP_PRIVATE, elf);
+ Elf *subelf = elf_begin (-1, elf->cmd, elf);
elf->kind = ELF_K_NONE;
if (unlikely (subelf == NULL))
error = DWFL_E_LIBELF;
@@ -215,8 +215,6 @@ __libdw_open_elf_memory (char *data, size_t size, Elf
**elfp, bool archive_ok)
{
return DWFL_E_LIBELF;
}
- /* Allow using this ELF as reference for subsequent elf_begin calls. */
- (*elfp)->cmd = ELF_C_READ_MMAP_PRIVATE;
return libdw_open_elf (&fd, elfp, false, archive_ok, true, false, true);
}
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 77f9b90d..cfed54b7 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -792,7 +792,7 @@ test_elf_cntl_gelf_getshdr_LDADD = $(libelf)
dwflsyms_LDADD = $(libdw) $(libelf) $(argp_LDADD)
dwfllines_LDADD = $(libeu) $(libdw) $(libelf) $(argp_LDADD)
dwfl_report_elf_align_LDADD = $(libeu) $(libdw)
-dwfl_report_offline_memory_LDADD = $(libeu) $(libdw)
+dwfl_report_offline_memory_LDADD = $(libeu) $(libdw) $(libelf)
dwfl_report_segment_contiguous_LDADD = $(libdw) $(libebl) $(libelf)
varlocs_LDADD = $(libeu) $(libdw) $(libelf) $(argp_LDADD)
backtrace_LDADD = $(libeu) $(libdw) $(libelf) $(argp_LDADD)
diff --git a/tests/dwfl-report-offline-memory.c
b/tests/dwfl-report-offline-memory.c
index b3b4d9bd..3ecb66b9 100644
--- a/tests/dwfl-report-offline-memory.c
+++ b/tests/dwfl-report-offline-memory.c
@@ -18,14 +18,19 @@
#include
#include
+#include
+#include
#include
#include
#include
#include
#include
+#include
#include
+
#include ELFUTILS_HEADER(dwfl)
-#include "system.h"
+#include ELFUTILS_HEADER(elf)
+#include
static const Dwfl_Callbacks offline_callbacks =
@@ -45,6 +50,20 @@ count_modules (Dwfl_Module *mod __attribute__ ((unused)),
return DWARF_CB_OK;
}
+static int
+count_sections (Elf *elf)
+{
+ int result = 0;
+ Elf_Scn *section = NULL;
+ G
Re: [PATCH 0/3] debuginfod: speed up extraction from kernel debuginfo packages by 200x
On Thu, Jul 11, 2024 at 04:16:25PM -0400, Frank Ch. Eigler wrote: > Hi, Omar - > > Thanks. I wish this sort of amazing kludge weren't necessary, but > given that it helps, so be it. > > I'd like to commend you on the effort needed to match your code up > with the stylistic idiosyncracies of the debuginfod c++ code. It > looks just like the other code. My only reservation is the schema > change. Reindexing some of our large repos takes WEEKS. Here's a > possible way to avoid that: > > - Preserve the current BUILDID schema id and tables as is. > > - Add a new table for the intra-archive coordinates. Think of it like a > cache. > Index it with archive-file-name and content-file-name (source0, source1 > IIRC). > > - During a fetch out of the archive-file-name, check whether the new > table has a record for that file. If yes, cache hit, go through to > the xz extraction stuff, winner! > > - If not, try the is_seekable() check on the archive. If it is true, we have > an > archive that should be seekable, but we don't have it in the intra-archive > cache. > So take this opportunity to index that archive (only), populate the cache > table, > as the archive is being extracted. (No need to use the new cache data > then, since > we've just paid the effort of decompressing/reading the whole thing > already.) > > - Need to confirm that during grooming, a disappeared > archive-file-name would also drop the corresponding intra-archive > rows. > > - Heck, during grooming or scanning, maybe the tool could preemptively > do the intra-archive coordinate cache thing if it's not already > done, just to defeat the latency of doing it on demand. > > > What do you think? Hi, Frank, I didn't realize how expensive reindexing could be, thank you for pointing that out. Your proposal makes sense to me, I'll rework this. Thanks, Omar
[COMMITTED] tests/run-sysroot.sh: Avoid testing output that depends on LZMA support
run-sysroot.sh checks whether a backtrace generated by eu-stack contains
symbol names found in binaries under a test sysroot. Two frames in
the backtrace contain symbol names that must be read from .gnu_debugdata.
However this section can only be read if elfutils was built with LZMA
support. If not, then the symbol names will be absent from the
backtrace.
Test the eu-stack output with these 2 frames removed in order to prevent
a test failure when LZMA support is missing.
Signed-off-by: Aaron Merey
---
tests/run-sysroot.sh | 11 ++-
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/tests/run-sysroot.sh b/tests/run-sysroot.sh
index 7f7d3f06..cfdc51e9 100755
--- a/tests/run-sysroot.sh
+++ b/tests/run-sysroot.sh
@@ -17,8 +17,6 @@
. $srcdir/test-subr.sh
-set -ex;
-
tmpdir="$(mktemp -d)"
trap "rm -rf -- ${tmpdir}" EXIT
@@ -28,13 +26,16 @@ tar xjf "${abs_srcdir}/testfile-sysroot.tar.bz2" -C
"${tmpdir}"
testrun "${abs_top_builddir}"/src/stack --core "${tmpdir}/core.bash" \
--sysroot "${tmpdir}/sysroot" >"${tmpdir}/stack.out"
+# Remove 2 stack frames with symbol names contained in .gnu_debugdata.
+# Whether or not these names appear in the output depends on if elfutils
+# was built with LZMA support.
+sed -i '4,5d' "${tmpdir}/stack.out"
+
# check that we are able to get fully symbolized backtrace
-testrun diff "${tmpdir}/stack.out" - <<\EOF
+testrun_compare cat "${tmpdir}/stack.out" <<\EOF
PID 431185 - core
TID 431185:
#0 0x8ebe5a8c kill
-#1 0xe5663f20 kill_shell
-#2 0xe5667a98 termsig_handler.part.0
#3 0xe562b2fc execute_command
#4 0xe561cbb4 reader_loop
#5 0xe5611bf0 main
--
2.45.2
[COMMITTED] tests/run-sysroot.sh: Delete file with tempfiles
Signed-off-by: Aaron Merey
---
tests/run-sysroot.sh | 2 ++
1 file changed, 2 insertions(+)
diff --git a/tests/run-sysroot.sh b/tests/run-sysroot.sh
index cfdc51e9..0bbd1aa4 100755
--- a/tests/run-sysroot.sh
+++ b/tests/run-sysroot.sh
@@ -31,6 +31,8 @@ testrun "${abs_top_builddir}"/src/stack --core
"${tmpdir}/core.bash" \
# was built with LZMA support.
sed -i '4,5d' "${tmpdir}/stack.out"
+tempfiles cat.out
+
# check that we are able to get fully symbolized backtrace
testrun_compare cat "${tmpdir}/stack.out" <<\EOF
PID 431185 - core
--
2.45.2
☠ Buildbot (Sourceware): elfutils - failed test (failure) (main)
A new failure has been detected on builder elfutils-fedora-x86_64 while building elfutils. Full details are available at: https://builder.sourceware.org/buildbot/#/builders/59/builds/348 Build state: failed test (failure) Revision: 39e962f063b5e8b9c602017c8bc79f03f31873bb Worker: bbo1-2 Build Reason: (unknown) Blamelist: Aaron Merey Steps: - 0: worker_preparation ( success ) - 1: set package name ( success ) - 2: git checkout ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/2/logs/stdio - 3: autoreconf ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/3/logs/stdio - 4: configure ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/4/logs/stdio - config.log: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/4/logs/config_log - 5: get version ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/5/logs/stdio - property changes: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/5/logs/property_changes - 6: make ( warnings ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/6/logs/stdio - warnings (3): https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/6/logs/warnings__3_ - 7: make check ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/7/logs/stdio - test-suite.log: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/7/logs/test-suite_log - 8: make distcheck ( failure ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/8/logs/stdio - test-suite.log: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/8/logs/test-suite_log - warnings (6): https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/8/logs/warnings__6_ - 9: make rpmbuild ( warnings ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/9/logs/stdio - warnings (30): https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/9/logs/warnings__30_ - 10: prep ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/10/logs/stdio - 11: build bunsen.cpio.gz ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/11/logs/stdio - 12: fetch bunsen.cpio.gz ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/12/logs/stdio - 13: unpack bunsen.cpio.gz ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/13/logs/stdio - 14: pass .bunsen.source.* ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/14/logs/stdio - 15: upload to bunsen ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/15/logs/stdio - 16: clean up ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/16/logs/stdio - 17: make distclean ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/348/steps/17/logs/stdio A new failure has been detected on builder elfutils-opensusetw-x86_64 while building elfutils. Full details are available at: https://builder.sourceware.org/buildbot/#/builders/88/builds/312 Build state: failed test (failure) Revision: 39e962f063b5e8b9c602017c8bc79f03f31873bb Worker: bbo1-1 Build Reason: (unknown) Blamelist: Aaron Merey Steps: - 0: worker_preparation ( success ) - 1: set package name ( success ) - 2: git checkout ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/88/builds/312/steps/2/logs/stdio - 3: autoreconf ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/88/builds/312/steps/3/logs/stdio - 4: configure ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/88/builds/312/steps/4/logs/stdio - config.log: https://builder.sourceware.org/buildbot/#/builders/88/builds/312/steps/4/logs/config_log - 5: get version ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/88/builds/312/steps/5/logs/stdio - property changes: https://builder.sourceware.org/buildbot/#/builders/88/builds/312/steps/5/logs/property_changes - 6: make ( warnings ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/88/builds/312/steps/6/logs/stdio - warnings (3): https://builder.sourceware.org/buildbo
☝ Buildbot (Sourceware): elfutils - worker not available (main)
A retry build has been detected on builder elfutils-debian-amd64 while building elfutils. Full details are available at: https://builder.sourceware.org/buildbot/#/builders/52/builds/348 Build state: worker not available Revision: (unknown) Worker: bb3 Build Reason: (unknown) Blamelist: Aaron Merey Steps: - 0: worker_preparation ( exception ) Logs: - err.text: https://builder.sourceware.org/buildbot/#/builders/52/builds/348/steps/0/logs/err_text - err.html: https://builder.sourceware.org/buildbot/#/builders/52/builds/348/steps/0/logs/err_html
☝ Buildbot (Sourceware): elfutils - worker not available (main)
A retry build has been detected on builder elfutils-debian-amd64 while building elfutils. Full details are available at: https://builder.sourceware.org/buildbot/#/builders/52/builds/350 Build state: worker not available Revision: (unknown) Worker: bb2-2 Build Reason: (unknown) Blamelist: Aaron Merey Steps: - 0: worker_preparation ( exception ) Logs: - err.text: https://builder.sourceware.org/buildbot/#/builders/52/builds/350/steps/0/logs/err_text - err.html: https://builder.sourceware.org/buildbot/#/builders/52/builds/350/steps/0/logs/err_html A restored build has been detected on builder elfutils-fedora-x86_64 while building elfutils. Full details are available at: https://builder.sourceware.org/buildbot/#/builders/59/builds/350 Build state: build successful Revision: 78c9dce1c77098c01c076c1bf6d15ab773d3b195 Worker: bbo1-2 Build Reason: (unknown) Blamelist: Aaron Merey Steps: - 0: worker_preparation ( success ) - 1: set package name ( success ) - 2: git checkout ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/2/logs/stdio - 3: autoreconf ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/3/logs/stdio - 4: configure ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/4/logs/stdio - config.log: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/4/logs/config_log - 5: get version ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/5/logs/stdio - property changes: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/5/logs/property_changes - 6: make ( warnings ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/6/logs/stdio - warnings (3): https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/6/logs/warnings__3_ - 7: make check ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/7/logs/stdio - test-suite.log: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/7/logs/test-suite_log - 8: make distcheck ( warnings ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/8/logs/stdio - test-suite.log: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/8/logs/test-suite_log - warnings (6): https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/8/logs/warnings__6_ - 9: make rpmbuild ( warnings ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/9/logs/stdio - warnings (30): https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/9/logs/warnings__30_ - 10: prep ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/10/logs/stdio - 11: build bunsen.cpio.gz ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/11/logs/stdio - 12: fetch bunsen.cpio.gz ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/12/logs/stdio - 13: unpack bunsen.cpio.gz ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/13/logs/stdio - 14: pass .bunsen.source.* ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/14/logs/stdio - 15: upload to bunsen ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/15/logs/stdio - 16: clean up ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/16/logs/stdio - 17: make distclean ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/59/builds/350/steps/17/logs/stdio A restored build has been detected on builder elfutils-opensusetw-x86_64 while building elfutils. Full details are available at: https://builder.sourceware.org/buildbot/#/builders/88/builds/314 Build state: build successful Revision: 78c9dce1c77098c01c076c1bf6d15ab773d3b195 Worker: bbo1-2 Build Reason: (unknown) Blamelist: Aaron Merey Steps: - 0: worker_preparation ( success ) - 1: set package name ( success ) - 2: git checkout ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/88/builds/314/steps/2/logs/stdio - 3: autoreconf ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/88/builds/314/steps/3/logs/stdio - 4: configure ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#/builders/88/builds/314/steps/4/logs/std
