Re: [PATCH] libelf, readelf, elflint: Add RELR support

[Omar Sandoval]

On Thu, Aug 24, 2023 at 12:02:11AM +0200, Mark Wielaard wrote:
> Handle RELR as defined here:
> https://groups.google.com/g/generic-abi/c/bX460iggiKg/m/YT2RrjpMAwAJ
> 
> Introduce new ELF_T_RELR Elf_Type and handle it for SHT_RELR.  Check
> various properties in elflint.  Print RELR relocations in
> readelf. Just the entries with -U.  Just the addresses with -N. And
> addresses pluse symbol/offsets by default.
> 
>   * libebl/eblsectiontypename.c (ebl_section_type_name): Add RELR
>   to knownstype.
>   * libelf/elf32_updatenull.c (updatenull_wrlock): Handle
>   sh_entsize for SHT_RELR.
>   * libelf/gelf.h (Gelf_Relr): New typedef for Elf64_Relr.
>   * libelf/gelf_fsize.c (__libelf_type_sizes): Add ELF_T_RELR.
>   * libelf/gelf_xlate.c (__elf_xfctstom): Likewise.
>   * libelf/gelf_xlate.h: Add RELR as FUNDAMENTAL.
>   * libelf/libelf.h (Elf_Type): Add ELF_T_RELR.
>   * libelf/libelfP.h: Define ELF32_FSZ_RELR and ELF64_FSZ_RELR.
>   * src/elflint.c (check_reloc_shdr): Check she_entsize for
>   ELF_T_RELR.
>   (check_relr): New function.
>   (check_dynamic): Handle DT_RELR.
>   (special_sections): Add SHT_RELR.
>   (check_sections): Call check_relr.
>   * src/readelf.c (print_relocs): Also accept a Dwfl_Module.
>   (handle_relocs_relr): New function.
>   (print_dwarf_addr): Make static and declare early.
>   (process_elf_file): Pass dwflmod to print_relocs.
>   (handle_dynamic): Handle DT_RELRSZ and DTRELRENT.
> 
> Signed-off-by: Mark Wielaard 
> ---
>  libebl/eblsectiontypename.c |   3 +-
>  libelf/elf32_updatenull.c   |   3 +
>  libelf/gelf.h   |   3 +
>  libelf/gelf_fsize.c |   3 +-
>  libelf/gelf_xlate.c |   3 +-
>  libelf/gelf_xlate.h |   1 +
>  libelf/libelf.h |   1 +
>  libelf/libelfP.h|   2 +
>  src/elflint.c   |  51 +--
>  src/readelf.c   | 125 ++--
>  10 files changed, 184 insertions(+), 11 deletions(-)
> 
> diff --git a/libebl/eblsectiontypename.c b/libebl/eblsectiontypename.c
> index 2008b95a..ade25d4a 100644
> --- a/libebl/eblsectiontypename.c
> +++ b/libebl/eblsectiontypename.c
> @@ -61,7 +61,8 @@ ebl_section_type_name (Ebl *ebl, int section, char *buf, 
> size_t len)
> KNOWNSTYPE (FINI_ARRAY),
> KNOWNSTYPE (PREINIT_ARRAY),
> KNOWNSTYPE (GROUP),
> -   KNOWNSTYPE (SYMTAB_SHNDX)
> +   KNOWNSTYPE (SYMTAB_SHNDX),
> +   KNOWNSTYPE (RELR)
>   };
>  
>/* Handle standard names.  */
> diff --git a/libelf/elf32_updatenull.c b/libelf/elf32_updatenull.c
> index 6c06e5e4..c5d26b00 100644
> --- a/libelf/elf32_updatenull.c
> +++ b/libelf/elf32_updatenull.c
> @@ -256,6 +256,9 @@ __elfw2(LIBELFBITS,updatenull_wrlock) (Elf *elf, int 
> *change_bop, size_t shnum)
>   case SHT_SUNW_syminfo:
> sh_entsize = elf_typesize (LIBELFBITS, ELF_T_SYMINFO, 1);
> break;
> + case SHT_RELR:
> +   sh_entsize = elf_typesize (LIBELFBITS, ELF_T_RELR, 1);
> +   break;
>   default:
> break;
>   }
> diff --git a/libelf/gelf.h b/libelf/gelf.h
> index 7a3c87aa..f032d7e1 100644
> --- a/libelf/gelf.h
> +++ b/libelf/gelf.h
> @@ -82,6 +82,9 @@ typedef Elf64_Rel GElf_Rel;
>  /* Relocation table entry with addend (in section of type SHT_RELA).  */
>  typedef Elf64_Rela GElf_Rela;
>  
> +/* Relative relocation entry (in section of type SHT_RELR).  */
> +typedef Elf64_Relr Gelf_Relr;

Should this be GElf_Relr (with an uppercase E in GElf like the other
typedefs)?

Re: [PATCH] libelf, readelf, elflint: Add RELR support

[Omar Sandoval]

On Thu, Aug 24, 2023 at 09:51:54PM +0200, Mark Wielaard wrote:
> Hi Omar,
> 
> On Thu, Aug 24, 2023 at 12:40:19PM -0700, Omar Sandoval wrote:
> > On Thu, Aug 24, 2023 at 12:02:11AM +0200, Mark Wielaard wrote:
> > >
> > >   * libelf/gelf.h (Gelf_Relr): New typedef for Elf64_Relr.
> > >  
> > > +/* Relative relocation entry (in section of type SHT_RELR).  */
> > > +typedef Elf64_Relr Gelf_Relr;
> > 
> > Should this be GElf_Relr (with an uppercase E in GElf like the other
> > typedefs)?
> 
> Oops. Yes, it should. Thanks for spotting that.

One more thing, if someone installs a newer elfutils with this change
but their glibc is older and doesn't define Elf64_Relr (which was
apparently added in glibc 2.36), then gelf.h will be unusable:

/usr/include/gelf.h:86:9: error: unknown type name 'Elf64_Relr'
   86 | typedef Elf64_Relr Gelf_Relr;
  | ^~

(A Koji build of drgn for Fedora 40 ran into this:
https://koji.fedoraproject.org/koji/taskinfo?taskID=105242437, except
that in my case it's because I was vendoring an old version of elf.h. I
fixed it by not vendoring elf.h anymore:
https://github.com/osandov/drgn/commit/3d07cb1682045a4ac90e1977bcbd5b4864911e32,
but this same issue can come up in more legitimate situations like the
one I mentioned.)

Maybe this can be wrapped in an #ifdef DT_RELR or something like that?

[PATCH 01/14] libdw: Make try_split_file static

[Omar Sandoval]

From: Omar Sandoval 

It's only used in libdw_find_split_unit.c.

Signed-off-by: Omar Sandoval 
---
 libdw/ChangeLog   | 4 
 libdw/libdw_find_split_unit.c | 2 +-
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/libdw/ChangeLog b/libdw/ChangeLog
index 406310ef..0014aeb1 100644
--- a/libdw/ChangeLog
+++ b/libdw/ChangeLog
@@ -1,3 +1,7 @@
+2023-09-27  Omar Sandoval  
+
+   * libdw_find_split_unit.c (try_split_file): Make static.
+
 2023-02-22  Mark Wielaard  
 
* dwarf_getscopes.c (origin_match): Don't free a->scopes.
diff --git a/libdw/libdw_find_split_unit.c b/libdw/libdw_find_split_unit.c
index a22e7bc9..533f8072 100644
--- a/libdw/libdw_find_split_unit.c
+++ b/libdw/libdw_find_split_unit.c
@@ -41,7 +41,7 @@
 #include 
 #include 
 
-void
+static void
 try_split_file (Dwarf_CU *cu, const char *dwo_path)
 {
   int split_fd = open (dwo_path, O_RDONLY);
-- 
2.41.0

[PATCH 00/14] elfutils: DWARF package (.dwp) file support

[Omar Sandoval]

From: Omar Sandoval 

Hi,

This patch series adds support for DWARF package files to libdw and the
elfutils tools. It supports the GNU DebugFission format for DWARF 4 [1]
and the format standardized in DWARF 5 (section 7.3.5 "DWARF Package
Files"). It supports both automatically opening the .dwp file for a
skeleton unit and examining the .dwp file alone, just like for .dwo
files.

Patch 1 is a trivial cleanup I found while developing this. Patches 2-6
are fixes for unrelated bugs in split DWARF and/or DWARF 5 that I
encountered while adding test cases for this series. I didn't add test
cases explicitly for those fixes because they are covered by the dwp
tests later in the series.

The actual dwp support is in patches 7-10 and 13, including test cases.
Patches 11 and 12 enable testing macro information in dwp files. Patch
14 adds support and tests for an LLVM extension to the dwp format.

More details in the individual commit messages.

Thanks!
Omar

P.S. drgn's dwp branch [2] demonstrates how the new
dwarf_cu_dwp_section_info function will be used. With this patch series,
drgn's test suite passes against a Linux kernel build using .dwp.

1: https://gcc.gnu.org/wiki/DebugFissionDWP
2: https://github.com/osandov/drgn/tree/dwp

Omar Sandoval (14):
  libdw: Make try_split_file static
  libdw: Handle split DWARF in dwarf_entrypc
  libdw: Handle DW_AT_ranges in split DWARF 5 skeleton in dwarf_ranges
  libdw: Handle other string forms in dwarf_macro_param2
  libdw: Fix dwarf_macro_getsrcfiles for DWARF 5
  libdw: Handle split DWARF in dwarf_macro_getsrcfiles
  libdw: Recognize .debug_[ct]u_index sections in dwarf_elf_begin
  libdw: Parse DWARF package file index sections
  libdw, libdwfl: Save original path of ELF file
  libdw: Try .dwp file in __libdw_find_split_unit()
  tests: Handle DW_MACRO_{define,undef}_{strx,sup} in dwarf-getmacros
  tests: Optionally dump all units in dwarf-getmacros
  libdw: Apply DWARF package file section offsets where appropriate
  libdw: Handle overflowed DW_SECT_INFO offsets in DWARF package file
indexes

 libdw/ChangeLog   |   72 +
 libdw/Makefile.am |2 +-
 libdw/dwarf_begin_elf.c   |   88 +-
 libdw/dwarf_cu_dwp_section_info.c |  529 ++
 libdw/dwarf_end.c |   27 +-
 libdw/dwarf_entrypc.c |   12 +-
 libdw/dwarf_error.c   |1 +
 libdw/dwarf_getlocation.c |6 +
 libdw/dwarf_getmacros.c   |   54 +-
 libdw/dwarf_macro_getsrcfiles.c   |8 +-
 libdw/dwarf_macro_param2.c|   21 +-
 libdw/dwarf_ranges.c  |4 +-
 libdw/libdw.h |   23 +
 libdw/libdw.map   |5 +
 libdw/libdwP.h|  112 +-
 libdw/libdw_find_split_unit.c |   77 +-
 libdw/libdw_findcu.c  |8 +
 libdwfl/ChangeLog |9 +
 libdwfl/dwfl_module.c |2 +-
 libdwfl/dwfl_module_getdwarf.c|   11 +-
 libdwfl/libdwflP.h|2 +-
 libdwfl/offline.c |4 +-
 tests/.gitignore  |1 +
 tests/ChangeLog   |   36 +
 tests/Makefile.am |   15 +-
 tests/cu-dwp-section-info.c   |   74 +
 tests/dwarf-getmacros.c   |   55 +-
 tests/run-all-dwarf-ranges.sh |  114 ++
 tests/run-cu-dwp-section-info.sh  |  168 ++
 tests/run-dwarf-getmacros.sh  | 1412 +
 tests/run-get-units-split.sh  |   18 +
 tests/run-large-elf-file.sh   |  174 ++
 tests/run-varlocs.sh  |  126 +-
 tests/testfile-dwp-4-cu-index-overflow.bz2|  Bin 0 -> 4490 bytes
 .../testfile-dwp-4-cu-index-overflow.dwp.bz2  |  Bin 0 -> 5584 bytes
 tests/testfile-dwp-4-strict.bz2   |  Bin 0 -> 4169 bytes
 tests/testfile-dwp-4-strict.dwp.bz2   |  Bin 0 -> 6871 bytes
 tests/testfile-dwp-4.bz2  |  Bin 0 -> 4194 bytes
 tests/testfile-dwp-4.dwp.bz2  |  Bin 0 -> 10098 bytes
 tests/testfile-dwp-5-cu-index-overflow.bz2|  Bin 0 -> 4544 bytes
 .../testfile-dwp-5-cu-index-overflow.dwp.bz2  |  Bin 0 -> 5790 bytes
 tests/testfile-dwp-5.bz2  |  Bin 0 -> 4223 bytes
 tests/testfile-dwp-5.dwp.bz2  |  Bin 0 -> 10313 bytes
 tests/testfile-dwp-cu-index-overflow.source   |   86 +
 tests/testfile-dwp.source |  102 ++
 45 files changed, 3337 insertions(+), 121 deletions(-)
 create mode 100644 libdw/dwarf_cu_dwp_section_info.c
 create mode 100644 tests/cu-dwp-section-info.c
 create mo

[PATCH 02/14] libdw: Handle split DWARF in dwarf_entrypc

[Omar Sandoval]

From: Omar Sandoval 

If a DIE has no DW_AT_entry_pc attribute, dwarf_entrypc looks for
DW_AT_low_pc in that DIE.  But for a split compilation unit DIE,
DW_AT_low_pc is in the corresponding skeleton DIE, so this fails.
dwarf_lowpc already handles this fallback, so use it instead.

Signed-off-by: Omar Sandoval 
---
 libdw/ChangeLog   |  1 +
 libdw/dwarf_entrypc.c | 12 +---
 tests/ChangeLog   |  4 
 tests/run-varlocs.sh  | 14 +++---
 4 files changed, 17 insertions(+), 14 deletions(-)

diff --git a/libdw/ChangeLog b/libdw/ChangeLog
index 0014aeb1..af74ce0d 100644
--- a/libdw/ChangeLog
+++ b/libdw/ChangeLog
@@ -1,6 +1,7 @@
 2023-09-27  Omar Sandoval  
 
* libdw_find_split_unit.c (try_split_file): Make static.
+   * dwarf_entrypc.c (dwarf_entrypc): Call dwarf_lowpc.
 
 2023-02-22  Mark Wielaard  
 
diff --git a/libdw/dwarf_entrypc.c b/libdw/dwarf_entrypc.c
index 0ef3b0ea..543567d4 100644
--- a/libdw/dwarf_entrypc.c
+++ b/libdw/dwarf_entrypc.c
@@ -37,12 +37,10 @@
 int
 dwarf_entrypc (Dwarf_Die *die, Dwarf_Addr *return_addr)
 {
-  Dwarf_Attribute attr_mem;
-
-  return INTUSE(dwarf_formaddr) (INTUSE(dwarf_attr) (die, DW_AT_entry_pc,
-&attr_mem)
-?: INTUSE(dwarf_attr) (die, DW_AT_low_pc,
-   &attr_mem),
-return_addr);
+  Dwarf_Attribute attr_mem, *attr;
+  if ((attr = INTUSE(dwarf_attr) (die, DW_AT_entry_pc, &attr_mem)) != NULL)
+return INTUSE(dwarf_formaddr) (attr, return_addr);
+  else
+return INTUSE(dwarf_lowpc) (die, return_addr);
 }
 INTDEF(dwarf_entrypc)
diff --git a/tests/ChangeLog b/tests/ChangeLog
index d816873c..f934c114 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,7 @@
+2023-09-27  Omar Sandoval  
+
+   * run-varlocs.sh: Add entry PC to split units.
+
 2023-04-21  Frank Ch. Eigler 
 
* run-debuginfod-IXr.sh: New test.
diff --git a/tests/run-varlocs.sh b/tests/run-varlocs.sh
index b2621776..cbbcf6c7 100755
--- a/tests/run-varlocs.sh
+++ b/tests/run-varlocs.sh
@@ -206,7 +206,7 @@ testfiles testfilesplitranges5.debug
 testfiles testfile-ranges-hello5.dwo testfile-ranges-world5.dwo
 testrun_compare ${abs_top_builddir}/tests/varlocs --debug -e 
testfilesplitranges5.debug <<\EOF
 module 'testfilesplitranges5.debug'
-[14] CU 'hello.c'
+[14] CU 'hello.c'@0
   [1d] function 'no_say'@401160
 frame_base: {call_frame_cfa {...}}
 [33] parameter 'prefix'
@@ -236,7 +236,7 @@ module 'testfilesplitranges5.debug'
 [111] parameter 'count'
   [401150,401160) {reg4}
 module 'testfilesplitranges5.debug'
-[14] CU 'world.c'
+[14] CU 'world.c'@401180
   [1d] function 'no_main'@4011d0
 frame_base: {call_frame_cfa {...}}
 [35] parameter 'argc'
@@ -282,7 +282,7 @@ testfiles testfilesplitranges4.debug
 testfiles testfile-ranges-hello.dwo testfile-ranges-world.dwo
 testrun_compare ${abs_top_builddir}/tests/varlocs --debug -e 
testfilesplitranges4.debug <<\EOF
 module 'testfilesplitranges4.debug'
-[b] CU 'hello.c'
+[b] CU 'hello.c'@0
   [18] function 'no_say'@4004f0
 frame_base: {call_frame_cfa {...}}
 [2f] parameter 'prefix'
@@ -310,7 +310,7 @@ module 'testfilesplitranges4.debug'
 [102] parameter 'count'
   [4004e0,4004f0) {reg4}
 module 'testfilesplitranges4.debug'
-[b] CU 'world.c'
+[b] CU 'world.c'@400500
   [18] function 'no_main'@400550
 frame_base: {call_frame_cfa {...}}
 [2f] parameter 'argc'
@@ -373,7 +373,7 @@ EOF
 testfiles testfile-addrx_constx-5 addrx_constx-5.dwo
 testrun_compare ${abs_top_builddir}/tests/varlocs --exprlocs -e 
testfile-addrx_constx-5 <<\EOF
 module 'testfile-addrx_constx-5'
-[14] CU 'addrx_constx.c'
+[14] CU 'addrx_constx.c'@0
   producer (strx)
   language (data1)
   name (strx)
@@ -467,7 +467,7 @@ EOF
 testfiles testfile-addrx_constx-4 addrx_constx-4.dwo
 testrun_compare ${abs_top_builddir}/tests/varlocs --exprlocs -e 
testfile-addrx_constx-4 <<\EOF
 module 'testfile-addrx_constx-4'
-[b] CU 'addrx_constx.c'
+[b] CU 'addrx_constx.c'@0
   producer (GNU_str_index)
   language (data1)
   name (GNU_str_index)
@@ -565,7 +565,7 @@ testfiles splitdwarf4-not-split4.dwo
 
 testrun_compare ${abs_top_builddir}/tests/varlocs --debug -e 
testfile-splitdwarf4-not-split4.debug <<\EOF
 module 'testfile-splitdwarf4-not-split4.debug'
-[b] CU 'splitdwarf4-not-split4.c'
+[b] CU 'splitdwarf4-not-split4.c'@0
   [18] function 'main'@401050
 frame_base: {call_frame_cfa {...}}
 [30] parameter 'argc'
-- 
2.41.0

[PATCH 06/14] libdw: Handle split DWARF in dwarf_macro_getsrcfiles

[Omar Sandoval]

From: Omar Sandoval 

Macro information references file names from the line number information
table, which is tricky in split DWARF for a couple of reasons.

First, the line number information for a macro unit comes from the
.debug_line.dwo section in the split file, not the .debug_line section
in the skeleton file.  This was not specified in the GNU DebugFission
design document [1] or the DWARF 5 standard, but it is how GCC and Clang
behave in practice and was clarified in DWARF standard issue
200602.1 [2] for the upcoming DWARF 6 standard.

dwarf_macro_getsrcfiles uses the line number information from whichever
Dwarf handle it was passed.  This is error-prone, since the most natural
thing to do is to pass the skeleton Dwarf handle.  Fix this by storing
the appropriate Dwarf handle in Dwarf_Macro_Op_Table and using that one.

Second, for .debug_macinfo.dwo in GNU DebugFission (generated by gcc
-gdwarf-4 -gstrict-dwarf -gsplit-dwarf), the offset into .debug_line.dwo
is implicitly 0.  Again, this isn't in any specification, but it's how
GCC behaves in practice (Clang never generates macro information for
DWARF 4 split DWARF).  Make get_macinfo_table default to 0 for split
DWARF when it can't find DW_AT_stmt_list.

1: https://gcc.gnu.org/wiki/DebugFission
2: https://dwarfstd.org/issues/200602.1.html

Signed-off-by: Omar Sandoval 
---
 libdw/ChangeLog |  8 +---
 libdw/dwarf_getmacros.c | 11 +--
 libdw/dwarf_macro_getsrcfiles.c |  5 -
 libdw/libdwP.h  |  2 ++
 4 files changed, 20 insertions(+), 6 deletions(-)

diff --git a/libdw/ChangeLog b/libdw/ChangeLog
index d3f36cc8..be1e40bc 100644
--- a/libdw/ChangeLog
+++ b/libdw/ChangeLog
@@ -9,15 +9,17 @@
DW_FORM_strx, and DW_FORM_strx[1-4].
* dwarf_getmacros.c (get_macinfo_table): Replace assignment of
table->is_64bit with assignments of table->address_size and
-   table->offset_size.
+   table->offset_size.  Assume default DW_AT_stmt_list of 0 for split
+   DWARF.  Set table->dbg.
(get_table_for_offset): Ditto.
(read_macros): Get fake CU offset_size from table->offset_size instead
of table->is_64bit.
* dwarf_macro_getsrcfiles.c (dwarf_macro_getsrcfiles): Get
address_size for __libdw_getsrclines from table->address_size instead
-   of table->is_64bit.
+   of table->is_64bit.  Get dbg for __libdw_getsrclines from table->dbg
+   instead of dbg parameter, which is now unused.
* libdwP.h (Dwarf_Macro_Op_Table): Replace is_64bit with address_size
-   and offset_size.
+   and offset_size.  Add dbg.
 
 2023-02-22  Mark Wielaard  
 
diff --git a/libdw/dwarf_getmacros.c b/libdw/dwarf_getmacros.c
index b7d98af4..a3a78884 100644
--- a/libdw/dwarf_getmacros.c
+++ b/libdw/dwarf_getmacros.c
@@ -124,13 +124,19 @@ get_macinfo_table (Dwarf *dbg, Dwarf_Word macoff, 
Dwarf_Die *cudie)
 = INTUSE(dwarf_attr) (cudie, DW_AT_stmt_list, &attr_mem);
   Dwarf_Off line_offset = (Dwarf_Off) -1;
   if (attr != NULL)
-if (unlikely (INTUSE(dwarf_formudata) (attr, &line_offset) != 0))
-  return NULL;
+{
+  if (unlikely (INTUSE(dwarf_formudata) (attr, &line_offset) != 0))
+   return NULL;
+}
+  else if (cudie->cu->unit_type == DW_UT_split_compile
+  && dbg->sectiondata[IDX_debug_line] != NULL)
+line_offset = 0;
 
   Dwarf_Macro_Op_Table *table = libdw_alloc (dbg, Dwarf_Macro_Op_Table,
 macinfo_data_size, 1);
   memcpy (table, macinfo_data, macinfo_data_size);
 
+  table->dbg = dbg;
   table->offset = macoff;
   table->sec_index = IDX_debug_macinfo;
   table->line_offset = line_offset;
@@ -263,6 +269,7 @@ get_table_for_offset (Dwarf *dbg, Dwarf_Word macoff,
 macop_table_size, 1);
 
   *table = (Dwarf_Macro_Op_Table) {
+.dbg = dbg,
 .offset = macoff,
 .sec_index = IDX_debug_macro,
 .line_offset = line_offset,
diff --git a/libdw/dwarf_macro_getsrcfiles.c b/libdw/dwarf_macro_getsrcfiles.c
index 4e8deeeb..11c587af 100644
--- a/libdw/dwarf_macro_getsrcfiles.c
+++ b/libdw/dwarf_macro_getsrcfiles.c
@@ -36,6 +36,9 @@ int
 dwarf_macro_getsrcfiles (Dwarf *dbg, Dwarf_Macro *macro,
 Dwarf_Files **files, size_t *nfiles)
 {
+  /* This was needed before Dwarf_Macro_Op_Table stored the Dwarf handle.  */
+  (void)dbg;
+
   /* macro is declared NN */
   Dwarf_Macro_Op_Table *const table = macro->table;
   if (table->files == NULL)
@@ -71,7 +74,7 @@ dwarf_macro_getsrcfiles (Dwarf *dbg, Dwarf_Macro *macro,
 the same unit through dwarf_getsrcfiles, and the file names
 will be broken.  */
 
-  if (__libdw_getsrclines (dbg, line_offset, table->comp_dir,
+  if (__libdw_getsrclines (table->dbg, line_offset, table->comp_dir,
   table->address_siz

[PATCH 07/14] libdw: Recognize .debug_[ct]u_index sections in dwarf_elf_begin

[Omar Sandoval]

From: Omar Sandoval 

DWARF package (.dwp) files have a .debug_cu_index section and,
optionally, a .debug_tu_index section.  Add them to the list of DWARF
sections.

Unfortunately, it's not that simple: the other debug sections in a dwp
file have names ending with .dwo, which confuses the checks introduced
by commit 5b21e70216b8 ("libdw: dwarf_elf_begin should use either plain,
dwo or lto DWARF sections.").  So, we also have to special case
.debug_cu_index and .debug_tu_index in scn_dwarf_type and check_section
to treat them as TYPE_DWO sections.

Signed-off-by: Omar Sandoval 
---
 libdw/ChangeLog |  8 +++
 libdw/dwarf_begin_elf.c | 53 +
 libdw/libdwP.h  |  2 ++
 3 files changed, 37 insertions(+), 26 deletions(-)

diff --git a/libdw/ChangeLog b/libdw/ChangeLog
index be1e40bc..52327688 100644
--- a/libdw/ChangeLog
+++ b/libdw/ChangeLog
@@ -20,6 +20,14 @@
instead of dbg parameter, which is now unused.
* libdwP.h (Dwarf_Macro_Op_Table): Replace is_64bit with address_size
and offset_size.  Add dbg.
+   Add IDX_debug_cu_index and IDX_debug_tu_index.
+   * dwarf_begin_elf.c (dwarf_scnnames): Add IDX_debug_cu_index and
+   IDX_debug_tu_index.
+   (scn_to_string_section_idx): Ditto.
+   (scn_dwarf_type): Check for .debug_cu_index, .debug_tu_index,
+   .zdebug_cu_index, and .zdebug_tu_index.
+   (check_section): Change .dwo suffix matching to account for
+   .debug_cu_index and .debug_tu_index.
 
 2023-02-22  Mark Wielaard  
 
diff --git a/libdw/dwarf_begin_elf.c b/libdw/dwarf_begin_elf.c
index 92d76d24..7936d343 100644
--- a/libdw/dwarf_begin_elf.c
+++ b/libdw/dwarf_begin_elf.c
@@ -67,6 +67,8 @@ static const char dwarf_scnnames[IDX_last][19] =
   [IDX_debug_macro] = ".debug_macro",
   [IDX_debug_ranges] = ".debug_ranges",
   [IDX_debug_rnglists] = ".debug_rnglists",
+  [IDX_debug_cu_index] = ".debug_cu_index",
+  [IDX_debug_tu_index] = ".debug_tu_index",
   [IDX_gnu_debugaltlink] = ".gnu_debugaltlink"
 };
 #define ndwarf_scnnames (sizeof (dwarf_scnnames) / sizeof (dwarf_scnnames[0]))
@@ -92,6 +94,8 @@ static const enum string_section_index 
scn_to_string_section_idx[IDX_last] =
   [IDX_debug_macro] = STR_SCN_IDX_last,
   [IDX_debug_ranges] = STR_SCN_IDX_last,
   [IDX_debug_rnglists] = STR_SCN_IDX_last,
+  [IDX_debug_cu_index] = STR_SCN_IDX_last,
+  [IDX_debug_tu_index] = STR_SCN_IDX_last,
   [IDX_gnu_debugaltlink] = STR_SCN_IDX_last
 };
 
@@ -109,6 +113,11 @@ scn_dwarf_type (Dwarf *result, size_t shstrndx, Elf_Scn 
*scn)
 {
   if (startswith (scnname, ".gnu.debuglto_.debug"))
return TYPE_GNU_LTO;
+  else if (strcmp (scnname, ".debug_cu_index") == 0
+  || strcmp (scnname, ".debug_tu_index") == 0
+  || strcmp (scnname, ".zdebug_cu_index") == 0
+  || strcmp (scnname, ".zdebug_tu_index") == 0)
+   return TYPE_DWO;
   else if (startswith (scnname, ".debug_") || startswith (scnname, 
".zdebug_"))
{
  size_t len = strlen (scnname);
@@ -173,42 +182,34 @@ check_section (Dwarf *result, size_t shstrndx, Elf_Scn 
*scn, bool inscngrp)
   bool gnu_compressed = false;
   for (cnt = 0; cnt < ndwarf_scnnames; ++cnt)
 {
+  /* .debug_cu_index and .debug_tu_index don't have a .dwo suffix,
+but they are for DWO.  */
+  if (result->type != TYPE_DWO
+ && (cnt == IDX_debug_cu_index || cnt == IDX_debug_tu_index))
+   continue;
+  bool need_dot_dwo =
+   (result->type == TYPE_DWO
+&& cnt != IDX_debug_cu_index
+&& cnt != IDX_debug_tu_index);
   size_t dbglen = strlen (dwarf_scnnames[cnt]);
   size_t scnlen = strlen (scnname);
   if (strncmp (scnname, dwarf_scnnames[cnt], dbglen) == 0
- && (dbglen == scnlen
- || (scnlen == dbglen + 4
+ && ((!need_dot_dwo && dbglen == scnlen)
+ || (need_dot_dwo
+ && scnlen == dbglen + 4
  && strstr (scnname, ".dwo") == scnname + dbglen)))
-   {
- if (dbglen == scnlen)
-   {
- if (result->type == TYPE_PLAIN)
-   break;
-   }
- else if (result->type == TYPE_DWO)
-   break;
-   }
+   break;
   else if (scnname[0] == '.' && scnname[1] == 'z'
   && (strncmp (&scnname[2], &dwarf_scnnames[cnt][1],
dbglen - 1) == 0
-  && (scnlen == dbglen + 1
-  || (scnlen == dbglen + 5
+  && ((!need_dot_dwo && scnlen == dbglen + 1)
+  || (need_dot_dwo
+  && scnlen == dbgle

[PATCH 03/14] libdw: Handle DW_AT_ranges in split DWARF 5 skeleton in dwarf_ranges

[Omar Sandoval]

From: Omar Sandoval 

When commit 879f3a4f99df ("libdw: Handle .debug_rnglists in
dwarf_ranges.") added support for split DWARF 5 in 2018, GCC put all
range lists for split DWARF in the .debug_rnglists section of the
skeleton file (similarly to GNU DebugFission, which puts all range lists
in .debug_ranges in the skeleton file).

In 2021, after a discussion on the dwarf-discuss mailing list [1], GCC
changed this to match Clang's behavior.  Now, ranges are in
.debug_rnglists.dwo in the split file, _except_ for one: the skeleton
unit DIE has a DW_AT_ranges attribute, and its ranges are in
.debug_rnglists in the skeleton file.  See GCC commit 4b33c5aaab9e
("dwarf2out: Fix up ranges for -gdwarf-5 -gsplit-dwarf [PR99490]") and
the Issue 210310.1 clarifying the DWARF standard [2].

Unfortunately, this confuses dwarf_ranges, which always uses
.debug_rnglists.dwo if it exists.  Fix it by special casing the unit
DIE: its range lists should be in .debug_rnglists if that exists, and
.debug_rnglists.dwo otherwise.

1: https://lists.dwarfstd.org/pipermail/dwarf-discuss/2021-March/002009.html
2: https://dwarfstd.org/issues/210310.1.html

Signed-off-by: Omar Sandoval 
---
 libdw/ChangeLog  | 2 ++
 libdw/dwarf_ranges.c | 4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/libdw/ChangeLog b/libdw/ChangeLog
index af74ce0d..e84432f6 100644
--- a/libdw/ChangeLog
+++ b/libdw/ChangeLog
@@ -2,6 +2,8 @@
 
* libdw_find_split_unit.c (try_split_file): Make static.
* dwarf_entrypc.c (dwarf_entrypc): Call dwarf_lowpc.
+   * dwarf_ranges.c (dwarf_ranges): Use skeleton ranges section for
+   skeleton units.
 
 2023-02-22  Mark Wielaard  
 
diff --git a/libdw/dwarf_ranges.c b/libdw/dwarf_ranges.c
index 520f9ffe..b853e4b9 100644
--- a/libdw/dwarf_ranges.c
+++ b/libdw/dwarf_ranges.c
@@ -489,10 +489,10 @@ dwarf_ranges (Dwarf_Die *die, ptrdiff_t offset, 
Dwarf_Addr *basep,
 
   size_t secidx = (cu->version < 5 ? IDX_debug_ranges : IDX_debug_rnglists);
   const Elf_Data *d = cu->dbg->sectiondata[secidx];
-  if (d == NULL && cu->unit_type == DW_UT_split_compile)
+  if (cu->unit_type == DW_UT_split_compile && (d == NULL || is_cudie (die)))
 {
   Dwarf_CU *skel = __libdw_find_split_unit (cu);
-  if (skel != NULL)
+  if (skel != NULL && skel->dbg->sectiondata[secidx] != NULL)
{
  cu = skel;
  d = cu->dbg->sectiondata[secidx];
-- 
2.41.0

[PATCH 04/14] libdw: Handle other string forms in dwarf_macro_param2

[Omar Sandoval]

From: Omar Sandoval 

dwarf_getmacros handles the additional macro string forms added by DWARF
5, but dwarf_macro_param2 doesn't.  Update it with the list of all
string forms allowed in .debug_macro.  In particular, GCC and Clang
generate DW_MACRO_define_strx and DW_MACRO_undef_strx, which
dwarf_macro_param2 couldn't handle.

Fixes: cdf865b890c2 ("readelf, libdw: Handle DWARF5 .debug_macro.")
Signed-off-by: Omar Sandoval 
---
 libdw/ChangeLog|  3 +++
 libdw/dwarf_macro_param2.c | 21 +++--
 2 files changed, 18 insertions(+), 6 deletions(-)

diff --git a/libdw/ChangeLog b/libdw/ChangeLog
index e84432f6..7528c093 100644
--- a/libdw/ChangeLog
+++ b/libdw/ChangeLog
@@ -4,6 +4,9 @@
* dwarf_entrypc.c (dwarf_entrypc): Call dwarf_lowpc.
* dwarf_ranges.c (dwarf_ranges): Use skeleton ranges section for
skeleton units.
+   * dwarf_macro_param2.c (dwarf_macro_param2): Change form condition to
+   switch statement and add DW_FORM_line_strp, DW_FORM_strp_sup,
+   DW_FORM_strx, and DW_FORM_strx[1-4].
 
 2023-02-22  Mark Wielaard  
 
diff --git a/libdw/dwarf_macro_param2.c b/libdw/dwarf_macro_param2.c
index cc902c99..f12e6f9a 100644
--- a/libdw/dwarf_macro_param2.c
+++ b/libdw/dwarf_macro_param2.c
@@ -44,12 +44,21 @@ dwarf_macro_param2 (Dwarf_Macro *macro, Dwarf_Word *paramp, 
const char **strp)
   if (dwarf_macro_param (macro, 1, ¶m) != 0)
 return -1;
 
-  if (param.form == DW_FORM_string
-  || param.form == DW_FORM_strp)
+  switch (param.form)
 {
-  *strp = dwarf_formstring (¶m);
-  return 0;
+  /* String forms allowed by libdw_valid_user_form.  */
+  case DW_FORM_line_strp:
+  case DW_FORM_string:
+  case DW_FORM_strp:
+  case DW_FORM_strp_sup:
+  case DW_FORM_strx:
+  case DW_FORM_strx1:
+  case DW_FORM_strx2:
+  case DW_FORM_strx3:
+  case DW_FORM_strx4:
+   *strp = dwarf_formstring (¶m);
+   return 0;
+  default:
+   return dwarf_formudata (¶m, paramp);
 }
-  else
-return dwarf_formudata (¶m, paramp);
 }
-- 
2.41.0

[PATCH 05/14] libdw: Fix dwarf_macro_getsrcfiles for DWARF 5

[Omar Sandoval]

From: Omar Sandoval 

Dwarf_Macro_Op_Table::is_64bit conflates the address size and the offset
size: for .debug_macinfo, it is initialized based on the compilation
unit's address size, but for .debug_macro, it is initialized based on
the macro unit's offset size.  is_64bit is used to determine the address
size to pass to __libdw_getsrclines.  For a 64-bit architecture using
DWARF 5 with 32-bit offsets (the common case), this fails because
read_srclines checks that the given address size matches the address
size from the line number program header.

Fix it by splitting is_64bit into separate address_size and offset_size
members.

Fixes: fb90bf3f84b5 ("Support .debug_macro")
Signed-off-by: Omar Sandoval 
---
 libdw/ChangeLog | 11 +++
 libdw/dwarf_getmacros.c | 17 ++---
 libdw/dwarf_macro_getsrcfiles.c |  3 +--
 libdw/libdwP.h  |  3 ++-
 4 files changed, 28 insertions(+), 6 deletions(-)

diff --git a/libdw/ChangeLog b/libdw/ChangeLog
index 7528c093..d3f36cc8 100644
--- a/libdw/ChangeLog
+++ b/libdw/ChangeLog
@@ -7,6 +7,17 @@
* dwarf_macro_param2.c (dwarf_macro_param2): Change form condition to
switch statement and add DW_FORM_line_strp, DW_FORM_strp_sup,
DW_FORM_strx, and DW_FORM_strx[1-4].
+   * dwarf_getmacros.c (get_macinfo_table): Replace assignment of
+   table->is_64bit with assignments of table->address_size and
+   table->offset_size.
+   (get_table_for_offset): Ditto.
+   (read_macros): Get fake CU offset_size from table->offset_size instead
+   of table->is_64bit.
+   * dwarf_macro_getsrcfiles.c (dwarf_macro_getsrcfiles): Get
+   address_size for __libdw_getsrclines from table->address_size instead
+   of table->is_64bit.
+   * libdwP.h (Dwarf_Macro_Op_Table): Replace is_64bit with address_size
+   and offset_size.
 
 2023-02-22  Mark Wielaard  
 
diff --git a/libdw/dwarf_getmacros.c b/libdw/dwarf_getmacros.c
index fd929669..b7d98af4 100644
--- a/libdw/dwarf_getmacros.c
+++ b/libdw/dwarf_getmacros.c
@@ -134,7 +134,8 @@ get_macinfo_table (Dwarf *dbg, Dwarf_Word macoff, Dwarf_Die 
*cudie)
   table->offset = macoff;
   table->sec_index = IDX_debug_macinfo;
   table->line_offset = line_offset;
-  table->is_64bit = cudie->cu->address_size == 8;
+  table->address_size = cudie->cu->address_size;
+  table->offset_size = cudie->cu->offset_size;
   table->comp_dir = __libdw_getcompdir (cudie);
 
   return table;
@@ -182,6 +183,15 @@ get_table_for_offset (Dwarf *dbg, Dwarf_Word macoff,
  return NULL;
 }
 
+  uint8_t address_size;
+  if (cudie != NULL)
+address_size = cudie->cu->address_size;
+  else
+{
+  char *ident = elf_getident (dbg->elf, NULL);
+  address_size = ident[EI_CLASS] == ELFCLASS32 ? 4 : 8;
+}
+
   /* """The macinfo entry types defined in this standard may, but
  might not, be described in the table""".
 
@@ -258,7 +268,8 @@ get_table_for_offset (Dwarf *dbg, Dwarf_Word macoff,
 .line_offset = line_offset,
 .header_len = readp - startp,
 .version = version,
-.is_64bit = is_64bit,
+.address_size = address_size,
+.offset_size = is_64bit ? 8 : 4,
 
 /* NULL if CUDIE is NULL or DW_AT_comp_dir is absent.  */
 .comp_dir = __libdw_getcompdir (cudie),
@@ -368,7 +379,7 @@ read_macros (Dwarf *dbg, int sec_index,
.dbg = dbg,
.sec_idx = sec_index,
.version = table->version,
-   .offset_size = table->is_64bit ? 8 : 4,
+   .offset_size = table->offset_size,
.str_off_base = str_offsets_base_off (dbg, (cudie != NULL
? cudie->cu: NULL)),
.startp = (void *) startp + offset,
diff --git a/libdw/dwarf_macro_getsrcfiles.c b/libdw/dwarf_macro_getsrcfiles.c
index 3b1794b1..4e8deeeb 100644
--- a/libdw/dwarf_macro_getsrcfiles.c
+++ b/libdw/dwarf_macro_getsrcfiles.c
@@ -72,8 +72,7 @@ dwarf_macro_getsrcfiles (Dwarf *dbg, Dwarf_Macro *macro,
 will be broken.  */
 
   if (__libdw_getsrclines (dbg, line_offset, table->comp_dir,
-  table->is_64bit ? 8 : 4,
-  NULL, &table->files) < 0)
+  table->address_size, NULL, &table->files) < 0)
table->files = (void *) -1;
 }
 
diff --git a/libdw/libdwP.h b/libdw/libdwP.h
index 5cbdc279..c3fe9f93 100644
--- a/libdw/libdwP.h
+++ b/libdw/libdwP.h
@@ -526,7 +526,8 @@ typedef struct
   Dwarf_Half header_len;
 
   uint16_t version;
-  bool is_64bit;
+  uint8_t address_size;
+  uint8_t offset_size;
   uint8_t sec_index;   /* IDX_debug_macro or IDX_debug_macinfo.  */
 
   /* Shows where in TABLE each opcode is defined.  Since opcode 0 is
-- 
2.41.0

[PATCH 09/14] libdw, libdwfl: Save original path of ELF file

[Omar Sandoval]

From: Omar Sandoval 

libdw and libdwfl currently save the path of the directory containing
the ELF file to use when searching for alt and dwo files.  To search for
dwp files, we need the file name too.  Add an elfpath field to Dwarf,
and set the debugdir field from it.  Also update libdwfl to set elfpath
and debugdir.

Signed-off-by: Omar Sandoval 
---
 libdw/ChangeLog| 11 ++-
 libdw/dwarf_begin_elf.c| 34 --
 libdw/dwarf_end.c  |  3 ++-
 libdw/libdwP.h | 12 ++--
 libdwfl/ChangeLog  |  9 +
 libdwfl/dwfl_module.c  |  2 +-
 libdwfl/dwfl_module_getdwarf.c | 11 +++
 libdwfl/libdwflP.h |  2 +-
 libdwfl/offline.c  |  4 ++--
 9 files changed, 62 insertions(+), 26 deletions(-)

diff --git a/libdw/ChangeLog b/libdw/ChangeLog
index 1d229094..f491587f 100644
--- a/libdw/ChangeLog
+++ b/libdw/ChangeLog
@@ -20,7 +20,7 @@
instead of dbg parameter, which is now unused.
* libdwP.h (Dwarf_Macro_Op_Table): Replace is_64bit with address_size
and offset_size.  Add dbg.
-   (Dwarf): Add cu_index and tu_index.
+   (Dwarf): Add cu_index and tu_index.  Add elfpath.
(Dwarf_CU): Add dwp_row.
(Dwarf_Package_Index): New type.
(DW_SECT_TYPES): New macro.
@@ -28,6 +28,9 @@
(dwarf_cu_dwp_section_info): New INTDECL.
Add IDX_debug_cu_index and IDX_debug_tu_index.  Add
DWARF_E_UNKNOWN_SECTION.
+   (__libdw_debugdir): Replace declaration with...
+   (__libdw_elfpath): New declaration.
+   (__libdw_set_debugdir): New declaration.
* dwarf_begin_elf.c (dwarf_scnnames): Add IDX_debug_cu_index and
IDX_debug_tu_index.
(scn_to_string_section_idx): Ditto.
@@ -35,8 +38,14 @@
.zdebug_cu_index, and .zdebug_tu_index.
(check_section): Change .dwo suffix matching to account for
.debug_cu_index and .debug_tu_index.
+   (__libdw_debugdir): Replace with..
+   (__libdw_elfpath): New function.
+   (__libdw_set_debugdir): New function.
+   (valid_p): Call __libdw_elfpath and __libdw_set_debugdir instead of
+   __libdw_debugdir.
* Makefile.am (libdw_a_SOURCES): Add dwarf_cu_dwp_section_info.c.
* dwarf_end.c (dwarf_end): Free dwarf->cu_index and dwarf->tu_index.
+   Free dwarf->elfpath.
* dwarf_error.c (errmsgs): Add DWARF_E_UNKNOWN_SECTION.
* libdw.h (dwarf_cu_dwp_section_info): New declaration.
* libdw.map (ELFUTILS_0.190): Add dwarf_cu_dwp_section_info.
diff --git a/libdw/dwarf_begin_elf.c b/libdw/dwarf_begin_elf.c
index 7936d343..323a91d0 100644
--- a/libdw/dwarf_begin_elf.c
+++ b/libdw/dwarf_begin_elf.c
@@ -272,24 +272,27 @@ check_section (Dwarf *result, size_t shstrndx, Elf_Scn 
*scn, bool inscngrp)
   return result;
 }
 
-
-/* Helper function to set debugdir field.  We want to cache the dir
-   where we found this Dwarf ELF file to locate alt and dwo files.  */
 char *
-__libdw_debugdir (int fd)
+__libdw_elfpath (int fd)
 {
   /* strlen ("/proc/self/fd/") = 14 + strlen () = 10 + 1 = 25.  */
   char devfdpath[25];
   sprintf (devfdpath, "/proc/self/fd/%u", fd);
-  char *fdpath = realpath (devfdpath, NULL);
-  char *fddir;
-  if (fdpath != NULL && fdpath[0] == '/'
-  && (fddir = strrchr (fdpath, '/')) != NULL)
-{
-  *++fddir = '\0';
-  return fdpath;
-}
-  return NULL;
+  return realpath (devfdpath, NULL);
+}
+
+
+void
+__libdw_set_debugdir (Dwarf *dbg)
+{
+  if (dbg->elfpath == NULL || dbg->elfpath[0] != '/')
+return;
+  size_t dirlen = strrchr (dbg->elfpath, '/') - dbg->elfpath + 1;
+  dbg->debugdir = malloc (dirlen + 1);
+  if (dbg->debugdir == NULL)
+return;
+  memcpy (dbg->debugdir, dbg->elfpath, dirlen);
+  dbg->debugdir[dirlen] = '\0';
 }
 
 
@@ -421,7 +424,10 @@ valid_p (Dwarf *result)
 }
 
   if (result != NULL)
-result->debugdir = __libdw_debugdir (result->elf->fildes);
+{
+  result->elfpath = __libdw_elfpath (result->elf->fildes);
+  __libdw_set_debugdir(result);
+}
 
   return result;
 }
diff --git a/libdw/dwarf_end.c b/libdw/dwarf_end.c
index 18a419ce..b7f817d9 100644
--- a/libdw/dwarf_end.c
+++ b/libdw/dwarf_end.c
@@ -147,7 +147,8 @@ dwarf_end (Dwarf *dwarf)
  close (dwarf->alt_fd);
}
 
-  /* The cached dir we found the Dwarf ELF file in.  */
+  /* The cached path and dir we found the Dwarf ELF file in.  */
+  free (dwarf->elfpath);
   free (dwarf->debugdir);
 
   /* Free the context descriptor.  */
diff --git a/libdw/libdwP.h b/libdw/libdwP.h
index 13ca58ce..214d1711 100644
--- a/libdw/libdwP.h
+++ b/libdw/libdwP.h
@@ -169,6 +169,10 @@ struct Dwarf
   /* The underlying ELF file.  */
   Elf *elf;
 
+  /* The (absolute) path to the ELF fil

[PATCH 10/14] libdw: Try .dwp file in __libdw_find_split_unit()

[Omar Sandoval]

From: Omar Sandoval 

Try opening the file in the location suggested by the standard (the
skeleton file name + ".dwp") and looking up the unit in the package
index.  The rest is similar to .dwo files, with slightly different
cleanup since a single Dwarf handle is shared.

Signed-off-by: Omar Sandoval 
---
 libdw/ChangeLog   | 11 -
 libdw/dwarf_begin_elf.c   |  1 +
 libdw/dwarf_cu_dwp_section_info.c | 19 
 libdw/dwarf_end.c | 10 -
 libdw/libdwP.h| 16 ++-
 libdw/libdw_find_split_unit.c | 75 ---
 6 files changed, 122 insertions(+), 10 deletions(-)

diff --git a/libdw/ChangeLog b/libdw/ChangeLog
index f491587f..f37d9411 100644
--- a/libdw/ChangeLog
+++ b/libdw/ChangeLog
@@ -1,6 +1,8 @@
 2023-09-27  Omar Sandoval  
 
* libdw_find_split_unit.c (try_split_file): Make static.
+   (try_dwp_file): New function.
+   (__libdw_find_split_unit): Call try_dwp_file.
* dwarf_entrypc.c (dwarf_entrypc): Call dwarf_lowpc.
* dwarf_ranges.c (dwarf_ranges): Use skeleton ranges section for
skeleton units.
@@ -20,7 +22,8 @@
instead of dbg parameter, which is now unused.
* libdwP.h (Dwarf_Macro_Op_Table): Replace is_64bit with address_size
and offset_size.  Add dbg.
-   (Dwarf): Add cu_index and tu_index.  Add elfpath.
+   (Dwarf): Add cu_index and tu_index.  Add elfpath.  Add dwp_dwarf and
+   dwp_fd.
(Dwarf_CU): Add dwp_row.
(Dwarf_Package_Index): New type.
(DW_SECT_TYPES): New macro.
@@ -31,6 +34,8 @@
(__libdw_debugdir): Replace declaration with...
(__libdw_elfpath): New declaration.
(__libdw_set_debugdir): New declaration.
+   (__libdw_dwp_findcu_id): New declaration.
+   (__libdw_link_skel_split): Handle .debug_addr for dwp.
* dwarf_begin_elf.c (dwarf_scnnames): Add IDX_debug_cu_index and
IDX_debug_tu_index.
(scn_to_string_section_idx): Ditto.
@@ -43,9 +48,11 @@
(__libdw_set_debugdir): New function.
(valid_p): Call __libdw_elfpath and __libdw_set_debugdir instead of
__libdw_debugdir.
+   (dwarf_begin_elf): Initialize result->dwp_fd.
* Makefile.am (libdw_a_SOURCES): Add dwarf_cu_dwp_section_info.c.
* dwarf_end.c (dwarf_end): Free dwarf->cu_index and dwarf->tu_index.
-   Free dwarf->elfpath.
+   Free dwarf->elfpath.  Free dwarf->dwp_dwarf and close dwarf->dwp_fd.
+   (cu_free): Don't free split dbg if it is dwp_dwarf.
* dwarf_error.c (errmsgs): Add DWARF_E_UNKNOWN_SECTION.
* libdw.h (dwarf_cu_dwp_section_info): New declaration.
* libdw.map (ELFUTILS_0.190): Add dwarf_cu_dwp_section_info.
diff --git a/libdw/dwarf_begin_elf.c b/libdw/dwarf_begin_elf.c
index 323a91d0..ca2b7e2a 100644
--- a/libdw/dwarf_begin_elf.c
+++ b/libdw/dwarf_begin_elf.c
@@ -567,6 +567,7 @@ dwarf_begin_elf (Elf *elf, Dwarf_Cmd cmd, Elf_Scn *scngrp)
 
   result->elf = elf;
   result->alt_fd = -1;
+  result->dwp_fd = -1;
 
   /* Initialize the memory handling.  Initial blocks are allocated on first
  actual allocation.  */
diff --git a/libdw/dwarf_cu_dwp_section_info.c 
b/libdw/dwarf_cu_dwp_section_info.c
index 6766fb9a..7bf08d9d 100644
--- a/libdw/dwarf_cu_dwp_section_info.c
+++ b/libdw/dwarf_cu_dwp_section_info.c
@@ -340,6 +340,25 @@ __libdw_dwp_find_unit (Dwarf *dbg, bool debug_types, 
Dwarf_Off off,
   abbrev_offsetp, NULL);
 }
 
+Dwarf_CU *
+internal_function
+__libdw_dwp_findcu_id (Dwarf *dbg, uint64_t unit_id8)
+{
+  Dwarf_Package_Index *index = __libdw_package_index (dbg, false);
+  uint32_t unit_row;
+  Dwarf_Off offset;
+  Dwarf_CU *cu;
+  if (__libdw_dwp_unit_row (index, unit_id8, &unit_row) == 0
+  && __libdw_dwp_section_info (index, unit_row, DW_SECT_INFO, &offset,
+  NULL) == 0
+  && (cu = __libdw_findcu (dbg, offset, false)) != NULL
+  && cu->unit_type == DW_UT_split_compile
+  && cu->unit_id8 == unit_id8)
+return cu;
+  else
+return NULL;
+}
+
 int
 dwarf_cu_dwp_section_info (Dwarf_CU *cu, unsigned int section,
   Dwarf_Off *offsetp, Dwarf_Off *sizep)
diff --git a/libdw/dwarf_end.c b/libdw/dwarf_end.c
index b7f817d9..78224ddb 100644
--- a/libdw/dwarf_end.c
+++ b/libdw/dwarf_end.c
@@ -66,7 +66,9 @@ cu_free (void *arg)
  /* The fake_addr_cu might be shared, only release one.  */
  if (p->dbg->fake_addr_cu == p->split->dbg->fake_addr_cu)
p->split->dbg->fake_addr_cu = NULL;
- INTUSE(dwarf_end) (p->split->dbg);
+ /* There is only one DWP file. We free it later.  */
+ if (p->split->dbg != p->dbg->dwp_dwarf)
+   INTUSE(dwarf_end) (p->split->dbg);
}
 }
 }
@@ -147,6 +149,12 @@ dwarf_e

[PATCH 14/14] libdw: Handle overflowed DW_SECT_INFO offsets in DWARF package file indexes

[Omar Sandoval]

From: Omar Sandoval 

Meta uses DWARF package files for our large, statically-linked C++
applications.  Some of our largest applications have more than 4GB in
.debug_info.dwo, but the section offsets in .debug_cu_index and
.debug_tu_index are 32 bits; see the discussion here [1].  We
implemented a workaround/extension for this in LLVM.  Implement the
equivalent in libdw.

To test this, we need files with more than 4GB in .debug_info.dwo.  I
created these artificially by editing GCC's assembly output.  They
compress down to 6KB.  I test them from run-large-elf-file.sh to take
advantage of the existing checks for large file support.

1: https://discourse.llvm.org/t/dwarf-dwp-4gb-limit/63902.

Signed-off-by: Omar Sandoval 
---
 libdw/ChangeLog   |   1 +
 libdw/dwarf_cu_dwp_section_info.c | 143 +-
 libdw/dwarf_end.c |  15 +-
 libdw/libdwP.h|   3 +
 tests/ChangeLog   |   7 +
 tests/Makefile.am |   6 +-
 tests/run-large-elf-file.sh   | 174 ++
 tests/testfile-dwp-4-cu-index-overflow.bz2| Bin 0 -> 4490 bytes
 .../testfile-dwp-4-cu-index-overflow.dwp.bz2  | Bin 0 -> 5584 bytes
 tests/testfile-dwp-5-cu-index-overflow.bz2| Bin 0 -> 4544 bytes
 .../testfile-dwp-5-cu-index-overflow.dwp.bz2  | Bin 0 -> 5790 bytes
 tests/testfile-dwp-cu-index-overflow.source   |  86 +
 12 files changed, 430 insertions(+), 5 deletions(-)
 create mode 100755 tests/testfile-dwp-4-cu-index-overflow.bz2
 create mode 100644 tests/testfile-dwp-4-cu-index-overflow.dwp.bz2
 create mode 100755 tests/testfile-dwp-5-cu-index-overflow.bz2
 create mode 100644 tests/testfile-dwp-5-cu-index-overflow.dwp.bz2
 create mode 100644 tests/testfile-dwp-cu-index-overflow.source

diff --git a/libdw/ChangeLog b/libdw/ChangeLog
index 08b2c3e6..158983e3 100644
--- a/libdw/ChangeLog
+++ b/libdw/ChangeLog
@@ -59,6 +59,7 @@
* dwarf_end.c (dwarf_end): Free dwarf->cu_index and dwarf->tu_index.
Free dwarf->elfpath.  Free dwarf->dwp_dwarf and close dwarf->dwp_fd.
(cu_free): Don't free split dbg if it is dwp_dwarf.
+   (dwarf_package_index_free): New function.
* dwarf_error.c (errmsgs): Add DWARF_E_UNKNOWN_SECTION.
* libdw.h (dwarf_cu_dwp_section_info): New declaration.
* libdw.map (ELFUTILS_0.190): Add dwarf_cu_dwp_section_info.
diff --git a/libdw/dwarf_cu_dwp_section_info.c 
b/libdw/dwarf_cu_dwp_section_info.c
index 7bf08d9d..8999c382 100644
--- a/libdw/dwarf_cu_dwp_section_info.c
+++ b/libdw/dwarf_cu_dwp_section_info.c
@@ -30,6 +30,8 @@
 # include 
 #endif
 
+#include 
+
 #include "libdwP.h"
 
 static Dwarf_Package_Index *
@@ -161,6 +163,7 @@ __libdw_read_package_index (Dwarf *dbg, bool tu)
   index->indices = indices;
   index->section_offsets = section_offsets;
   index->section_sizes = section_sizes;
+  index->debug_info_offsets = NULL;
 
   return index;
 }
@@ -177,6 +180,137 @@ __libdw_package_index (Dwarf *dbg, bool tu)
   if (index == NULL)
 return NULL;
 
+  /* Offsets in the section offset table are 32-bit unsigned integers.  In
+ practice, the .debug_info.dwo section for very large executables can be
+ larger than 4GB.  GNU dwp as of binutils 2.41 and llvm-dwp before LLVM 15
+ both accidentally truncate offsets larger than 4GB.
+
+ LLVM 15 detects the overflow and errors out instead; see LLVM commit
+ f8df8114715b ("[DWP][DWARF] Detect and error on debug info offset
+ overflow").  However, lldb in LLVM 16 supports using dwp files with
+ truncated offsets by recovering them directly from the unit headers in the
+ .debug_info.dwo section; see LLVM commit c0db06227721 ("[DWARFLibrary] Add
+ support to re-construct cu-index").  Since LLVM 17, the overflow error can
+ be turned into a warning instead; see LLVM commit 53a483cee801 ("[DWP] add
+ overflow check for llvm-dwp tools if offset overflow").
+
+ LLVM's support for > 4GB offsets is effectively an extension to the DWARF
+ package file format, which we implement here.  The strategy is to walk the
+ unit headers in .debug_info.dwo in lockstep with the DW_SECT_INFO columns
+ in the section offset tables.  As long as they are in the same order
+ (which they are in practice for both GNU dwp and llvm-dwp), we can
+ correlate the truncated offset and produce a corrected array of offsets.
+
+ Note that this will be fixed properly in DWARF 6:
+ https://dwarfstd.org/issues/220708.2.html.  */
+  if (index->sections[DW_SECT_INFO - 1] != UINT32_MAX
+  && dbg->sectiondata[IDX_debug_info]->d_size > UINT32_MAX)
+{
+  Dwarf_Package_Index *cu_index, *tu_index = NULL;
+  if (tu)
+   {
+ tu_index = index;
+ assert 

[PATCH 11/14] tests: Handle DW_MACRO_{define, undef}_{strx, sup} in dwarf-getmacros

[Omar Sandoval]

From: Omar Sandoval 

Signed-off-by: Omar Sandoval 
---
 tests/ChangeLog | 3 +++
 tests/dwarf-getmacros.c | 4 
 2 files changed, 7 insertions(+)

diff --git a/tests/ChangeLog b/tests/ChangeLog
index 687a9f32..4380c57f 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -14,6 +14,9 @@
* testfile-dwp-5.bz2: New test file.
* testfile-dwp-5.dwp.bz2: New test file.
* testfile-dwp.source: New file.
+   * dwarf-getmacros.c (mac): Add DW_MACRO_define_sup,
+   DW_MACRO_define_strx, DW_MACRO_undef_sup, and DW_MACRO_undef_strx
+   cases to opcode switch statement.
 
 2023-04-21  Frank Ch. Eigler 
 
diff --git a/tests/dwarf-getmacros.c b/tests/dwarf-getmacros.c
index ac70248d..e291bfd2 100644
--- a/tests/dwarf-getmacros.c
+++ b/tests/dwarf-getmacros.c
@@ -82,6 +82,8 @@ mac (Dwarf_Macro *macro, void *dbg)
 
 case DW_MACINFO_define:
 case DW_MACRO_define_strp:
+case DW_MACRO_define_sup:
+case DW_MACRO_define_strx:
   {
const char *value;
dwarf_macro_param2 (macro, NULL, &value);
@@ -91,6 +93,8 @@ mac (Dwarf_Macro *macro, void *dbg)
 
 case DW_MACINFO_undef:
 case DW_MACRO_undef_strp:
+case DW_MACRO_undef_sup:
+case DW_MACRO_undef_strx:
   break;
 
 default:
-- 
2.41.0

[PATCH 12/14] tests: Optionally dump all units in dwarf-getmacros

[Omar Sandoval]

From: Omar Sandoval 

If instead of a CU offset an empty string is given as the second
argument, dump all units.

Signed-off-by: Omar Sandoval 
---
 tests/ChangeLog |  3 +++
 tests/dwarf-getmacros.c | 51 +
 2 files changed, 44 insertions(+), 10 deletions(-)

diff --git a/tests/ChangeLog b/tests/ChangeLog
index 4380c57f..97261444 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -17,6 +17,9 @@
* dwarf-getmacros.c (mac): Add DW_MACRO_define_sup,
DW_MACRO_define_strx, DW_MACRO_undef_sup, and DW_MACRO_undef_strx
cases to opcode switch statement.
+   (getmacros): New function.
+   (main): If second argument is empty, call getmacros on every unit.
+   Otherwise, call getmacros on given unit.
 
 2023-04-21  Frank Ch. Eigler 
 
diff --git a/tests/dwarf-getmacros.c b/tests/dwarf-getmacros.c
index e291bfd2..8381d42c 100644
--- a/tests/dwarf-getmacros.c
+++ b/tests/dwarf-getmacros.c
@@ -121,26 +121,57 @@ include (Dwarf *dbg, Dwarf_Off macoff, ptrdiff_t token)
   }
 }
 
+static void
+getmacros (Dwarf *dbg, Dwarf_Die *die, bool new_style)
+{
+  for (ptrdiff_t off = new_style ? DWARF_GETMACROS_START : 0;
+   (off = dwarf_getmacros (die, mac, dbg, off)); )
+if (off == -1)
+  {
+   puts (dwarf_errmsg (-1));
+   break;
+  }
+}
+
 int
 main (int argc, char *argv[])
 {
   assert (argc >= 3);
   const char *name = argv[1];
-  ptrdiff_t cuoff = strtol (argv[2], NULL, 0);
   bool new_style = argc > 3;
 
   int fd = open (name, O_RDONLY);
   Dwarf *dbg = dwarf_begin (fd, DWARF_C_READ);
 
-  Dwarf_Die cudie_mem, *cudie = dwarf_offdie (dbg, cuoff, &cudie_mem);
-
-  for (ptrdiff_t off = new_style ? DWARF_GETMACROS_START : 0;
-   (off = dwarf_getmacros (cudie, mac, dbg, off)); )
-if (off == -1)
-  {
-   puts (dwarf_errmsg (dwarf_errno ()));
-   break;
-  }
+  if (argv[2][0] == '\0')
+{
+  Dwarf_CU *cu = NULL;
+  Dwarf_Die cudie, subdie;
+  uint8_t unit_type;
+  while (dwarf_get_units (dbg, cu, &cu, NULL,
+ &unit_type, &cudie, &subdie) == 0)
+   {
+ Dwarf_Die *die = (unit_type == DW_UT_skeleton
+   ? &subdie : &cudie);
+ if (! dwarf_hasattr (die, DW_AT_macro_info)
+ && ! dwarf_hasattr (die, DW_AT_GNU_macros)
+ && ! dwarf_hasattr (die, DW_AT_macros))
+   continue;
+ printf ("CU %s\n", dwarf_diename (die));
+ getmacros (dbg, die, new_style);
+   }
+}
+  else
+{
+  ptrdiff_t cuoff = strtol (argv[2], NULL, 0);
+  Dwarf_Die cudie_mem, *cudie = dwarf_offdie (dbg, cuoff, &cudie_mem);
+  if (cudie == NULL)
+   {
+ puts (dwarf_errmsg (-1));
+ return 1;
+   }
+  getmacros (dbg, cudie, new_style);
+}
 
   dwarf_end (dbg);
 
-- 
2.41.0

[PATCH 08/14] libdw: Parse DWARF package file index sections

[Omar Sandoval]

From: Omar Sandoval 

The .debug_cu_index and .debug_tu_index sections in DWARF package files
are basically hash tables mapping a unit's 8 byte signature to an offset
and size in each section used by that unit [1].  Add support for parsing
and doing lookups in the index sections.

We look up a unit in the index when we intern it and cache its hash
table row in Dwarf_CU.  Then, a new function, dwarf_cu_dwp_section_info,
can be used to look up the section offsets and sizes for a unit.  This
will mostly be used internally in libdw, but it will also be needed in
static inline functions shared with eu-readelf.  Additionally, making it
public it makes dwp support much easier for external tools that do their
own low-level parsing of DWARF information, like drgn [2].

1: 
https://gcc.gnu.org/wiki/DebugFissionDWP#Format_of_the_CU_and_TU_Index_Sections
2: https://github.com/osandov/drgn

Signed-off-by: Omar Sandoval 
---
 libdw/ChangeLog |  18 +-
 libdw/Makefile.am   |   2 +-
 libdw/dwarf_cu_dwp_section_info.c   | 371 
 libdw/dwarf_end.c   |   3 +
 libdw/dwarf_error.c |   1 +
 libdw/libdw.h   |  23 ++
 libdw/libdw.map |   5 +
 libdw/libdwP.h  |  36 +++
 libdw/libdw_findcu.c|   8 +
 tests/.gitignore|   1 +
 tests/ChangeLog |  13 +
 tests/Makefile.am   |  11 +-
 tests/cu-dwp-section-info.c |  74 ++
 tests/run-cu-dwp-section-info.sh| 168 +
 tests/testfile-dwp-4-strict.bz2 | Bin 0 -> 4169 bytes
 tests/testfile-dwp-4-strict.dwp.bz2 | Bin 0 -> 6871 bytes
 tests/testfile-dwp-4.bz2| Bin 0 -> 4194 bytes
 tests/testfile-dwp-4.dwp.bz2| Bin 0 -> 10098 bytes
 tests/testfile-dwp-5.bz2| Bin 0 -> 4223 bytes
 tests/testfile-dwp-5.dwp.bz2| Bin 0 -> 10313 bytes
 tests/testfile-dwp.source   | 102 
 21 files changed, 831 insertions(+), 5 deletions(-)
 create mode 100644 libdw/dwarf_cu_dwp_section_info.c
 create mode 100644 tests/cu-dwp-section-info.c
 create mode 100755 tests/run-cu-dwp-section-info.sh
 create mode 100755 tests/testfile-dwp-4-strict.bz2
 create mode 100644 tests/testfile-dwp-4-strict.dwp.bz2
 create mode 100755 tests/testfile-dwp-4.bz2
 create mode 100644 tests/testfile-dwp-4.dwp.bz2
 create mode 100755 tests/testfile-dwp-5.bz2
 create mode 100644 tests/testfile-dwp-5.dwp.bz2
 create mode 100644 tests/testfile-dwp.source

diff --git a/libdw/ChangeLog b/libdw/ChangeLog
index 52327688..1d229094 100644
--- a/libdw/ChangeLog
+++ b/libdw/ChangeLog
@@ -20,7 +20,14 @@
instead of dbg parameter, which is now unused.
* libdwP.h (Dwarf_Macro_Op_Table): Replace is_64bit with address_size
and offset_size.  Add dbg.
-   Add IDX_debug_cu_index and IDX_debug_tu_index.
+   (Dwarf): Add cu_index and tu_index.
+   (Dwarf_CU): Add dwp_row.
+   (Dwarf_Package_Index): New type.
+   (DW_SECT_TYPES): New macro.
+   (__libdw_dwp_find_unit): New declaration.
+   (dwarf_cu_dwp_section_info): New INTDECL.
+   Add IDX_debug_cu_index and IDX_debug_tu_index.  Add
+   DWARF_E_UNKNOWN_SECTION.
* dwarf_begin_elf.c (dwarf_scnnames): Add IDX_debug_cu_index and
IDX_debug_tu_index.
(scn_to_string_section_idx): Ditto.
@@ -28,6 +35,15 @@
.zdebug_cu_index, and .zdebug_tu_index.
(check_section): Change .dwo suffix matching to account for
.debug_cu_index and .debug_tu_index.
+   * Makefile.am (libdw_a_SOURCES): Add dwarf_cu_dwp_section_info.c.
+   * dwarf_end.c (dwarf_end): Free dwarf->cu_index and dwarf->tu_index.
+   * dwarf_error.c (errmsgs): Add DWARF_E_UNKNOWN_SECTION.
+   * libdw.h (dwarf_cu_dwp_section_info): New declaration.
+   * libdw.map (ELFUTILS_0.190): Add dwarf_cu_dwp_section_info.
+   * libdw_findcu.c (__libdw_intern_next_unit): Call
+   __libdw_dwp_find_unit, and use it to adjust abbrev_offset and assign
+   newp->dwp_row.
+   * dwarf_cu_dwp_section_info.c: New file.
 
 2023-02-22  Mark Wielaard  
 
diff --git a/libdw/Makefile.am b/libdw/Makefile.am
index e548f38c..5363c02a 100644
--- a/libdw/Makefile.am
+++ b/libdw/Makefile.am
@@ -93,7 +93,7 @@ libdw_a_SOURCES = dwarf_begin.c dwarf_begin_elf.c dwarf_end.c 
dwarf_getelf.c \
  dwarf_cu_die.c dwarf_peel_type.c dwarf_default_lower_bound.c \
  dwarf_die_addr_die.c dwarf_get_units.c \
  libdw_find_split_unit.c dwarf_cu_info.c \
- dwarf_next_lines.c
+ dwarf_next_lines.c dwarf_cu_dwp_section_info.c
 
 if MAINTAINER_MODE
 BUILT_SOURCES = $(srcdir)/known-dwarf.h
diff --git a/libdw/dwarf_cu_dwp_section_info.c 
b/libdw/dwarf_cu_dwp_section_info.c
new file mode 100644
index ..6766fb9a
--- /dev/null
+++ b/libdw/dwarf_cu_dwp_section_i

[PATCH 13/14] libdw: Apply DWARF package file section offsets where appropriate

[Omar Sandoval]

From: Omar Sandoval 

The final piece of DWARF package file support is that offsets have to be
interpreted relative to the section offset from the package index.
.debug_abbrev.dwo is already covered, so sprinkle around calls to
dwarf_cu_dwp_section_info for the remaining sections: .debug_line.dwo,
.debug_loclists.dwo/.debug_loc.dwo, .debug_str_offsets.dwo,
.debug_macro.dwo/.debug_macinfo.dwo, and .debug_rnglists.dwo.  With all
of that in place, we can finally test various libdw functions on dwp
files.

Signed-off-by: Omar Sandoval 
---
 libdw/ChangeLog   |   10 +-
 libdw/dwarf_getlocation.c |6 +
 libdw/dwarf_getmacros.c   |   26 +-
 libdw/libdwP.h|   38 +-
 tests/ChangeLog   |8 +-
 tests/run-all-dwarf-ranges.sh |  114 +++
 tests/run-dwarf-getmacros.sh  | 1412 +
 tests/run-get-units-split.sh  |   18 +
 tests/run-varlocs.sh  |  112 +++
 9 files changed, 1726 insertions(+), 18 deletions(-)

diff --git a/libdw/ChangeLog b/libdw/ChangeLog
index f37d9411..08b2c3e6 100644
--- a/libdw/ChangeLog
+++ b/libdw/ChangeLog
@@ -12,10 +12,13 @@
* dwarf_getmacros.c (get_macinfo_table): Replace assignment of
table->is_64bit with assignments of table->address_size and
table->offset_size.  Assume default DW_AT_stmt_list of 0 for split
-   DWARF.  Set table->dbg.
+   DWARF.  Set table->dbg.  Call dwarf_cu_dwp_section_info and add offset
+   to line_offset.
(get_table_for_offset): Ditto.
(read_macros): Get fake CU offset_size from table->offset_size instead
of table->is_64bit.
+   (get_offset_from): Call dwarf_cu_dwp_section_info and add offset to
+   *retp.
* dwarf_macro_getsrcfiles.c (dwarf_macro_getsrcfiles): Get
address_size for __libdw_getsrclines from table->address_size instead
of table->is_64bit.  Get dbg for __libdw_getsrclines from table->dbg
@@ -36,6 +39,9 @@
(__libdw_set_debugdir): New declaration.
(__libdw_dwp_findcu_id): New declaration.
(__libdw_link_skel_split): Handle .debug_addr for dwp.
+   (str_offsets_base_off): Call dwarf_cu_dwp_section_info and add offset.
+   (__libdw_cu_ranges_base): Ditto.
+   (__libdw_cu_locs_base): Ditto.
* dwarf_begin_elf.c (dwarf_scnnames): Add IDX_debug_cu_index and
IDX_debug_tu_index.
(scn_to_string_section_idx): Ditto.
@@ -60,6 +66,8 @@
__libdw_dwp_find_unit, and use it to adjust abbrev_offset and assign
newp->dwp_row.
* dwarf_cu_dwp_section_info.c: New file.
+   * dwarf_getlocation.c (initial_offset): Call dwarf_cu_dwp_section_info
+   and add offset to start_offset.
 
 2023-02-22  Mark Wielaard  
 
diff --git a/libdw/dwarf_getlocation.c b/libdw/dwarf_getlocation.c
index 553fdc98..37b32fc1 100644
--- a/libdw/dwarf_getlocation.c
+++ b/libdw/dwarf_getlocation.c
@@ -812,6 +812,12 @@ initial_offset (Dwarf_Attribute *attr, ptrdiff_t *offset)
: DWARF_E_NO_DEBUG_LOCLISTS),
NULL, &start_offset) == NULL)
return -1;
+
+  Dwarf_Off loc_off;
+  if (INTUSE(dwarf_cu_dwp_section_info) (attr->cu, DW_SECT_LOCLISTS,
+&loc_off, NULL) != 0)
+   return -1;
+  start_offset += loc_off;
 }
 
   *offset = start_offset;
diff --git a/libdw/dwarf_getmacros.c b/libdw/dwarf_getmacros.c
index a3a78884..2667eb45 100644
--- a/libdw/dwarf_getmacros.c
+++ b/libdw/dwarf_getmacros.c
@@ -47,7 +47,15 @@ get_offset_from (Dwarf_Die *die, int name, Dwarf_Word *retp)
 return -1;
 
   /* Offset into the corresponding section.  */
-  return INTUSE(dwarf_formudata) (&attr, retp);
+  if (INTUSE(dwarf_formudata) (&attr, retp) != 0)
+return -1;
+
+  Dwarf_Off offset;
+  if (INTUSE(dwarf_cu_dwp_section_info) (die->cu, DW_SECT_MACRO, &offset, NULL)
+  != 0)
+return -1;
+  *retp += offset;
+  return 0;
 }
 
 static int
@@ -131,6 +139,14 @@ get_macinfo_table (Dwarf *dbg, Dwarf_Word macoff, 
Dwarf_Die *cudie)
   else if (cudie->cu->unit_type == DW_UT_split_compile
   && dbg->sectiondata[IDX_debug_line] != NULL)
 line_offset = 0;
+  if (line_offset != (Dwarf_Off) -1)
+{
+  Dwarf_Off dwp_offset;
+  if (INTUSE(dwarf_cu_dwp_section_info) (cudie->cu, DW_SECT_LINE,
+&dwp_offset, NULL) != 0)
+   return NULL;
+  line_offset += dwp_offset;
+}
 
   Dwarf_Macro_Op_Table *table = libdw_alloc (dbg, Dwarf_Macro_Op_Table,
 macinfo_data_size, 1);
@@ -188,6 +204,14 @@ get_table_for_offset (Dwarf *dbg, Dwarf_Word macoff,
if (unlikely (INTUSE(dwarf_formudata) (attr, &line_offset) != 0))
  return NULL;
 }
+  if (line_offset != (Dwarf_Off) -1 && cudie != NULL)
+{
+  

Re: [PATCH 00/14] elfutils: DWARF package (.dwp) file support

[Omar Sandoval]

On Wed, Sep 27, 2023 at 03:20:40PM -0400, Frank Ch. Eigler wrote:
> Hi -
> 
> > This patch series adds support for DWARF package files to libdw and the
> > elfutils tools. It supports the GNU DebugFission format for DWARF 4 [1]
> > and the format standardized in DWARF 5 (section 7.3.5 "DWARF Package
> > Files"). It supports both automatically opening the .dwp file for a
> > skeleton unit and examining the .dwp file alone, just like for .dwo
> > files. [...]
> 
> Does this work have any implications for debuginfod?

Until anyone using debuginfod wants to use DWARF package files, not
really. The problems that split DWARF solves aren't super relevant to
Linux distributions, so I don't expect that will happen any time soon.

If that changes, then debuginfod could add a /$build_id/debuginfo.dwp
endpoint. Note that dwp files don't contain a build ID note. They are
associated with the main debug file only by name. To quote the DWARF 5
standard: "a package file is typically placed in the same directory as
the application, and is given the same name with a '.dwp' extension."

Re: [PATCH 07/14] libdw: Recognize .debug_[ct]u_index sections in dwarf_elf_begin

[Omar Sandoval]

On Wed, Nov 01, 2023 at 03:03:57PM +0100, Mark Wielaard wrote:
> Hi Omar,
> 
> On Wed, 2023-09-27 at 11:20 -0700, Omar Sandoval wrote:
> > From: Omar Sandoval 
> > 
> > DWARF package (.dwp) files have a .debug_cu_index section and,
> > optionally, a .debug_tu_index section.  Add them to the list of DWARF
> > sections.
> > 
> > Unfortunately, it's not that simple: the other debug sections in a dwp
> > file have names ending with .dwo, which confuses the checks introduced
> > by commit 5b21e70216b8 ("libdw: dwarf_elf_begin should use either plain,
> > dwo or lto DWARF sections.").  So, we also have to special case
> > .debug_cu_index and .debug_tu_index in scn_dwarf_type and check_section
> > to treat them as TYPE_DWO sections.
> 
> This seems to work, but I wonder if we should have a specific TYPE_DWP?

I tried this, and it made check_section even more confusing (because
then we need a more complicated check than result->type == section
type). I came to the same conclusion that since this is internal for
now, it didn't really matter. Although maybe the name TYPE_SPLIT would
make more sense now rather than overloading TYPE_DWO.

When this becomes public, separate TYPE_DWO and TYPE_DWP types might be
nicer so that a hypothetical dwarf_get_type function could tell you
whether you got a dwo file or a dwp file.

> It doesn't really matter now, because the enum dwarf_type is only used
> internally. But I was hoping to extend the dwarf_begin interface with a
> flag so that you can open a DWARF as a specific type. For example there
> are single file split DWARF files. Which contain both "plain" and
> ".dwo" sections. Currently you can only open them as "plain", but there
> are actually two "views" of such files.
> 
> https://sourceware.org/bugzilla/show_bug.cgi?id=28573
> 
> Do you think there are reasons to open a file as either TYPE_DWO or
> TYPE_DWP? Or doesn't that not make sense?

If we were to treat a dwp file as a dwo file, I guess we'd have to
pretend it only contained one unit and ignore the rest, which I don't
think makes much sense. So even if we had separate types for them, I
don't know if we should allow that.

Re: [PATCH 08/14] libdw: Parse DWARF package file index sections

[Omar Sandoval]

On Thu, Nov 02, 2023 at 12:07:04AM +0100, Mark Wielaard wrote:
> Hi Omar,
> 
> On Wed, Sep 27, 2023 at 11:20:57AM -0700, Omar Sandoval wrote:
> > The .debug_cu_index and .debug_tu_index sections in DWARF package files
> > are basically hash tables mapping a unit's 8 byte signature to an offset
> > and size in each section used by that unit [1].  Add support for parsing
> > and doing lookups in the index sections.
> > 
> > We look up a unit in the index when we intern it and cache its hash
> > table row in Dwarf_CU.
> 
> This looks good. Thanks for the various testcases.
> 
> Do I understand correctly that binutils dwp only does the DWARF4 GNU
> extension and llvm-dwp only does the DWARF5 standardized format? Maybe
> we should create a eu-dwp that does both.

llvm-dwp can do both the DWARF4 GNU format and the DWARF5 format.
binutils dwp can only do DWARF4.

> It looks like you are very careful checking boundaries, but it would
> be bad to do some fuzzing on this.
> 
> > Then, a new function, dwarf_cu_dwp_section_info,
> > can be used to look up the section offsets and sizes for a unit.  This
> > will mostly be used internally in libdw, but it will also be needed in
> > static inline functions shared with eu-readelf.  Additionally, making it
> > public it makes dwp support much easier for external tools that do their
> > own low-level parsing of DWARF information, like drgn [2].
> 
> Although I am not against this new interface, I am not super
> enthousiastic about it.
> 
> There is one odd part, DW_SECT_TYPES should be defined in dwarf.h with
> the other DW_SECT constants, otherwise it cannot be used (unless you
> define it yourself to 2).

Yeah, I wasn't sure about adding it or not since it's not technically
defined in the DWARF5 standard. But if we can count on future DWARF
standards not reusing the value 2 (which we probably can), I agree that
it seems better to add it to dwarf.h.

> For eu-readelf we don't really need it being public, it already cheats
> a little and uses some (non-public) libdwP.h functions. That is
> actually not great either. So if there is a public function that is
> available that is actually preferred. But if it is just for
> eu-readelf, I don't think it is really needed. And it seems you
> haven't actually added the support to eu-readelf to print these
> offsets.

Right, eu-readelf only uses it indirectly via the static inline
functions modified in patch 13. It looks like eu-readelf dynamically
links to libdw (on Fedora, at least), so either some part of the dwp
implementation needs to be exported, or a big chunk of it needs to be
inlined in libdwP.h, right?

Either way, I'd still love to have this interface for drgn.

> It is fine to expose these offsets and sizes, but how exactly are you
> using them in drgn? It seems we don't have any other interfaces in
> libdw that you can then use these with.

Here's the branch I linked to in my cover letter:
https://github.com/osandov/drgn/tree/dwp. I need this interface for the
couple of places where drgn parses DWARF itself.

One of those places is in the global name indexing step drgn does at
startup. We do this by enumerating all CUs using libdw, then using our
own purpose-built DWARF parser to do a fast, parallelized scan.

Our bespoke parser needs to know the base of the abbreviation table and
the string offsets table, which is easy without dwp. This interface also
makes it easy with dwp. Without this interface, I'd need to reimplement
the CU index parser in drgn :(

> Can we split off this public interface from the rest of this patch?
> But then we also need to split off the tests. So maybe keep them together?

Yeah, I included the interface in this patch exactly so that I could
test the implementation in the same patch. I'm happy to split that up
however you'd prefer.

[snip]

> > +  Dwarf_Package_Index *index = malloc (sizeof (*index));
> > +  if (index == NULL)
> > +{
> > +  __libdw_seterrno (DWARF_E_NOMEM);
> > +  return NULL;
> > +}
> > +
> > +  index->dbg = dbg;
> > +  /* Set absent sections to UINT32_MAX.  */
> > +  memset (index->sections, 0xff, sizeof (index->sections));
> 
> OK, although I would have preferred a simple for loop and let the
> compiler optimize it.

Ok, I can fix that.

[snip]

> > +static int
> > +__libdw_dwp_section_info (Dwarf_Package_Index *index, uint32_t unit_row,
> > + unsigned int section, Dwarf_Off *offsetp,
> > + Dwarf_Off *sizep)
> > +{
> > +  if (index == NULL)
> > +return -1;
> > +  if (unit_row == 0)
> > +{
> > +  __libdw_seterrno (DWARF_E_INVALID_DWARF);
> > +  return -1;
> > +}
> 
> OK, but why isn't the caller checking this?

Partially becase it simplified callers like __libdw_dwp_findcu_id to not
require a separate check, and partially just to be defensive.

[snip]

Thanks for taking a look!

Re: [PATCH 09/14] libdw, libdwfl: Save original path of ELF file

[Omar Sandoval]

On Thu, Nov 02, 2023 at 06:04:27PM +0100, Mark Wielaard wrote:
> Hi Omar,
> 
> On Wed, 2023-09-27 at 11:20 -0700, Omar Sandoval wrote:
> > libdw and libdwfl currently save the path of the directory containing
> > the ELF file to use when searching for alt and dwo files.  To search for
> > dwp files, we need the file name too.  Add an elfpath field to Dwarf,
> > and set the debugdir field from it.  Also update libdwfl to set elfpath
> > and debugdir.
> 
> This looks good. We will need some locking around this code when we
> integrate the thread-safety work. But that should be pretty clear.
> 
> > 
> > Signed-off-by: Omar Sandoval 
> > ---
> >  libdw/ChangeLog| 11 ++-
> >  libdw/dwarf_begin_elf.c| 34 --
> >  libdw/dwarf_end.c  |  3 ++-
> >  libdw/libdwP.h | 12 ++--
> >  libdwfl/ChangeLog  |  9 +
> >  libdwfl/dwfl_module.c  |  2 +-
> >  libdwfl/dwfl_module_getdwarf.c | 11 +++
> >  libdwfl/libdwflP.h |  2 +-
> >  libdwfl/offline.c  |  4 ++--
> >  9 files changed, 62 insertions(+), 26 deletions(-)
> > 
> > diff --git a/libdw/ChangeLog b/libdw/ChangeLog
> > index 1d229094..f491587f 100644
> > --- a/libdw/ChangeLog
> > +++ b/libdw/ChangeLog
> > @@ -20,7 +20,7 @@
> > instead of dbg parameter, which is now unused.
> > * libdwP.h (Dwarf_Macro_Op_Table): Replace is_64bit with address_size
> > and offset_size.  Add dbg.
> > -   (Dwarf): Add cu_index and tu_index.
> > +   (Dwarf): Add cu_index and tu_index.  Add elfpath.
> > (Dwarf_CU): Add dwp_row.
> > (Dwarf_Package_Index): New type.
> > (DW_SECT_TYPES): New macro.
> > @@ -28,6 +28,9 @@
> > (dwarf_cu_dwp_section_info): New INTDECL.
> > Add IDX_debug_cu_index and IDX_debug_tu_index.  Add
> > DWARF_E_UNKNOWN_SECTION.
> > +   (__libdw_debugdir): Replace declaration with...
> > +   (__libdw_elfpath): New declaration.
> > +   (__libdw_set_debugdir): New declaration.
> > * dwarf_begin_elf.c (dwarf_scnnames): Add IDX_debug_cu_index and
> > IDX_debug_tu_index.
> > (scn_to_string_section_idx): Ditto.
> > @@ -35,8 +38,14 @@
> > .zdebug_cu_index, and .zdebug_tu_index.
> > (check_section): Change .dwo suffix matching to account for
> > .debug_cu_index and .debug_tu_index.
> > +   (__libdw_debugdir): Replace with..
> > +   (__libdw_elfpath): New function.
> > +   (__libdw_set_debugdir): New function.
> > +   (valid_p): Call __libdw_elfpath and __libdw_set_debugdir instead of
> > +   __libdw_debugdir.
> > * Makefile.am (libdw_a_SOURCES): Add dwarf_cu_dwp_section_info.c.
> > * dwarf_end.c (dwarf_end): Free dwarf->cu_index and dwarf->tu_index.
> > +   Free dwarf->elfpath.
> > * dwarf_error.c (errmsgs): Add DWARF_E_UNKNOWN_SECTION.
> > * libdw.h (dwarf_cu_dwp_section_info): New declaration.
> > * libdw.map (ELFUTILS_0.190): Add dwarf_cu_dwp_section_info.
> 
> I had to recreate the ChangeLog entry because we skipped your patch 08.
> In the future lets just move the ChangeLog Entry into the commit
> message (we just updated CONTRIBUTING to recommend this). That makes
> rebasing slightly

Thanks for updating that, that's so much easier to work with :)

Re: [PATCH 10/14] libdw: Try .dwp file in __libdw_find_split_unit()

[Omar Sandoval]

On Thu, Nov 02, 2023 at 08:56:14PM +0100, Mark Wielaard wrote:
> Hi Omar,
> 
> On Wed, Sep 27, 2023 at 11:20:59AM -0700, Omar Sandoval wrote:
> > Try opening the file in the location suggested by the standard (the
> > skeleton file name + ".dwp") and looking up the unit in the package
> > index.  The rest is similar to .dwo files, with slightly different
> > cleanup since a single Dwarf handle is shared.
> 
> This seems a good default given it is what the standard says.
> But do you know of any distro doing this?

I don't. As far as I know, Meta and Google are the only ones using dwp
widely, and we're putting it in the location recommended by the
standard. (I think some Rust tooling is starting to pick dwp up, too,
but I haven't looked into it much.)

> The case I am wondering about is for separate .debug files (which
> surprisingly isn't standardized). In that case this would be
> foobar.debug.dwz (and not foobar.dwz).
> 
> It might also be an idea to just create one file with both the
> skeletons and the .dwo and dwz index sections.
>  
> > Signed-off-by: Omar Sandoval 
> > ---
> >  libdw/ChangeLog   | 11 -
> >  libdw/dwarf_begin_elf.c   |  1 +
> >  libdw/dwarf_cu_dwp_section_info.c | 19 
> >  libdw/dwarf_end.c | 10 -
> >  libdw/libdwP.h| 16 ++-
> >  libdw/libdw_find_split_unit.c | 75 ---
> >  6 files changed, 122 insertions(+), 10 deletions(-)
> > 
> > diff --git a/libdw/ChangeLog b/libdw/ChangeLog
> > index f491587f..f37d9411 100644
> > --- a/libdw/ChangeLog
> > +++ b/libdw/ChangeLog
> > @@ -1,6 +1,8 @@
> >  2023-09-27  Omar Sandoval  
> >  
> > * libdw_find_split_unit.c (try_split_file): Make static.
> > +   (try_dwp_file): New function.
> > +   (__libdw_find_split_unit): Call try_dwp_file.
> > * dwarf_entrypc.c (dwarf_entrypc): Call dwarf_lowpc.
> > * dwarf_ranges.c (dwarf_ranges): Use skeleton ranges section for
> > skeleton units.
> > @@ -20,7 +22,8 @@
> > instead of dbg parameter, which is now unused.
> > * libdwP.h (Dwarf_Macro_Op_Table): Replace is_64bit with address_size
> > and offset_size.  Add dbg.
> > -   (Dwarf): Add cu_index and tu_index.  Add elfpath.
> > +   (Dwarf): Add cu_index and tu_index.  Add elfpath.  Add dwp_dwarf and
> > +   dwp_fd.
> > (Dwarf_CU): Add dwp_row.
> > (Dwarf_Package_Index): New type.
> > (DW_SECT_TYPES): New macro.
> > @@ -31,6 +34,8 @@
> > (__libdw_debugdir): Replace declaration with...
> > (__libdw_elfpath): New declaration.
> > (__libdw_set_debugdir): New declaration.
> > +   (__libdw_dwp_findcu_id): New declaration.
> > +   (__libdw_link_skel_split): Handle .debug_addr for dwp.
> > * dwarf_begin_elf.c (dwarf_scnnames): Add IDX_debug_cu_index and
> > IDX_debug_tu_index.
> > (scn_to_string_section_idx): Ditto.
> > @@ -43,9 +48,11 @@
> > (__libdw_set_debugdir): New function.
> > (valid_p): Call __libdw_elfpath and __libdw_set_debugdir instead of
> > __libdw_debugdir.
> > +   (dwarf_begin_elf): Initialize result->dwp_fd.
> > * Makefile.am (libdw_a_SOURCES): Add dwarf_cu_dwp_section_info.c.
> > * dwarf_end.c (dwarf_end): Free dwarf->cu_index and dwarf->tu_index.
> > -   Free dwarf->elfpath.
> > +   Free dwarf->elfpath.  Free dwarf->dwp_dwarf and close dwarf->dwp_fd.
> > +   (cu_free): Don't free split dbg if it is dwp_dwarf.
> > * dwarf_error.c (errmsgs): Add DWARF_E_UNKNOWN_SECTION.
> > * libdw.h (dwarf_cu_dwp_section_info): New declaration.
> > * libdw.map (ELFUTILS_0.190): Add dwarf_cu_dwp_section_info.
> > diff --git a/libdw/dwarf_begin_elf.c b/libdw/dwarf_begin_elf.c
> > index 323a91d0..ca2b7e2a 100644
> > --- a/libdw/dwarf_begin_elf.c
> > +++ b/libdw/dwarf_begin_elf.c
> > @@ -567,6 +567,7 @@ dwarf_begin_elf (Elf *elf, Dwarf_Cmd cmd, Elf_Scn 
> > *scngrp)
> >  
> >result->elf = elf;
> >result->alt_fd = -1;
> > +  result->dwp_fd = -1;
> >  
> >/* Initialize the memory handling.  Initial blocks are allocated on first
> >   actual allocation.  */
> > diff --git a/libdw/dwarf_cu_dwp_section_info.c 
> > b/libdw/dwarf_cu_dwp_section_info.c
> > index 6766fb9a..7bf08d9d 100644
> > --- a/libdw/dwarf_cu_dwp_section_info.c
> > +++ b/libdw/dwarf_cu_dwp_section_info.c
> > @@ -340,6 +340,25 @@ __libdw_dwp_find_unit (Dwarf *dbg, bool debug_types, 
> > Dwarf_Off off,
> >  

Re: [PATCH 13/14] libdw: Apply DWARF package file section offsets where appropriate

[Omar Sandoval]

On Thu, Nov 02, 2023 at 11:20:33PM +0100, Mark Wielaard wrote:
> Hi Omar,
> 
> On Wed, Sep 27, 2023 at 11:21:02AM -0700, Omar Sandoval wrote:
> > The final piece of DWARF package file support is that offsets have to be
> > interpreted relative to the section offset from the package index.
> > .debug_abbrev.dwo is already covered, so sprinkle around calls to
> > dwarf_cu_dwp_section_info for the remaining sections: .debug_line.dwo,
> > .debug_loclists.dwo/.debug_loc.dwo, .debug_str_offsets.dwo,
> > .debug_macro.dwo/.debug_macinfo.dwo, and .debug_rnglists.dwo.  With all
> > of that in place, we can finally test various libdw functions on dwp
> > files.
> 
> This looks good, but obviously needs some earlier patches.

[snip]

> > @@ -1222,18 +1223,22 @@ __libdw_cu_ranges_base (Dwarf_CU *cu)
> > }
> >else
> > {
> > + Dwarf_Off dwp_offset = 0;
> > + dwarf_cu_dwp_section_info (cu, DW_SECT_RNGLISTS, &dwp_offset, NULL);
> 
> Shouldn't we check there wasn't an error?

Yeah, I was a little confused by the error handling in these foo_base
functions. It doesn't seem like most of the callers are checking for
errors. The foo_base functions also seem to be treating invalid
attributes as if they don't exist (e.g., when dwarf_formudata returns an
error):

  if (dwarf_attr (&cu_die, DW_AT_rnglists_base, &attr) != NULL)
{
  Dwarf_Word off;
  if (dwarf_formudata (&attr, &off) == 0)
offset += off;
}

So I suppose the equivalent for this change would be

  Dwarf_Off dwp_offset = 0;
  if (dwarf_cu_dwp_section_info (cu, DW_SECT_RNGLISTS, &dwp_offset, NULL) == 0)
offset = dwp_offset;

[snip]

Thanks!

Re: [PATCH 00/14] elfutils: DWARF package (.dwp) file support

[Omar Sandoval]

On Fri, Nov 03, 2023 at 12:05:57AM +0100, Mark Wielaard wrote:
> Hi Omar,
> 
> On Wed, Sep 27, 2023 at 11:20:49AM -0700, Omar Sandoval wrote:
> > This patch series adds support for DWARF package files to libdw and the
> > elfutils tools. It supports the GNU DebugFission format for DWARF 4 [1]
> > and the format standardized in DWARF 5 (section 7.3.5 "DWARF Package
> > Files"). It supports both automatically opening the .dwp file for a
> > skeleton unit and examining the .dwp file alone, just like for .dwo
> > files.
> 
> I had hoped to review and apply all this for the 0.190 release
> tomorrow, but wasn't fast enough.

No problem, I'll be happy to see it in 0.191 instead.

> In general it looks very good and
> most has been applied as is. Except for the last patch I don't have
> real concerns, just a few questions (see the specific reviews).
> 
> > Patch 1 is a trivial cleanup I found while developing this. Patches 2-6
> > are fixes for unrelated bugs in split DWARF and/or DWARF 5 that I
> > encountered while adding test cases for this series.
> 
> These have all been applied some time ago.
> 
> > The actual dwp support is in patches 7-10 and 13, including test cases.
> 
> 7 and 9 have been applied already. 8 has some comments/nitpicks, I can
> probably easily be convinced to accept it. 10 looks good, but depends
> on 8, some questions, specificly about handling split type units. 13
> looks good, but depends on 8 and 10.

I replied to patch 8 with how drgn wants to use the new interface. I
also replied with a couple of questions about how you'd like me to split
up patch 8 and handle errors in a couple of the existing codepaths.

> > Patches 11 and 12 enable testing macro information in dwp files.
> 
> Both applied.
> 
> > Patch 14 adds support and tests for an LLVM extension to the dwp
> > format.
> 
> I have to think a bit more on this.

Yeah, the whole situation is very unfortunate. I tried to keep the
workaround as unintrusive as possible, so it wouldn't be the end of the
world if we had to carry our own patch for it. Of course, since LLVM
supports it upstream, I'd definitely prefer for elfutils to support it
upstream, too.

I'll rebase my remaining patches and address the comments so far while I
wait for your responses.

Thanks!

[PATCH v2 0/4] elfutils: DWARF package (.dwp) file support

[Omar Sandoval]

From: Omar Sandoval 

Hi,

This is version 2 of my patch series adding support for DWARF package
files to libdw and the elfutils tools. Version 1 is here [1].

Patches 1-3 add the main implementation and tests for dwp files.

Most of this support is internal to libdw, but patch 1 adds a new public
function, dwarf_cu_dwp_section_info. drgn's dwp branch [2] demonstrates
how that function will be used. Also see [3] for more context on why
drgn needs this.

Patch 4 adds support and tests for an LLVM extension to the dwp format.
The "extension" is ugly because of an oversight in the design of the
format that LLVM had to make the best of, but unfortunately it's
necessary for a lot of our use cases.

With this patch series, drgn's test suite passes against a Linux kernel
build using .dwp.

Changes from v1:

* Rebased on main and dropped patches that were already merged.
* Moved ChangeLog entries to commit messages.
* Updated version in libdw.map to 0.191.
* Moved DW_SECT_TYPES definition to dwarf.h.
* Added copyright years.
* Added error handling for dwarf_cu_dwp_section_info calls in
  str_offsets_base_off, __libdw_cu_ranges_base, and __libdw_cu_locs_base
* Changed memset initialization of index->sections to an explicit
  loop.
* Added comment explaining __libdw_link_skel_split change.

There were a couple of things that were mentioned in review that I
didn't change:

* I kept dwarf_cu_dwp_section_info in patch 1 instead of separating it
  into its own patch so that I could test the dwp index implementation
  in the same commit that I introduced it in.
* I didn't make try_dwp_file return an error since try_split_file that
  it's based on doesn't either.

Thanks!
Omar

1: https://sourceware.org/pipermail/elfutils-devel/2023q3/006410.html
2: https://github.com/osandov/drgn/tree/dwp
3: https://sourceware.org/pipermail/elfutils-devel/2023q4/006630.html

Omar Sandoval (4):
  libdw: Parse DWARF package file index sections
  libdw: Try .dwp file in __libdw_find_split_unit()
  libdw: Apply DWARF package file section offsets where appropriate
  libdw: Handle overflowed DW_SECT_INFO offsets in DWARF package file
indexes

 libdw/Makefile.am |2 +-
 libdw/dwarf.h |2 +-
 libdw/dwarf_begin_elf.c   |1 +
 libdw/dwarf_cu_dwp_section_info.c |  531 +++
 libdw/dwarf_end.c |   24 +-
 libdw/dwarf_error.c   |1 +
 libdw/dwarf_getlocation.c |6 +
 libdw/dwarf_getmacros.c   |   26 +-
 libdw/libdw.h |   23 +
 libdw/libdw.map   |5 +
 libdw/libdwP.h|  101 +-
 libdw/libdw_find_split_unit.c |   75 +-
 libdw/libdw_findcu.c  |8 +
 tests/.gitignore  |1 +
 tests/Makefile.am |   15 +-
 tests/cu-dwp-section-info.c   |   73 +
 tests/run-all-dwarf-ranges.sh |  114 ++
 tests/run-cu-dwp-section-info.sh  |  168 ++
 tests/run-dwarf-getmacros.sh  | 1412 +
 tests/run-get-units-split.sh  |   18 +
 tests/run-large-elf-file.sh   |  174 ++
 tests/run-varlocs.sh  |  112 ++
 tests/testfile-dwp-4-cu-index-overflow.bz2|  Bin 0 -> 4490 bytes
 .../testfile-dwp-4-cu-index-overflow.dwp.bz2  |  Bin 0 -> 5584 bytes
 tests/testfile-dwp-4-strict.bz2   |  Bin 0 -> 4169 bytes
 tests/testfile-dwp-4-strict.dwp.bz2   |  Bin 0 -> 6871 bytes
 tests/testfile-dwp-4.bz2  |  Bin 0 -> 4194 bytes
 tests/testfile-dwp-4.dwp.bz2  |  Bin 0 -> 10098 bytes
 tests/testfile-dwp-5-cu-index-overflow.bz2|  Bin 0 -> 4544 bytes
 .../testfile-dwp-5-cu-index-overflow.dwp.bz2  |  Bin 0 -> 5790 bytes
 tests/testfile-dwp-5.bz2  |  Bin 0 -> 4223 bytes
 tests/testfile-dwp-5.dwp.bz2  |  Bin 0 -> 10313 bytes
 tests/testfile-dwp-cu-index-overflow.source   |   86 +
 tests/testfile-dwp.source |  102 ++
 34 files changed, 3051 insertions(+), 29 deletions(-)
 create mode 100644 libdw/dwarf_cu_dwp_section_info.c
 create mode 100644 tests/cu-dwp-section-info.c
 create mode 100755 tests/run-cu-dwp-section-info.sh
 create mode 100755 tests/testfile-dwp-4-cu-index-overflow.bz2
 create mode 100644 tests/testfile-dwp-4-cu-index-overflow.dwp.bz2
 create mode 100755 tests/testfile-dwp-4-strict.bz2
 create mode 100644 tests/testfile-dwp-4-strict.dwp.bz2
 create mode 100755 tests/testfile-dwp-4.bz2
 create mode 100644 tests/testfile-dwp-4.dwp.bz2
 create mode 100755 tests/testfile-dwp-5-cu-index-overflow.bz2
 create mode 100644 tests/testfile-dwp-5-cu-index-overflow.dwp.bz2
 create mode 10

[PATCH v2 1/4] libdw: Parse DWARF package file index sections

[Omar Sandoval]

From: Omar Sandoval 

The .debug_cu_index and .debug_tu_index sections in DWARF package files
are basically hash tables mapping a unit's 8 byte signature to an offset
and size in each section used by that unit [1].  Add support for parsing
and doing lookups in the index sections.

We look up a unit in the index when we intern it and cache its hash
table row in Dwarf_CU.  Then, a new function, dwarf_cu_dwp_section_info,
can be used to look up the section offsets and sizes for a unit.  This
will mostly be used internally in libdw, but it will also be needed in
static inline functions shared with eu-readelf.  Additionally, making it
public it makes dwp support much easier for external tools that do their
own low-level parsing of DWARF information, like drgn [2].

1: 
https://gcc.gnu.org/wiki/DebugFissionDWP#Format_of_the_CU_and_TU_Index_Sections
2: https://github.com/osandov/drgn

* libdw/dwarf.h: Add DW_SECT_TYPES.
* libdw/libdwP.h (Dwarf): Add cu_index and tu_index.
(Dwarf_CU): Add dwp_row.
(Dwarf_Package_Index): New type.
(__libdw_dwp_find_unit): New declaration.
(dwarf_cu_dwp_section_info): New INTDECL.
Add DWARF_E_UNKNOWN_SECTION.
* libdw/Makefile.am (libdw_a_SOURCES): Add
dwarf_cu_dwp_section_info.c.
* libdw/dwarf_end.c (dwarf_end): Free dwarf->cu_index and
dwarf->tu_index.
* libdw/dwarf_error.c (errmsgs): Add DWARF_E_UNKNOWN_SECTION.
* libdw/libdw.h (dwarf_cu_dwp_section_info): New declaration.
* libdw/libdw.map (ELFUTILS_0.190): Add
dwarf_cu_dwp_section_info.
* libdw/libdw_findcu.c (__libdw_intern_next_unit): Call
__libdw_dwp_find_unit, and use it to adjust abbrev_offset and
assign newp->dwp_row.
* libdw/dwarf_cu_dwp_section_info.c: New file.
* tests/Makefile.am (check_PROGRAMS): Add cu-dwp-section-info.
(TESTS): Add run-cu-dwp-section-info.sh
(EXTRA_DIST): Add run-cu-dwp-section-info.sh and new test files.
(cu_dwp_section_info_LDADD): New variable.
* tests/cu-dwp-section-info.c: New test.
* tests/run-cu-dwp-section-info.sh: New test.
* tests/testfile-dwp-4-strict.bz2: New test file.
* tests/testfile-dwp-4-strict.dwp.bz2: New test file.
* tests/testfile-dwp-4.bz2: New test file.
* tests/testfile-dwp-4.dwp.bz2: New test file.
* tests/testfile-dwp-5.bz2: New test file.
* tests/testfile-dwp-5.dwp.bz2: New test file.
* tests/testfile-dwp.source: New file.

Signed-off-by: Omar Sandoval 
---
 libdw/Makefile.am   |   2 +-
 libdw/dwarf.h   |   2 +-
 libdw/dwarf_cu_dwp_section_info.c   | 371 
 libdw/dwarf_end.c   |   3 +
 libdw/dwarf_error.c |   1 +
 libdw/libdw.h   |  23 ++
 libdw/libdw.map |   5 +
 libdw/libdwP.h  |  33 +++
 libdw/libdw_findcu.c|   8 +
 tests/.gitignore|   1 +
 tests/Makefile.am   |  11 +-
 tests/cu-dwp-section-info.c |  73 ++
 tests/run-cu-dwp-section-info.sh| 168 +
 tests/testfile-dwp-4-strict.bz2 | Bin 0 -> 4169 bytes
 tests/testfile-dwp-4-strict.dwp.bz2 | Bin 0 -> 6871 bytes
 tests/testfile-dwp-4.bz2| Bin 0 -> 4194 bytes
 tests/testfile-dwp-4.dwp.bz2| Bin 0 -> 10098 bytes
 tests/testfile-dwp-5.bz2| Bin 0 -> 4223 bytes
 tests/testfile-dwp-5.dwp.bz2| Bin 0 -> 10313 bytes
 tests/testfile-dwp.source   | 102 
 20 files changed, 798 insertions(+), 5 deletions(-)
 create mode 100644 libdw/dwarf_cu_dwp_section_info.c
 create mode 100644 tests/cu-dwp-section-info.c
 create mode 100755 tests/run-cu-dwp-section-info.sh
 create mode 100755 tests/testfile-dwp-4-strict.bz2
 create mode 100644 tests/testfile-dwp-4-strict.dwp.bz2
 create mode 100755 tests/testfile-dwp-4.bz2
 create mode 100644 tests/testfile-dwp-4.dwp.bz2
 create mode 100755 tests/testfile-dwp-5.bz2
 create mode 100644 tests/testfile-dwp-5.dwp.bz2
 create mode 100644 tests/testfile-dwp.source

diff --git a/libdw/Makefile.am b/libdw/Makefile.am
index e548f38c..5363c02a 100644
--- a/libdw/Makefile.am
+++ b/libdw/Makefile.am
@@ -93,7 +93,7 @@ libdw_a_SOURCES = dwarf_begin.c dwarf_begin_elf.c dwarf_end.c 
dwarf_getelf.c \
  dwarf_cu_die.c dwarf_peel_type.c dwarf_default_lower_bound.c \
  dwarf_die_addr_die.c dwarf_get_units.c \
  libdw_find_split_unit.c dwarf_cu_info.c \
- dwarf_next_lines.c
+ dwarf_next_lines.c dwarf_cu_dwp_section_info.c
 
 if MAINTAINER_MODE
 BUILT_SOURCES = $(srcdir)/known-dwarf.h
diff --git a/libdw/dwarf.h b/libdw/dwarf.h
index b2e49db2..4be32de5 100644
--- a/libdw/dwarf.h
+++ b/libdw/dwarf.h
@@ -942,7 +942,7 @@ enum
 enum
   {
 DW_SECT_INFO = 1,
-/* Reserve

[PATCH v2 3/4] libdw: Apply DWARF package file section offsets where appropriate

[Omar Sandoval]

From: Omar Sandoval 

The final piece of DWARF package file support is that offsets have to be
interpreted relative to the section offset from the package index.
.debug_abbrev.dwo is already covered, so sprinkle around calls to
dwarf_cu_dwp_section_info for the remaining sections: .debug_line.dwo,
.debug_loclists.dwo/.debug_loc.dwo, .debug_str_offsets.dwo,
.debug_macro.dwo/.debug_macinfo.dwo, and .debug_rnglists.dwo.  With all
of that in place, we can finally test various libdw functions on dwp
files.

* libdw/dwarf_getmacros.c (get_macinfo_table): Call
dwarf_cu_dwp_section_info and add offset to line_offset.
(get_offset_from): Call dwarf_cu_dwp_section_info and add offset
to *retp.
* libdw/libdwP.h (str_offsets_base_off): Call
dwarf_cu_dwp_section_info and add offset.
(__libdw_cu_ranges_base): Ditto.
(__libdw_cu_locs_base): Ditto.
* libdw/dwarf_getlocation.c (initial_offset): Call
dwarf_cu_dwp_section_info and add offset to start_offset.
* tests/run-varlocs.sh: Check testfile-dwp-5 and testfile-dwp-4.
* tests/run-all-dwarf-ranges.sh: Check testfile-dwp-5 and
testfile-dwp-4.
* tests/run-dwarf-getmacros.sh: Check testfile-dwp-5 and
testfile-dwp-4-strict.
* tests/run-get-units-split.sh: Check testfile-dwp-5,
testfile-dwp-4, and testfile-dwp-4-strict.

Signed-off-by: Omar Sandoval 
---
 libdw/dwarf_getlocation.c |6 +
 libdw/dwarf_getmacros.c   |   26 +-
 libdw/libdwP.h|   42 +-
 tests/run-all-dwarf-ranges.sh |  114 +++
 tests/run-dwarf-getmacros.sh  | 1412 +
 tests/run-get-units-split.sh  |   18 +
 tests/run-varlocs.sh  |  112 +++
 7 files changed, 1715 insertions(+), 15 deletions(-)

diff --git a/libdw/dwarf_getlocation.c b/libdw/dwarf_getlocation.c
index 553fdc98..37b32fc1 100644
--- a/libdw/dwarf_getlocation.c
+++ b/libdw/dwarf_getlocation.c
@@ -812,6 +812,12 @@ initial_offset (Dwarf_Attribute *attr, ptrdiff_t *offset)
: DWARF_E_NO_DEBUG_LOCLISTS),
NULL, &start_offset) == NULL)
return -1;
+
+  Dwarf_Off loc_off;
+  if (INTUSE(dwarf_cu_dwp_section_info) (attr->cu, DW_SECT_LOCLISTS,
+&loc_off, NULL) != 0)
+   return -1;
+  start_offset += loc_off;
 }
 
   *offset = start_offset;
diff --git a/libdw/dwarf_getmacros.c b/libdw/dwarf_getmacros.c
index a3a78884..2667eb45 100644
--- a/libdw/dwarf_getmacros.c
+++ b/libdw/dwarf_getmacros.c
@@ -47,7 +47,15 @@ get_offset_from (Dwarf_Die *die, int name, Dwarf_Word *retp)
 return -1;
 
   /* Offset into the corresponding section.  */
-  return INTUSE(dwarf_formudata) (&attr, retp);
+  if (INTUSE(dwarf_formudata) (&attr, retp) != 0)
+return -1;
+
+  Dwarf_Off offset;
+  if (INTUSE(dwarf_cu_dwp_section_info) (die->cu, DW_SECT_MACRO, &offset, NULL)
+  != 0)
+return -1;
+  *retp += offset;
+  return 0;
 }
 
 static int
@@ -131,6 +139,14 @@ get_macinfo_table (Dwarf *dbg, Dwarf_Word macoff, 
Dwarf_Die *cudie)
   else if (cudie->cu->unit_type == DW_UT_split_compile
   && dbg->sectiondata[IDX_debug_line] != NULL)
 line_offset = 0;
+  if (line_offset != (Dwarf_Off) -1)
+{
+  Dwarf_Off dwp_offset;
+  if (INTUSE(dwarf_cu_dwp_section_info) (cudie->cu, DW_SECT_LINE,
+&dwp_offset, NULL) != 0)
+   return NULL;
+  line_offset += dwp_offset;
+}
 
   Dwarf_Macro_Op_Table *table = libdw_alloc (dbg, Dwarf_Macro_Op_Table,
 macinfo_data_size, 1);
@@ -188,6 +204,14 @@ get_table_for_offset (Dwarf *dbg, Dwarf_Word macoff,
if (unlikely (INTUSE(dwarf_formudata) (attr, &line_offset) != 0))
  return NULL;
 }
+  if (line_offset != (Dwarf_Off) -1 && cudie != NULL)
+{
+  Dwarf_Off dwp_offset;
+  if (INTUSE(dwarf_cu_dwp_section_info) (cudie->cu, DW_SECT_LINE,
+&dwp_offset, NULL) != 0)
+   return NULL;
+  line_offset += dwp_offset;
+}
 
   uint8_t address_size;
   if (cudie != NULL)
diff --git a/libdw/libdwP.h b/libdw/libdwP.h
index 54445886..64d86bbd 100644
--- a/libdw/libdwP.h
+++ b/libdw/libdwP.h
@@ -1108,25 +1108,30 @@ str_offsets_base_off (Dwarf *dbg, Dwarf_CU *cu)
cu = first_cu;
 }
 
+  Dwarf_Off off = 0;
   if (cu != NULL)
 {
   if (cu->str_off_base == (Dwarf_Off) -1)
{
+ Dwarf_Off dwp_offset;
+ if (dwarf_cu_dwp_section_info (cu, DW_SECT_STR_OFFSETS, &dwp_offset,
+NULL) == 0)
+   off = dwp_offset;
  Dwarf_Die cu_die = CUDIE(cu);
  Dwarf_Attribute attr;
  if (dwarf_attr (&cu_die, DW_AT_str_offsets_base, &attr) != NULL)

[PATCH v2 4/4] libdw: Handle overflowed DW_SECT_INFO offsets in DWARF package file indexes

[Omar Sandoval]

From: Omar Sandoval 

Meta uses DWARF package files for our large, statically-linked C++
applications.  Some of our largest applications have more than 4GB in
.debug_info.dwo, but the section offsets in .debug_cu_index and
.debug_tu_index are 32 bits; see the discussion here [1].  We
implemented a workaround/extension for this in LLVM.  Implement the
equivalent in libdw.

To test this, we need files with more than 4GB in .debug_info.dwo.  I
created these artificially by editing GCC's assembly output.  They
compress down to 6KB.  I test them from run-large-elf-file.sh to take
advantage of the existing checks for large file support.

1: https://discourse.llvm.org/t/dwarf-dwp-4gb-limit/63902.

* libdw/dwarf_end.c (dwarf_package_index_free): New function.
* tests/testfile-dwp-4-cu-index-overflow.bz2: New test file.
* tests/testfile-dwp-4-cu-index-overflow.dwp.bz2: New test file.
* tests/testfile-dwp-5-cu-index-overflow.bz2: New test file.
* tests/testfile-dwp-5-cu-index-overflow.dwp.bz2: New test file.
* tests/testfile-dwp-cu-index-overflow.source: New file.
* tests/run-large-elf-file.sh: Check
testfile-dwp-5-cu-index-overflow and
testfile-dwp-4-cu-index-overflow.

Signed-off-by: Omar Sandoval 
---
 libdw/dwarf_cu_dwp_section_info.c | 147 ++-
 libdw/dwarf_end.c |  15 +-
 libdw/libdwP.h|   3 +
 tests/Makefile.am |   6 +-
 tests/run-large-elf-file.sh   | 174 ++
 tests/testfile-dwp-4-cu-index-overflow.bz2| Bin 0 -> 4490 bytes
 .../testfile-dwp-4-cu-index-overflow.dwp.bz2  | Bin 0 -> 5584 bytes
 tests/testfile-dwp-5-cu-index-overflow.bz2| Bin 0 -> 4544 bytes
 .../testfile-dwp-5-cu-index-overflow.dwp.bz2  | Bin 0 -> 5790 bytes
 tests/testfile-dwp-cu-index-overflow.source   |  86 +
 10 files changed, 425 insertions(+), 6 deletions(-)
 create mode 100755 tests/testfile-dwp-4-cu-index-overflow.bz2
 create mode 100644 tests/testfile-dwp-4-cu-index-overflow.dwp.bz2
 create mode 100755 tests/testfile-dwp-5-cu-index-overflow.bz2
 create mode 100644 tests/testfile-dwp-5-cu-index-overflow.dwp.bz2
 create mode 100644 tests/testfile-dwp-cu-index-overflow.source

diff --git a/libdw/dwarf_cu_dwp_section_info.c 
b/libdw/dwarf_cu_dwp_section_info.c
index 298f36f9..3d11c87a 100644
--- a/libdw/dwarf_cu_dwp_section_info.c
+++ b/libdw/dwarf_cu_dwp_section_info.c
@@ -30,6 +30,8 @@
 # include 
 #endif
 
+#include 
+
 #include "libdwP.h"
 
 static Dwarf_Package_Index *
@@ -110,7 +112,9 @@ __libdw_read_package_index (Dwarf *dbg, bool tu)
 
   index->dbg = dbg;
   /* Set absent sections to UINT32_MAX.  */
-  memset (index->sections, 0xff, sizeof (index->sections));
+  for (size_t i = 0;
+   i < sizeof (index->sections) / sizeof (index->sections[0]); i++)
+index->sections[i] = UINT32_MAX;
   for (size_t i = 0; i < section_count; i++)
 {
   uint32_t section = read_4ubyte_unaligned (dbg, sections + i * 4);
@@ -161,6 +165,7 @@ __libdw_read_package_index (Dwarf *dbg, bool tu)
   index->indices = indices;
   index->section_offsets = section_offsets;
   index->section_sizes = section_sizes;
+  index->debug_info_offsets = NULL;
 
   return index;
 }
@@ -177,6 +182,137 @@ __libdw_package_index (Dwarf *dbg, bool tu)
   if (index == NULL)
 return NULL;
 
+  /* Offsets in the section offset table are 32-bit unsigned integers.  In
+ practice, the .debug_info.dwo section for very large executables can be
+ larger than 4GB.  GNU dwp as of binutils 2.41 and llvm-dwp before LLVM 15
+ both accidentally truncate offsets larger than 4GB.
+
+ LLVM 15 detects the overflow and errors out instead; see LLVM commit
+ f8df8114715b ("[DWP][DWARF] Detect and error on debug info offset
+ overflow").  However, lldb in LLVM 16 supports using dwp files with
+ truncated offsets by recovering them directly from the unit headers in the
+ .debug_info.dwo section; see LLVM commit c0db06227721 ("[DWARFLibrary] Add
+ support to re-construct cu-index").  Since LLVM 17, the overflow error can
+ be turned into a warning instead; see LLVM commit 53a483cee801 ("[DWP] add
+ overflow check for llvm-dwp tools if offset overflow").
+
+ LLVM's support for > 4GB offsets is effectively an extension to the DWARF
+ package file format, which we implement here.  The strategy is to walk the
+ unit headers in .debug_info.dwo in lockstep with the DW_SECT_INFO columns
+ in the section offset tables.  As long as they are in the same order
+ (which they are in practice for both GNU dwp and llvm-dwp), we can
+ correlate the truncated offset and produce a corrected array of offsets.
+
+ Note that this will be fixed properly in DWARF 6:
+ https://dwarfstd.org/issues

[PATCH v2 2/4] libdw: Try .dwp file in __libdw_find_split_unit()

[Omar Sandoval]

From: Omar Sandoval 

Try opening the file in the location suggested by the standard (the
skeleton file name + ".dwp") and looking up the unit in the package
index.  The rest is similar to .dwo files, with slightly different
cleanup since a single Dwarf handle is shared.

* libdw/libdw_find_split_unit.c (try_dwp_file): New function.
(__libdw_find_split_unit): Call try_dwp_file.
* libdw/libdwP.h (Dwarf): Add dwp_dwarf and dwp_fd.
(__libdw_dwp_findcu_id): New declaration.
(__libdw_link_skel_split): Handle .debug_addr for dwp.
* libdw/libdw_begin_elf.c (dwarf_begin_elf): Initialize
result->dwp_fd.
* libdw/dwarf_end.c (dwarf_end): Free dwarf->dwp_dwarf and close
dwarf->dwp_fd.
(cu_free): Don't free split dbg if it is dwp_dwarf.

Signed-off-by: Omar Sandoval 
---
 libdw/dwarf_begin_elf.c   |  1 +
 libdw/dwarf_cu_dwp_section_info.c | 19 
 libdw/dwarf_end.c | 10 -
 libdw/libdwP.h| 23 --
 libdw/libdw_find_split_unit.c | 75 ---
 5 files changed, 119 insertions(+), 9 deletions(-)

diff --git a/libdw/dwarf_begin_elf.c b/libdw/dwarf_begin_elf.c
index 323a91d0..ca2b7e2a 100644
--- a/libdw/dwarf_begin_elf.c
+++ b/libdw/dwarf_begin_elf.c
@@ -567,6 +567,7 @@ dwarf_begin_elf (Elf *elf, Dwarf_Cmd cmd, Elf_Scn *scngrp)
 
   result->elf = elf;
   result->alt_fd = -1;
+  result->dwp_fd = -1;
 
   /* Initialize the memory handling.  Initial blocks are allocated on first
  actual allocation.  */
diff --git a/libdw/dwarf_cu_dwp_section_info.c 
b/libdw/dwarf_cu_dwp_section_info.c
index 4a4eac8c..298f36f9 100644
--- a/libdw/dwarf_cu_dwp_section_info.c
+++ b/libdw/dwarf_cu_dwp_section_info.c
@@ -340,6 +340,25 @@ __libdw_dwp_find_unit (Dwarf *dbg, bool debug_types, 
Dwarf_Off off,
   abbrev_offsetp, NULL);
 }
 
+Dwarf_CU *
+internal_function
+__libdw_dwp_findcu_id (Dwarf *dbg, uint64_t unit_id8)
+{
+  Dwarf_Package_Index *index = __libdw_package_index (dbg, false);
+  uint32_t unit_row;
+  Dwarf_Off offset;
+  Dwarf_CU *cu;
+  if (__libdw_dwp_unit_row (index, unit_id8, &unit_row) == 0
+  && __libdw_dwp_section_info (index, unit_row, DW_SECT_INFO, &offset,
+  NULL) == 0
+  && (cu = __libdw_findcu (dbg, offset, false)) != NULL
+  && cu->unit_type == DW_UT_split_compile
+  && cu->unit_id8 == unit_id8)
+return cu;
+  else
+return NULL;
+}
+
 int
 dwarf_cu_dwp_section_info (Dwarf_CU *cu, unsigned int section,
   Dwarf_Off *offsetp, Dwarf_Off *sizep)
diff --git a/libdw/dwarf_end.c b/libdw/dwarf_end.c
index b7f817d9..78224ddb 100644
--- a/libdw/dwarf_end.c
+++ b/libdw/dwarf_end.c
@@ -66,7 +66,9 @@ cu_free (void *arg)
  /* The fake_addr_cu might be shared, only release one.  */
  if (p->dbg->fake_addr_cu == p->split->dbg->fake_addr_cu)
p->split->dbg->fake_addr_cu = NULL;
- INTUSE(dwarf_end) (p->split->dbg);
+ /* There is only one DWP file. We free it later.  */
+ if (p->split->dbg != p->dbg->dwp_dwarf)
+   INTUSE(dwarf_end) (p->split->dbg);
}
 }
 }
@@ -147,6 +149,12 @@ dwarf_end (Dwarf *dwarf)
  close (dwarf->alt_fd);
}
 
+  if (dwarf->dwp_fd != -1)
+   {
+ INTUSE(dwarf_end) (dwarf->dwp_dwarf);
+ close (dwarf->dwp_fd);
+   }
+
   /* The cached path and dir we found the Dwarf ELF file in.  */
   free (dwarf->elfpath);
   free (dwarf->debugdir);
diff --git a/libdw/libdwP.h b/libdw/libdwP.h
index 7f8d69b5..54445886 100644
--- a/libdw/libdwP.h
+++ b/libdw/libdwP.h
@@ -180,6 +180,9 @@ struct Dwarf
   /* dwz alternate DWARF file.  */
   Dwarf *alt_dwarf;
 
+  /* DWARF package file.  */
+  Dwarf *dwp_dwarf;
+
   /* The section data.  */
   Elf_Data *sectiondata[IDX_last];
 
@@ -197,6 +200,9 @@ struct Dwarf
  close this file descriptor.  */
   int alt_fd;
 
+  /* File descriptor of DWARF package file.  */
+  int dwp_fd;
+
   /* Information for traversing the .debug_pubnames section.  This is
  an array and separately allocated with malloc.  */
   struct pubnames_s
@@ -716,6 +722,10 @@ extern int __libdw_dwp_find_unit (Dwarf *dbg, bool 
debug_types, Dwarf_Off off,
  Dwarf_Off *abbrev_offsetp)
  __nonnull_attribute__ (1, 7, 8) internal_function;
 
+/* Find the compilation unit in a DWARF package file with the given id.  */
+extern Dwarf_CU *__libdw_dwp_findcu_id (Dwarf *dbg, uint64_t unit_id8)
+ __nonnull_attribute__ (1) internal_function;
+
 /* Get abbreviation with given code.  */
 extern Dwarf_Abbrev *__libdw_findabbrev (struct Dwarf_CU *cu,
 unsigned int code)
@@ -1367,12 +1377,19 @@ __libdw_link_skel_

Re: [PATCH v2 0/4] elfutils: DWARF package (.dwp) file support

[Omar Sandoval]

On Wed, Dec 06, 2023 at 01:22:15AM -0800, Omar Sandoval wrote:
> From: Omar Sandoval 
> 
> Hi,
> 
> This is version 2 of my patch series adding support for DWARF package
> files to libdw and the elfutils tools. Version 1 is here [1].
> 
> Patches 1-3 add the main implementation and tests for dwp files.
> 
> Most of this support is internal to libdw, but patch 1 adds a new public
> function, dwarf_cu_dwp_section_info. drgn's dwp branch [2] demonstrates
> how that function will be used. Also see [3] for more context on why
> drgn needs this.
> 
> Patch 4 adds support and tests for an LLVM extension to the dwp format.
> The "extension" is ugly because of an oversight in the design of the
> format that LLVM had to make the best of, but unfortunately it's
> necessary for a lot of our use cases.
> 
> With this patch series, drgn's test suite passes against a Linux kernel
> build using .dwp.
> 
> Changes from v1:
> 
> * Rebased on main and dropped patches that were already merged.
> * Moved ChangeLog entries to commit messages.
> * Updated version in libdw.map to 0.191.
> * Moved DW_SECT_TYPES definition to dwarf.h.
> * Added copyright years.
> * Added error handling for dwarf_cu_dwp_section_info calls in
>   str_offsets_base_off, __libdw_cu_ranges_base, and __libdw_cu_locs_base
> * Changed memset initialization of index->sections to an explicit
>   loop.
> * Added comment explaining __libdw_link_skel_split change.
> 
> There were a couple of things that were mentioned in review that I
> didn't change:
> 
> * I kept dwarf_cu_dwp_section_info in patch 1 instead of separating it
>   into its own patch so that I could test the dwp index implementation
>   in the same commit that I introduced it in.
> * I didn't make try_dwp_file return an error since try_split_file that
>   it's based on doesn't either.
> 
> Thanks!
> Omar
> 
> 1: https://sourceware.org/pipermail/elfutils-devel/2023q3/006410.html
> 2: https://github.com/osandov/drgn/tree/dwp
> 3: https://sourceware.org/pipermail/elfutils-devel/2023q4/006630.html
> 
> Omar Sandoval (4):
>   libdw: Parse DWARF package file index sections
>   libdw: Try .dwp file in __libdw_find_split_unit()
>   libdw: Apply DWARF package file section offsets where appropriate
>   libdw: Handle overflowed DW_SECT_INFO offsets in DWARF package file
> indexes

Ping, and happy new year :)

Re: [PATCH] libdw: Update dwarf_cu_dwp_section_info documentation

[Omar Sandoval]

On Fri, Feb 16, 2024 at 02:34:18PM +0100, Mark Wielaard wrote:
> Update the documentation of dwarf_cu_dwp_section_info to make clear
> that the function only returns an error if the DWARF package file data
> couldn't be read or an unknown section constant is provided.  Missing
> DWP information for a given CU isn't an error and will set both OFFSET
> and SIZE to zero.  It also makes sure the documentation is < 76 chars
> wide.
> 
>   * libdw/libdw.h (dwarf_cu_dwp_section_info): Update docs.
> 
> Signed-off-by: Mark Wielaard 

This looks good to me, thanks Mark.

Re: [PATCH v2 3/4] libdw: Apply DWARF package file section offsets where appropriate

[Omar Sandoval]

On Fri, Feb 16, 2024 at 04:00:47PM +0100, Mark Wielaard wrote:
> Hi Omar,
> 
> On Wed, 2023-12-06 at 01:22 -0800, Omar Sandoval wrote:
> > The final piece of DWARF package file support is that offsets have to be
> > interpreted relative to the section offset from the package index.
> > .debug_abbrev.dwo is already covered, so sprinkle around calls to
> > dwarf_cu_dwp_section_info for the remaining sections: .debug_line.dwo,
> > .debug_loclists.dwo/.debug_loc.dwo, .debug_str_offsets.dwo,
> > .debug_macro.dwo/.debug_macinfo.dwo, and .debug_rnglists.dwo.  With all
> > of that in place, we can finally test various libdw functions on dwp
> > files.
> 
> So the offsets for DW_SECT_INFO, DW_SECT_TYPES and DW_SECT_ABBREV were
> already taken into account when setting up a cu from a dwp.
> 
> With this patch __libdw_cu_str_off_base/str_offsets_base_off handles
> DW_SECT_STR_OFFSETS which is used in dwarf_formstring and
> dwarf_getmacros.
> 
> __libdw_cu_ranges_base handles DW_SECT_RNGLISTS, which is used by
> dwarf_ranges. And __libdw_formptr has a special case for
> DW_FORM_sec_offset for IDX_debug_ranges && version < 5 && unit_type ==
> DW_UT_split_compile to also uses __libdw_cu_ranges_base.
> 
> __libdw_cu_locs_base handles DW_SECT_LOCLISTS which is used in
> dwarf_getlocation through initial_offset. I do wonder why the special
> case in __libdw_formptr isn't needed here too.
> 
> dwarf_getmacros handles DW_SECT_MACRO through get_offset_from. And when
> the macros need to refer to the line table, it also handles
> DW_SECT_LINE.
> 
> Don't we also need to handle DW_SECT_LINE in dwarf_getsrclines and
> dwarf_next_lines when looking for DW_AT_stmt_list?

.debug_line is the odd one out in split DWARF: the skeleton file
contains the full .debug_line, and the DWO or DWP files have a skeleton
.debug_line.dwo that only contains the directory and file name tables
(for DW_AT_file and macro info to reference). dwarf_getsrclines and co.
read from the skeleton file, not the DWP file, meaning they shouldn't
use DW_SECT_LINE.

> > * libdw/dwarf_getmacros.c (get_macinfo_table): Call
> > dwarf_cu_dwp_section_info and add offset to line_offset.
> > (get_offset_from): Call dwarf_cu_dwp_section_info and add offset
> > to *retp.
> > * libdw/libdwP.h (str_offsets_base_off): Call
> > dwarf_cu_dwp_section_info and add offset.
> > (__libdw_cu_ranges_base): Ditto.
> > (__libdw_cu_locs_base): Ditto.
> > * libdw/dwarf_getlocation.c (initial_offset): Call
> > dwarf_cu_dwp_section_info and add offset to start_offset.
> > * tests/run-varlocs.sh: Check testfile-dwp-5 and testfile-dwp-4.
> > * tests/run-all-dwarf-ranges.sh: Check testfile-dwp-5 and
> > testfile-dwp-4.
> > * tests/run-dwarf-getmacros.sh: Check testfile-dwp-5 and
> > testfile-dwp-4-strict.
> > * tests/run-get-units-split.sh: Check testfile-dwp-5,
> > testfile-dwp-4, and testfile-dwp-4-strict.
> 
> The code and tests look good. run-varlocs.sh seems good, which seems to
> confirm DW_SECT_LOCLISTS is handled correctly (but why doesn't it need
> a hack similar to ranges in __libdw_formptr?).

I think it's because ranges have the uniquely weird behavior in DWARF 4
GNU Debug Fission that DW_AT_GNU_ranges_base is in the skeleton file but
used to interpret the split file. This was cleaned up for DWARF 5 (as I
documented in commit c9c9ffae725009b192b40e2d89035f353ae7055f), and
there was no base attribute for location lists in DWARF 4, so it's not
applicable.

> We might want to add a test for run-next-lines.sh and run-next-
> files.sh?

Good idea, I'll send an updated version with those tests.

Thanks!
Omar

Re: [PATCH v2 3/4] libdw: Apply DWARF package file section offsets where appropriate

[Omar Sandoval]

On Thu, Feb 22, 2024 at 04:53:19PM -0800, Omar Sandoval wrote:
> On Fri, Feb 16, 2024 at 04:00:47PM +0100, Mark Wielaard wrote:
> > Hi Omar,
> > 
> > On Wed, 2023-12-06 at 01:22 -0800, Omar Sandoval wrote:
> > > The final piece of DWARF package file support is that offsets have to be
> > > interpreted relative to the section offset from the package index.
> > > .debug_abbrev.dwo is already covered, so sprinkle around calls to
> > > dwarf_cu_dwp_section_info for the remaining sections: .debug_line.dwo,
> > > .debug_loclists.dwo/.debug_loc.dwo, .debug_str_offsets.dwo,
> > > .debug_macro.dwo/.debug_macinfo.dwo, and .debug_rnglists.dwo.  With all
> > > of that in place, we can finally test various libdw functions on dwp
> > > files.
> > 
> > So the offsets for DW_SECT_INFO, DW_SECT_TYPES and DW_SECT_ABBREV were
> > already taken into account when setting up a cu from a dwp.
> > 
> > With this patch __libdw_cu_str_off_base/str_offsets_base_off handles
> > DW_SECT_STR_OFFSETS which is used in dwarf_formstring and
> > dwarf_getmacros.
> > 
> > __libdw_cu_ranges_base handles DW_SECT_RNGLISTS, which is used by
> > dwarf_ranges. And __libdw_formptr has a special case for
> > DW_FORM_sec_offset for IDX_debug_ranges && version < 5 && unit_type ==
> > DW_UT_split_compile to also uses __libdw_cu_ranges_base.
> > 
> > __libdw_cu_locs_base handles DW_SECT_LOCLISTS which is used in
> > dwarf_getlocation through initial_offset. I do wonder why the special
> > case in __libdw_formptr isn't needed here too.
> > 
> > dwarf_getmacros handles DW_SECT_MACRO through get_offset_from. And when
> > the macros need to refer to the line table, it also handles
> > DW_SECT_LINE.
> > 
> > Don't we also need to handle DW_SECT_LINE in dwarf_getsrclines and
> > dwarf_next_lines when looking for DW_AT_stmt_list?
> 
> .debug_line is the odd one out in split DWARF: the skeleton file
> contains the full .debug_line, and the DWO or DWP files have a skeleton
> .debug_line.dwo that only contains the directory and file name tables
> (for DW_AT_file and macro info to reference). dwarf_getsrclines and co.
> read from the skeleton file, not the DWP file, meaning they shouldn't
> use DW_SECT_LINE.

Ah, I guess you can call dwarf_getsrclines/dwarf_next_lines on the dwp
file itself, where DW_SECT_LINE may be applicable. I need to think about
that some more...

Re: [PATCH v2 3/4] libdw: Apply DWARF package file section offsets where appropriate

[Omar Sandoval]

On Sat, Feb 24, 2024 at 03:00:04PM +0100, Mark Wielaard wrote:
> Hi Omar,
> 
> On Thu, Feb 22, 2024 at 05:03:44PM -0800, Omar Sandoval wrote:
> > On Thu, Feb 22, 2024 at 04:53:19PM -0800, Omar Sandoval wrote:
> > > On Fri, Feb 16, 2024 at 04:00:47PM +0100, Mark Wielaard wrote:
> > > > Don't we also need to handle DW_SECT_LINE in dwarf_getsrclines and
> > > > dwarf_next_lines when looking for DW_AT_stmt_list?
> > > 
> > > .debug_line is the odd one out in split DWARF: the skeleton file
> > > contains the full .debug_line, and the DWO or DWP files have a skeleton
> > > .debug_line.dwo that only contains the directory and file name tables
> > > (for DW_AT_file and macro info to reference). dwarf_getsrclines and co.
> > > read from the skeleton file, not the DWP file, meaning they shouldn't
> > > use DW_SECT_LINE.
> > 
> > Ah, I guess you can call dwarf_getsrclines/dwarf_next_lines on the dwp
> > file itself, where DW_SECT_LINE may be applicable. I need to think about
> > that some more...
> 
> So reading the DWARF5 spec, it says a split DWARF CU/TU DIE may have a
> DW_AT_stmt_list, which are interpreted as relative to the base offset
> for .debug_line.dwo. And these tables contain only the directory and
> filename lists needed to interpret DW_AT_decl_file attributes in the
> debugging information entries. Actual line number tables remain in the
> .debug_line section, and remain in the relocatable object (.o) files.
> 
> So I think the intention is that the main .debug_line (skeleton)
> section does contain the actual line number table, but only those
> file/dir table entries that are referenced from that. Not any that are
> only referenced from the DW_AT_decl_file attributes (which should only
> appear in the split DWARF DIEs). Maybe in practice these overlap
> completely, so there is no savings and they are in practice
> identical. But I don't see anything in the spec that implies you
> should interpret a lineptr in the .debug_info.dwo as relative to the
> .debug_line section in the skeleton.

Sorry, my reply was confusing, and I hadn't checked all of the relevant
functions yet. Let me try again:

As you said, the skeleton file has a .debug_line with the actual line
number table, and the DWP file has a .debug_line.dwo with only
file/directory name tables. If we're reading from the skeleton file,
then we don't need to apply the DW_SECT_LINE offset.

dwarf_getsrclines in particular always uses the skeleton's .debug_line
even if called on a split CU, so it doesn't need to be updated.
dwarf_getsrcfiles and dwarf_next_lines can use the split
.debug_line.dwo, so they do need to be updated.

Updated patch series incoming...

Thanks,
Omar

[PATCH v3 0/4] elfutils: DWARF package (.dwp) file support

[Omar Sandoval]

From: Omar Sandoval 

Hi,

This is version 3 of my patch series adding support for DWARF package
files to libdw and the elfutils tools. Version 2 is here [1], and
version 1 is [here]. This version fixes handling of line number
information.

Patches 1 and 2 are new in this version. Patch 1 fixes dwarf_decl_file
for split DWARF (not just DWP). Patch 2 is primarly a refactor to
prepare for patch 3, but I also stumbled across a theoretical bug that
it fixes.

Patch 3 is the last piece of core DWP support. It was updated in this
version to handle DWP files in dwarf_getsrcfiles and dwarf_next_lines
and run the declfiles, get-lines, next-lines, get-files, and next-files
tests on DWP files.

Patch 4 is unchanged from version 2. It adds support and tests for an
LLVM extension to the dwp format. The "extension" is ugly because of an
oversight in the design of the format that LLVM had to make the best of,
but unfortunately it's necessary for a lot of our use cases.

Thanks!

Omar

1: https://sourceware.org/pipermail/elfutils-devel/2023q4/006718.html
2: https://sourceware.org/pipermail/elfutils-devel/2023q3/006410.html

Omar Sandoval (4):
  libdw: Handle split DWARF in dwarf_decl_file
  libdw: Refactor dwarf_next_lines and fix skipped CU
  libdw: Apply DWARF package file section offsets where appropriate
  libdw: Handle overflowed DW_SECT_INFO offsets in DWARF package file
indexes

 libdw/dwarf_cu_dwp_section_info.c |  147 +-
 libdw/dwarf_decl_file.c   |   30 +-
 libdw/dwarf_end.c |   15 +-
 libdw/dwarf_getlocation.c |6 +
 libdw/dwarf_getmacros.c   |   26 +-
 libdw/dwarf_getsrcfiles.c |   17 +-
 libdw/dwarf_next_lines.c  |   80 +-
 libdw/libdwP.h|   45 +-
 tests/.gitignore  |1 +
 tests/Makefile.am |   13 +-
 tests/declfiles.c |   90 ++
 tests/run-all-dwarf-ranges.sh |  114 ++
 tests/run-declfiles.sh|  323 
 tests/run-dwarf-getmacros.sh  | 1412 +
 tests/run-get-files.sh|  115 ++
 tests/run-get-lines.sh|  244 +++
 tests/run-get-units-split.sh  |   18 +
 tests/run-large-elf-file.sh   |  174 ++
 tests/run-next-files.sh   |  100 ++
 tests/run-next-lines.sh   |  242 +++
 tests/run-varlocs.sh  |  112 ++
 tests/testfile-dwp-4-cu-index-overflow.bz2|  Bin 0 -> 4490 bytes
 .../testfile-dwp-4-cu-index-overflow.dwp.bz2  |  Bin 0 -> 5584 bytes
 tests/testfile-dwp-5-cu-index-overflow.bz2|  Bin 0 -> 4544 bytes
 .../testfile-dwp-5-cu-index-overflow.dwp.bz2  |  Bin 0 -> 5790 bytes
 tests/testfile-dwp-cu-index-overflow.source   |   86 +
 26 files changed, 3313 insertions(+), 97 deletions(-)
 create mode 100644 tests/declfiles.c
 create mode 100755 tests/run-declfiles.sh
 create mode 100755 tests/testfile-dwp-4-cu-index-overflow.bz2
 create mode 100644 tests/testfile-dwp-4-cu-index-overflow.dwp.bz2
 create mode 100755 tests/testfile-dwp-5-cu-index-overflow.bz2
 create mode 100644 tests/testfile-dwp-5-cu-index-overflow.dwp.bz2
 create mode 100644 tests/testfile-dwp-cu-index-overflow.source

-- 
2.43.2

[PATCH v3 1/4] libdw: Handle split DWARF in dwarf_decl_file

[Omar Sandoval]

From: Omar Sandoval 

Calling dwarf_decl_file on a split DWARF DIE fails this assertion:

  dwarf_decl_file.c:72: dwarf_decl_file: Assertion `cu->files != NULL && 
cu->files != (void *) -1l' failed.

This is because dwarf_decl_file calls dwarf_getsrclines to populate
cu->files.  For normal units, cu->files is cached by dwarf_getsrclines
when it parses the line number information.  However, for split units,
the line number information is parsed for the skeleton unit, then copied
to the split unit's cu->lines.  Split units have their own file name
table, so cu->files is not copied.

The obvious solution is to use dwarf_getsrcfiles instead of relying on
implicit caching.

Also add a test case for dwarf_decl_file.

* libdw/dwarf_decl_file.c (dwarf_decl_file): Use
dwarf_getsrcfiles instead of dwarf_getsrclines.
* tests/Makefile.am (check_PROGRAMS): Add declfiles.
(TESTS): Add run-declfiles.sh.
(EXTRA_DIST): Add run-declfiles.sh.
(declfiles_LDADD): New variable.
* tests/declfiles.c: New test.
* tests/run-declfiles.sh: New test.

Signed-off-by: Omar Sandoval 
---
 libdw/dwarf_decl_file.c |  30 ++
 tests/.gitignore|   1 +
 tests/Makefile.am   |   8 +-
 tests/declfiles.c   |  90 
 tests/run-declfiles.sh  | 233 
 5 files changed, 335 insertions(+), 27 deletions(-)
 create mode 100644 tests/declfiles.c
 create mode 100755 tests/run-declfiles.sh

diff --git a/libdw/dwarf_decl_file.c b/libdw/dwarf_decl_file.c
index 7dde4af0..1b91a4e9 100644
--- a/libdw/dwarf_decl_file.c
+++ b/libdw/dwarf_decl_file.c
@@ -48,35 +48,17 @@ dwarf_decl_file (Dwarf_Die *die)
   &idx) != 0)
 return NULL;
 
-  /* Get the array of source files for the CU.  */
-  struct Dwarf_CU *cu = attr_mem.cu;
-  if (cu->lines == NULL)
-{
-  Dwarf_Lines *lines;
-  size_t nlines;
+  Dwarf_Files *files;
+  size_t nfiles;
+  if (INTUSE(dwarf_getsrcfiles) (&CUDIE (attr_mem.cu), &files, &nfiles) != 0)
+return NULL;
 
-  /* Let the more generic function do the work.  It'll create more
-data but that will be needed in an real program anyway.  */
-  (void) INTUSE(dwarf_getsrclines) (&CUDIE (cu), &lines, &nlines);
-  assert (cu->lines != NULL);
-}
-
-  if (cu->lines == (void *) -1l)
-{
-  /* If DW_AT_decl_file was present, there should be file information
-   available.  */
-  __libdw_seterrno (DWARF_E_INVALID_DWARF);
-  return NULL;
-}
-
-  assert (cu->files != NULL && cu->files != (void *) -1l);
-
-  if (idx >= cu->files->nfiles)
+  if (idx >= nfiles)
 {
   __libdw_seterrno (DWARF_E_INVALID_DWARF);
   return NULL;
 }
 
-  return cu->files->info[idx].name;
+  return files->info[idx].name;
 }
 OLD_VERSION (dwarf_decl_file, ELFUTILS_0.122)
diff --git a/tests/.gitignore b/tests/.gitignore
index d38f0f9e..0289959d 100644
--- a/tests/.gitignore
+++ b/tests/.gitignore
@@ -32,6 +32,7 @@
 /debugaltlink
 /debuginfod_build_id_find
 /debuglink
+/declfiles
 /deleted
 /dwarf-die-addr-die
 /dwarf-getmacros
diff --git a/tests/Makefile.am b/tests/Makefile.am
index d7c53144..3f80c451 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -63,7 +63,7 @@ check_PROGRAMS = arextract arsymtest newfile saridx scnnames 
sectiondump \
  getphdrnum leb128 read_unaligned \
  msg_tst system-elf-libelf-test system-elf-gelf-test \
  nvidia_extended_linemap_libdw elf-print-reloc-syms \
- cu-dwp-section-info \
+ cu-dwp-section-info declfiles \
  $(asm_TESTS)
 
 asm_TESTS = asm-tst1 asm-tst2 asm-tst3 asm-tst4 asm-tst5 \
@@ -213,7 +213,8 @@ TESTS = run-arextract.sh run-arsymtest.sh run-ar.sh newfile 
test-nlist \
$(asm_TESTS) run-disasm-bpf.sh run-low_high_pc-dw-form-indirect.sh \
run-nvidia-extended-linemap-libdw.sh 
run-nvidia-extended-linemap-readelf.sh \
run-readelf-dw-form-indirect.sh run-strip-largealign.sh \
-   run-readelf-Dd.sh run-dwfl-core-noncontig.sh run-cu-dwp-section-info.sh
+   run-readelf-Dd.sh run-dwfl-core-noncontig.sh run-cu-dwp-section-info.sh 
\
+   run-declfiles.sh
 
 if !BIARCH
 export ELFUTILS_DISABLE_BIARCH = 1
@@ -640,7 +641,7 @@ EXTRA_DIST = run-arextract.sh run-arsymtest.sh run-ar.sh \
 testfile-dwp-4.bz2 testfile-dwp-4.dwp.bz2 \
 testfile-dwp-4-strict.bz2 testfile-dwp-4-strict.dwp.bz2 \
 testfile-dwp-5.bz2 testfile-dwp-5.dwp.bz2 testfile-dwp.source \
-run-cu-dwp-section-info.sh
+run-cu-dwp-section-info.sh run-declfiles.sh
 
 
 if USE_VALGRIND
@@ -818,6 +819,7 @@ read_unaligned_LDADD = $(libelf) $(libdw)
 nvidia_extended_linemap_libdw_LDADD = $(libelf) $(libdw)
 elf_print_reloc_syms_LDADD = $(libelf)
 cu_dwp_section_i

[PATCH v3 3/4] libdw: Apply DWARF package file section offsets where appropriate

[Omar Sandoval]

From: Omar Sandoval 

The final piece of DWARF package file support is that offsets have to be
interpreted relative to the section offset from the package index.
.debug_abbrev.dwo is already covered, so sprinkle around calls to
dwarf_cu_dwp_section_info for the remaining sections: .debug_line.dwo,
.debug_loclists.dwo/.debug_loc.dwo, .debug_str_offsets.dwo,
.debug_macro.dwo/.debug_macinfo.dwo, and .debug_rnglists.dwo.  With all
of that in place, we can finally test various libdw functions on dwp
files.

* libdw/dwarf_getlocation.c (initial_offset): Call
dwarf_cu_dwp_section_info and add offset to start_offset.
* libdw/dwarf_getmacros.c (get_macinfo_table): Call
dwarf_cu_dwp_section_info and add offset to line_offset.
(get_offset_from): Call dwarf_cu_dwp_section_info and add offset
to *retp.
* libdw/dwarf_getsrcfiles.c (dwarf_getsrcfiles): Call
dwarf_cu_dwp_section_info and pass offset to
__libdw_getsrclines.
* libdw/dwarf_next_lines.c (dwarf_next_lines): Call
dwarf_cu_dwp_section_info and add offset to stmt_off.
* libdw/libdwP.h (str_offsets_base_off): Call
dwarf_cu_dwp_section_info and add offset.
(__libdw_cu_ranges_base): Ditto.
(__libdw_cu_locs_base): Ditto.
* tests/run-all-dwarf-ranges.sh: Check testfile-dwp-5 and
testfile-dwp-4.
* tests/run-declfiles.sh: Ditto.
* tests/run-get-lines.sh: Ditto.
* tests/run-next-lines.sh: Ditto.
* tests/run-varlocs.sh: Ditto.
* tests/run-get-files.sh: Check testfile-dwp-5,
testfile-dwp-5.dwp, testfile-dwp-4, and testfile-dwp-4.dwp
* tests/run-next-files.sh: Ditto.
* tests/run-dwarf-getmacros.sh: Check testfile-dwp-5 and
testfile-dwp-4-strict.
* tests/run-get-units-split.sh: Ditto.

Signed-off-by: Omar Sandoval 
---
 libdw/dwarf_getlocation.c |6 +
 libdw/dwarf_getmacros.c   |   26 +-
 libdw/dwarf_getsrcfiles.c |   17 +-
 libdw/dwarf_next_lines.c  |5 +
 libdw/libdwP.h|   42 +-
 tests/run-all-dwarf-ranges.sh |  114 +++
 tests/run-declfiles.sh|   90 +++
 tests/run-dwarf-getmacros.sh  | 1412 +
 tests/run-get-files.sh|  115 +++
 tests/run-get-lines.sh|  244 ++
 tests/run-get-units-split.sh  |   18 +
 tests/run-next-files.sh   |  100 +++
 tests/run-next-lines.sh   |  242 ++
 tests/run-varlocs.sh  |  112 +++
 14 files changed, 2522 insertions(+), 21 deletions(-)

diff --git a/libdw/dwarf_getlocation.c b/libdw/dwarf_getlocation.c
index 553fdc98..37b32fc1 100644
--- a/libdw/dwarf_getlocation.c
+++ b/libdw/dwarf_getlocation.c
@@ -812,6 +812,12 @@ initial_offset (Dwarf_Attribute *attr, ptrdiff_t *offset)
: DWARF_E_NO_DEBUG_LOCLISTS),
NULL, &start_offset) == NULL)
return -1;
+
+  Dwarf_Off loc_off;
+  if (INTUSE(dwarf_cu_dwp_section_info) (attr->cu, DW_SECT_LOCLISTS,
+&loc_off, NULL) != 0)
+   return -1;
+  start_offset += loc_off;
 }
 
   *offset = start_offset;
diff --git a/libdw/dwarf_getmacros.c b/libdw/dwarf_getmacros.c
index a3a78884..2667eb45 100644
--- a/libdw/dwarf_getmacros.c
+++ b/libdw/dwarf_getmacros.c
@@ -47,7 +47,15 @@ get_offset_from (Dwarf_Die *die, int name, Dwarf_Word *retp)
 return -1;
 
   /* Offset into the corresponding section.  */
-  return INTUSE(dwarf_formudata) (&attr, retp);
+  if (INTUSE(dwarf_formudata) (&attr, retp) != 0)
+return -1;
+
+  Dwarf_Off offset;
+  if (INTUSE(dwarf_cu_dwp_section_info) (die->cu, DW_SECT_MACRO, &offset, NULL)
+  != 0)
+return -1;
+  *retp += offset;
+  return 0;
 }
 
 static int
@@ -131,6 +139,14 @@ get_macinfo_table (Dwarf *dbg, Dwarf_Word macoff, 
Dwarf_Die *cudie)
   else if (cudie->cu->unit_type == DW_UT_split_compile
   && dbg->sectiondata[IDX_debug_line] != NULL)
 line_offset = 0;
+  if (line_offset != (Dwarf_Off) -1)
+{
+  Dwarf_Off dwp_offset;
+  if (INTUSE(dwarf_cu_dwp_section_info) (cudie->cu, DW_SECT_LINE,
+&dwp_offset, NULL) != 0)
+   return NULL;
+  line_offset += dwp_offset;
+}
 
   Dwarf_Macro_Op_Table *table = libdw_alloc (dbg, Dwarf_Macro_Op_Table,
 macinfo_data_size, 1);
@@ -188,6 +204,14 @@ get_table_for_offset (Dwarf *dbg, Dwarf_Word macoff,
if (unlikely (INTUSE(dwarf_formudata) (attr, &line_offset) != 0))
  return NULL;
 }
+  if (line_offset != (Dwarf_Off) -1 && cudie != NULL)
+{
+  Dwarf_Off dwp_offset;
+  if (INTUSE(dwarf_cu_dwp_section_info) (cudie->cu, DW_SECT_LINE,
+&dwp_offset, NULL) != 0)
+   return NULL;
+  line_offset += dwp_offset;
+

[PATCH v3 4/4] libdw: Handle overflowed DW_SECT_INFO offsets in DWARF package file indexes

[Omar Sandoval]

From: Omar Sandoval 

Meta uses DWARF package files for our large, statically-linked C++
applications.  Some of our largest applications have more than 4GB in
.debug_info.dwo, but the section offsets in .debug_cu_index and
.debug_tu_index are 32 bits; see the discussion here [1].  We
implemented a workaround/extension for this in LLVM.  Implement the
equivalent in libdw.

To test this, we need files with more than 4GB in .debug_info.dwo.  I
created these artificially by editing GCC's assembly output.  They
compress down to 6KB.  I test them from run-large-elf-file.sh to take
advantage of the existing checks for large file support.

1: https://discourse.llvm.org/t/dwarf-dwp-4gb-limit/63902.

* libdw/dwarf_end.c (dwarf_package_index_free): New function.
* tests/testfile-dwp-4-cu-index-overflow.bz2: New test file.
* tests/testfile-dwp-4-cu-index-overflow.dwp.bz2: New test file.
* tests/testfile-dwp-5-cu-index-overflow.bz2: New test file.
* tests/testfile-dwp-5-cu-index-overflow.dwp.bz2: New test file.
* tests/testfile-dwp-cu-index-overflow.source: New file.
* tests/run-large-elf-file.sh: Check
testfile-dwp-5-cu-index-overflow and
testfile-dwp-4-cu-index-overflow.

Signed-off-by: Omar Sandoval 
---
 libdw/dwarf_cu_dwp_section_info.c | 147 ++-
 libdw/dwarf_end.c |  15 +-
 libdw/libdwP.h|   3 +
 tests/Makefile.am |   7 +-
 tests/run-large-elf-file.sh   | 174 ++
 tests/testfile-dwp-4-cu-index-overflow.bz2| Bin 0 -> 4490 bytes
 .../testfile-dwp-4-cu-index-overflow.dwp.bz2  | Bin 0 -> 5584 bytes
 tests/testfile-dwp-5-cu-index-overflow.bz2| Bin 0 -> 4544 bytes
 .../testfile-dwp-5-cu-index-overflow.dwp.bz2  | Bin 0 -> 5790 bytes
 tests/testfile-dwp-cu-index-overflow.source   |  86 +
 10 files changed, 426 insertions(+), 6 deletions(-)
 create mode 100755 tests/testfile-dwp-4-cu-index-overflow.bz2
 create mode 100644 tests/testfile-dwp-4-cu-index-overflow.dwp.bz2
 create mode 100755 tests/testfile-dwp-5-cu-index-overflow.bz2
 create mode 100644 tests/testfile-dwp-5-cu-index-overflow.dwp.bz2
 create mode 100644 tests/testfile-dwp-cu-index-overflow.source

diff --git a/libdw/dwarf_cu_dwp_section_info.c 
b/libdw/dwarf_cu_dwp_section_info.c
index 298f36f9..3d11c87a 100644
--- a/libdw/dwarf_cu_dwp_section_info.c
+++ b/libdw/dwarf_cu_dwp_section_info.c
@@ -30,6 +30,8 @@
 # include 
 #endif
 
+#include 
+
 #include "libdwP.h"
 
 static Dwarf_Package_Index *
@@ -110,7 +112,9 @@ __libdw_read_package_index (Dwarf *dbg, bool tu)
 
   index->dbg = dbg;
   /* Set absent sections to UINT32_MAX.  */
-  memset (index->sections, 0xff, sizeof (index->sections));
+  for (size_t i = 0;
+   i < sizeof (index->sections) / sizeof (index->sections[0]); i++)
+index->sections[i] = UINT32_MAX;
   for (size_t i = 0; i < section_count; i++)
 {
   uint32_t section = read_4ubyte_unaligned (dbg, sections + i * 4);
@@ -161,6 +165,7 @@ __libdw_read_package_index (Dwarf *dbg, bool tu)
   index->indices = indices;
   index->section_offsets = section_offsets;
   index->section_sizes = section_sizes;
+  index->debug_info_offsets = NULL;
 
   return index;
 }
@@ -177,6 +182,137 @@ __libdw_package_index (Dwarf *dbg, bool tu)
   if (index == NULL)
 return NULL;
 
+  /* Offsets in the section offset table are 32-bit unsigned integers.  In
+ practice, the .debug_info.dwo section for very large executables can be
+ larger than 4GB.  GNU dwp as of binutils 2.41 and llvm-dwp before LLVM 15
+ both accidentally truncate offsets larger than 4GB.
+
+ LLVM 15 detects the overflow and errors out instead; see LLVM commit
+ f8df8114715b ("[DWP][DWARF] Detect and error on debug info offset
+ overflow").  However, lldb in LLVM 16 supports using dwp files with
+ truncated offsets by recovering them directly from the unit headers in the
+ .debug_info.dwo section; see LLVM commit c0db06227721 ("[DWARFLibrary] Add
+ support to re-construct cu-index").  Since LLVM 17, the overflow error can
+ be turned into a warning instead; see LLVM commit 53a483cee801 ("[DWP] add
+ overflow check for llvm-dwp tools if offset overflow").
+
+ LLVM's support for > 4GB offsets is effectively an extension to the DWARF
+ package file format, which we implement here.  The strategy is to walk the
+ unit headers in .debug_info.dwo in lockstep with the DW_SECT_INFO columns
+ in the section offset tables.  As long as they are in the same order
+ (which they are in practice for both GNU dwp and llvm-dwp), we can
+ correlate the truncated offset and produce a corrected array of offsets.
+
+ Note that this will be fixed properly in DWARF 6:
+ https://dwarfstd.org/issues

[PATCH v3 2/4] libdw: Refactor dwarf_next_lines and fix skipped CU

[Omar Sandoval]

From: Omar Sandoval 

dwarf_next_lines has two loops over CUs: one from the CU after the given
CU to the end, and one from the first CU up to _but not including_ the
given CU.  This means that the given CU is never checked.

This is unlikely to matter in practice since CUs usually correspond 1:1
with line number tables in the same order, but let's fix it anyways.
Refactoring it to one loop fixes the problem and simplifies the next
change to support DWARF package files.

* libdw/dwarf_next_lines.c (dwarf_next_lines): Refactor loops
over CUs into one loop.

Signed-off-by: Omar Sandoval 
---
 libdw/dwarf_next_lines.c | 75 +---
 1 file changed, 31 insertions(+), 44 deletions(-)

diff --git a/libdw/dwarf_next_lines.c b/libdw/dwarf_next_lines.c
index 74854ecd..6a9fe361 100644
--- a/libdw/dwarf_next_lines.c
+++ b/libdw/dwarf_next_lines.c
@@ -95,62 +95,49 @@ dwarf_next_lines (Dwarf *dbg, Dwarf_Off off,
 {
   /* We need to find the matching CU to get the comp_dir.  Use the
 given CU as hint where to start searching.  Normally it will
-be the next CU that has a statement list. */
+be the next CU that has a statement list.  If the CUs are in a
+different order from the line tables, then we do a linear
+search.  */
   Dwarf_CU *given_cu = *cu;
   Dwarf_CU *next_cu = given_cu;
-  bool found = false;
-  while (INTUSE(dwarf_get_units) (dbg, next_cu, &next_cu, NULL, NULL,
- &cudie, NULL) == 0)
+  bool restarted = false;
+  while (1)
{
+ if (restarted && next_cu == given_cu)
+   {
+ /* We checked all of the CUs and there was no match.  */
+ *cu = NULL;
+ break;
+   }
+ if (INTUSE(dwarf_get_units) (dbg, next_cu, &next_cu, NULL, NULL,
+  &cudie, NULL) != 0)
+   {
+ /* We didn't find the matching CU after the starting point.
+Check the CUs up to the starting point.  */
+ next_cu = NULL;
+ restarted = true;
+ continue;
+   }
+
+ Dwarf_Word stmt_off = 0;
  if (dwarf_hasattr (&cudie, DW_AT_stmt_list))
{
  Dwarf_Attribute attr;
- Dwarf_Word stmt_off;
  if (dwarf_formudata (dwarf_attr (&cudie, DW_AT_stmt_list, &attr),
-  &stmt_off) == 0
- && stmt_off == off)
-   {
- found = true;
- break;
-   }
+  &stmt_off) != 0)
+   continue;
}
- else if (off == 0
-  && (next_cu->unit_type == DW_UT_split_compile
-  || next_cu->unit_type == DW_UT_split_type))
+ /* Split units have an implicit offset of 0.  */
+ else if (next_cu->unit_type != DW_UT_split_compile
+  && next_cu->unit_type != DW_UT_split_type)
+   continue;
+
+ if (stmt_off == off)
{
- /* For split units (in .dwo files) there is only one table
-at offset zero (containing just the files, no lines).  */
- found = true;
+ *cu = next_cu;
  break;
}
}
-
-  if (!found && given_cu != NULL)
-   {
- /* The CUs might be in a different order from the line
-tables. Need to do a linear search (but stop at the given
-CU, since we already searched those.  */
- next_cu = NULL;
- while (INTUSE(dwarf_get_units) (dbg, next_cu, &next_cu, NULL, NULL,
- &cudie, NULL) == 0
-&& next_cu != given_cu)
-   {
- Dwarf_Attribute attr;
- Dwarf_Word stmt_off;
- if (dwarf_formudata (dwarf_attr (&cudie, DW_AT_stmt_list, &attr),
-  &stmt_off) == 0
- && stmt_off == off)
-   {
- found = true;
- break;
-   }
-   }
-   }
-
-  if (found)
-   *cu = next_cu;
-  else
-   *cu = NULL;
 }
   else
 *cu = NULL;
-- 
2.43.2

Re: [PATCH v3 4/4] libdw: Handle overflowed DW_SECT_INFO offsets in DWARF package file indexes

[Omar Sandoval]

On Fri, Mar 01, 2024 at 03:59:22PM +0100, Mark Wielaard wrote:
> Hi Omar,
> 
> On Mon, 2024-02-26 at 11:32 -0800, Omar Sandoval wrote:
> > Meta uses DWARF package files for our large, statically-linked C++
> > applications.  Some of our largest applications have more than 4GB in
> > .debug_info.dwo, but the section offsets in .debug_cu_index and
> > .debug_tu_index are 32 bits; see the discussion here [1].  We
> > implemented a workaround/extension for this in LLVM.  Implement the
> > equivalent in libdw.
> > 
> > To test this, we need files with more than 4GB in .debug_info.dwo.  I
> > created these artificially by editing GCC's assembly output.  They
> > compress down to 6KB.  I test them from run-large-elf-file.sh to take
> > advantage of the existing checks for large file support.
> > 
> > 1: https://discourse.llvm.org/t/dwarf-dwp-4gb-limit/63902.
> > 
> > * libdw/dwarf_end.c (dwarf_package_index_free): New function.
> > * tests/testfile-dwp-4-cu-index-overflow.bz2: New test file.
> > * tests/testfile-dwp-4-cu-index-overflow.dwp.bz2: New test file.
> > * tests/testfile-dwp-5-cu-index-overflow.bz2: New test file.
> > * tests/testfile-dwp-5-cu-index-overflow.dwp.bz2: New test file.
> > * tests/testfile-dwp-cu-index-overflow.source: New file.
> > * tests/run-large-elf-file.sh: Check
> > testfile-dwp-5-cu-index-overflow and
> > testfile-dwp-4-cu-index-overflow.
> 
> The hack is kind of horrible, but given that this doesn't really
> impacts "normal" dwp files and it does work with clang/lldb, lets just
> support it too.

Yup, I hate it, too. Thanks for merging this series and for the followup
fixes!

[PATCH 0/3] debuginfod: speed up extraction from kernel debuginfo packages by 200x

[Omar Sandoval]

From: Omar Sandoval 

drgn [1] currently uses debuginfod with great success for debugging
userspace processes.  However, for debugging the Linux kernel (drgn's
main use case), we have had some performance issues with debuginfod, so
we intentionally avoid using it.  Specifically, it sometimes takes over
a minute for debuginfod to respond to queries for vmlinux and kernel
modules (not including the actual download time).

The reason for the slowness is that Linux kernel debuginfo packages are
very large and contain lots of files.  To respond to a query for a Linux
kernel debuginfo file, debuginfod has to decompress and iterate through
the whole package until it finds that file.  If the file is towards the
end of the package, this can take a very long time.  This was previously
reported for vdso files [2][3], which debuginfod was able to mitigate
with improved caching and prefetching.  However, kernel modules are far
greater in number, vary drastically by hardware and workload, and can be
spread all over the package, so in practice I've still been seeing long
delays.  This was also discussed on the drgn issue tracker [4].

The fundamental limitation is that Linux packages, which are essentially
compressed archives with extra metadata headers, don't support random
access to specific files.  However, the multi-threaded xz compression
format does actually support random access.  And, luckily, the kernel
debuginfo packages on Fedora, Debian, and Ubuntu all happen to use
multi-threaded xz compression!

debuginfod can take advantage of this: when it scans a package, if it is
a seekable xz archive, it can save the uncompressed offset and size of
each file.  Then, when it needs a file, it can seek to that offset and
extract it from there.  This requires some understanding of the xz
format and low-level liblzma code, but the speedup is massive: where the
worst case was previously about 50 seconds just to find a file in a
kernel debuginfo package, with this change the worst case is 0.25
seconds, a ~200x improvement! This works for both .rpm and .deb files.

Patch 1 is a preparatory refactor.  Patch 2 implements saving the
uncompressed offsets and sizes in the database.  Patch 3 implements the
seekable xz extraction.

I tested this by requesting and verifying the digest of every file from
a few kernel debuginfo rpms and debs [5].

P.S. The biggest downside of this change is that it depends on a very
specific compression format.  I think this is something we should
formalize with Linux distributions: large debuginfo packages should use
a seekable format.  Currently, xz in multi-threaded mode is the only
option, but Zstandard also has an experimental seekable format that is
worth looking into [6].

1: https://github.com/osandov/drgn
2: https://sourceware.org/bugzilla/show_bug.cgi?id=29478
3: https://bugzilla.redhat.com/show_bug.cgi?id=1970578
4: https://github.com/osandov/drgn/pull/380
5: https://gist.github.com/osandov/89d521fdc6c9a07aa8bb0ebf91974346
6: https://github.com/facebook/zstd/tree/dev/contrib/seekable_format

Omar Sandoval (3):
  debuginfod: factor out common code for responding from an archive
  debuginfod: add archive entry size, mtime, and uncompressed offset to
database
  debuginfod: optimize extraction from seekable xz archives

 configure.ac  |   5 +
 debuginfod/Makefile.am|   2 +-
 debuginfod/debuginfod.cxx | 870 +++---
 3 files changed, 722 insertions(+), 155 deletions(-)

-- 
2.45.2

[PATCH 1/3] debuginfod: factor out common code for responding from an archive

[Omar Sandoval]

From: Omar Sandoval 

handle_buildid_r_match has two very similar branches where it optionally
extracts a section and then creates a microhttpd response.  In
preparation for adding a third one, factor it out into a function.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx | 213 +-
 1 file changed, 96 insertions(+), 117 deletions(-)

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 305edde8..2d709026 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -1965,6 +1965,81 @@ string canonicalized_archive_entry_pathname(struct 
archive_entry *e)
 }
 
 
+// NB: takes ownership of, and may reassign, fd.
+static struct MHD_Response*
+create_buildid_r_response (int64_t b_mtime0,
+   const string& b_source0,
+   const string& b_source1,
+   const string& section,
+   const string& ima_sig,
+   const char* tmppath,
+   int& fd,
+   off_t size,
+   time_t mtime,
+   const string& metric,
+   const struct timespec& extract_begin)
+{
+  if (tmppath != NULL)
+{
+  struct timespec extract_end;
+  clock_gettime (CLOCK_MONOTONIC, &extract_end);
+  double extract_time = (extract_end.tv_sec - extract_begin.tv_sec)
++ (extract_end.tv_nsec - extract_begin.tv_nsec)/1.e9;
+  fdcache.intern(b_source0, b_source1, tmppath, size, true, extract_time);
+}
+
+  if (!section.empty ())
+{
+  int scn_fd = extract_section (fd, b_mtime0,
+b_source0 + ":" + b_source1,
+section, extract_begin);
+  close (fd);
+  if (scn_fd >= 0)
+fd = scn_fd;
+  else
+{
+  if (verbose)
+obatched (clog) << "cannot find section " << section
+<< " for archive " << b_source0
+<< " file " << b_source1 << endl;
+  return 0;
+}
+
+  struct stat fs;
+  if (fstat (fd, &fs) < 0)
+{
+  close (fd);
+  throw libc_exception (errno,
+string ("fstat ") + b_source0 + string (" ") + section);
+}
+  size = fs.st_size;
+}
+
+  struct MHD_Response* r = MHD_create_response_from_fd (size, fd);
+  if (r == 0)
+{
+  if (verbose)
+obatched(clog) << "cannot create fd-response for " << b_source0 << 
endl;
+  close(fd);
+}
+  else
+{
+  inc_metric ("http_responses_total","result",metric);
+  add_mhd_response_header (r, "Content-Type", "application/octet-stream");
+  add_mhd_response_header (r, "X-DEBUGINFOD-SIZE", 
to_string(size).c_str());
+  add_mhd_response_header (r, "X-DEBUGINFOD-ARCHIVE", b_source0.c_str());
+  add_mhd_response_header (r, "X-DEBUGINFOD-FILE", b_source1.c_str());
+  if(!ima_sig.empty()) add_mhd_response_header(r, 
"X-DEBUGINFOD-IMASIGNATURE", ima_sig.c_str());
+  add_mhd_last_modified (r, mtime);
+  if (verbose > 1)
+obatched(clog) << "serving " << metric << " " << b_source0
+   << " file " << b_source1
+   << " section=" << section
+   << " IMA signature=" << ima_sig << endl;
+  /* libmicrohttpd will close fd. */
+}
+  return r;
+}
 
 static struct MHD_Response*
 handle_buildid_r_match (bool internal_req_p,
@@ -2142,57 +2217,15 @@ handle_buildid_r_match (bool internal_req_p,
   break; // branch out of if "loop", to try new libarchive fetch 
attempt
 }
 
-  if (!section.empty ())
-   {
- int scn_fd = extract_section (fd, fs.st_mtime,
-   b_source0 + ":" + b_source1,
-   section, extract_begin);
- close (fd);
- if (scn_fd >= 0)
-   fd = scn_fd;
- else
-   {
- if (verbose)
-   obatched (clog) << "cannot find section " << section
-   << " for archive " << b_source0
-   << " file " << b_source1 << endl;
- return 0;
-   }
-
- rc = fstat(fd, &fs);
- if (rc < 0)
-   {
- close (fd);
- throw libc_exception (errno,
-  

[PATCH 2/3] debuginfod: add archive entry size, mtime, and uncompressed offset to database

[Omar Sandoval]

From: Omar Sandoval 

In order to extract a file from a seekable archive, we need to know
where in the uncompressed archive the file data starts and its size.
Additionally, in order to populate the response headers, we need the
file modification time (since we won't be able to get it from the
archive metadata).  Add the size, modification time, and uncompressed
offset to the _r_de table and _query_d and _query_e views.  Note that
_r_de already has a column for the mtime of the archive itself, so that
one is renamed to mtime0 and the one for the entry is mtime1.

We need a little bit of liblzma magic to detect whether a file is
seekable.  If so, we populate the uncompressed_offset column, and
otherwise we set it to null.  size and mtime1 are populated
unconditionally for simplicity.

Before this change, the database for a single kernel debuginfo RPM
(kernel-debuginfo-6.9.6-200.fc40.x86_64.rpm) was about 15MB.  This
change increases that by about 70kB, only a 0.5% increase.

Signed-off-by: Omar Sandoval 
---
 configure.ac  |   5 +
 debuginfod/Makefile.am|   2 +-
 debuginfod/debuginfod.cxx | 234 +-
 3 files changed, 213 insertions(+), 28 deletions(-)

diff --git a/configure.ac b/configure.ac
index 24e68d94..9c5f7e51 100644
--- a/configure.ac
+++ b/configure.ac
@@ -441,8 +441,13 @@ eu_ZIPLIB(bzlib,BZLIB,bz2,BZ2_bzdopen,bzip2)
 # We need this since bzip2 doesn't have a pkgconfig file.
 BZ2_LIB="$LIBS"
 AC_SUBST([BZ2_LIB])
+save_LIBS="$LIBS"
+LIBS=
 eu_ZIPLIB(lzma,LZMA,lzma,lzma_auto_decoder,[LZMA (xz)])
+lzma_LIBS="$LIBS"
+LIBS="$lzma_LIBS $save_LIBS"
 AS_IF([test "x$with_lzma" = xyes], [LIBLZMA="liblzma"], [LIBLZMA=""])
+AC_SUBST([lzma_LIBS])
 AC_SUBST([LIBLZMA])
 eu_ZIPLIB(zstd,ZSTD,zstd,ZSTD_decompress,[ZSTD (zst)])
 AS_IF([test "x$with_zstd" = xyes], [LIBZSTD="libzstd"], [LIBLZSTD=""])
diff --git a/debuginfod/Makefile.am b/debuginfod/Makefile.am
index b74e3673..e199dc0c 100644
--- a/debuginfod/Makefile.am
+++ b/debuginfod/Makefile.am
@@ -70,7 +70,7 @@ bin_PROGRAMS += debuginfod-find
 endif
 
 debuginfod_SOURCES = debuginfod.cxx
-debuginfod_LDADD = $(libdw) $(libelf) $(libeu) $(libdebuginfod) $(argp_LDADD) 
$(fts_LIBS) $(libmicrohttpd_LIBS) $(sqlite3_LIBS) $(libarchive_LIBS) 
$(rpm_LIBS) $(jsonc_LIBS) $(libcurl_LIBS) -lpthread -ldl
+debuginfod_LDADD = $(libdw) $(libelf) $(libeu) $(libdebuginfod) $(argp_LDADD) 
$(fts_LIBS) $(libmicrohttpd_LIBS) $(sqlite3_LIBS) $(libarchive_LIBS) 
$(rpm_LIBS) $(jsonc_LIBS) $(libcurl_LIBS) $(lzma_LIBS) -lpthread -ldl
 
 debuginfod_find_SOURCES = debuginfod-find.c
 debuginfod_find_LDADD = $(libdw) $(libelf) $(libeu) $(libdebuginfod) 
$(argp_LDADD) $(fts_LIBS) $(jsonc_LIBS)
diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 2d709026..95a7d941 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -63,6 +63,10 @@ extern "C" {
 #undef __attribute__ /* glibc bug - rhbz 1763325 */
 #endif
 
+#ifdef USE_LZMA
+#include 
+#endif
+
 #include 
 #include 
 #include 
@@ -162,7 +166,7 @@ string_endswith(const string& haystack, const string& 
needle)
 
 
 // Roll this identifier for every sqlite schema incompatibility.
-#define BUILDIDS "buildids10"
+#define BUILDIDS "buildids11"
 
 #if SQLITE_VERSION_NUMBER >= 3008000
 #define WITHOUT_ROWID "without rowid"
@@ -239,15 +243,18 @@ static const char DEBUGINFOD_SQLITE_DDL[] =
   "debuginfo_p integer not null,\n"
   "executable_p integer not null,\n"
   "file integer not null,\n"
-  "mtime integer not null,\n"
+  "mtime0 integer not null,\n"
   "content integer not null,\n"
+  "size integer not null,\n"
+  "mtime1 integer not null,\n"
+  "uncompressed_offset integer,\n"
   "foreign key (file) references " BUILDIDS "_files(id) on update 
cascade on delete cascade,\n"
   "foreign key (content) references " BUILDIDS "_files(id) on update 
cascade on delete cascade,\n"
   "foreign key (buildid) references " BUILDIDS "_buildids(id) on 
update cascade on delete cascade,\n"
-  "primary key (buildid, debuginfo_p, executable_p, file, content, 
mtime)\n"
+  "primary key (buildid, debuginfo_p, executable_p, file, content, 
mtime0)\n"
   ") " WITHOUT_ROWID ";\n"
   // Index for faster delete by archive file identifier
-  "create index if not exists " BUILDIDS "_r_de_idx on " BUILDIDS "_r_de 
(file, mtime);\n"
+  "create index if not exists " BUILDIDS "_r_de_idx on " BUILDIDS "_r_de 
(file, mtime0);\n"
   // Index for metadata searches
   &quo

[PATCH 3/3] debuginfod: optimize extraction from seekable xz archives

[Omar Sandoval]

From: Omar Sandoval 

The kernel debuginfo packages on Fedora, Debian, and Ubuntu, and many of
their downstreams, are all compressed with xz in multi-threaded mode,
which allows random access.  We can use this to bypass the full archive
extraction and dramatically speed up kernel debuginfo requests (from ~50
seconds in the worst case to < 0.25 seconds).

This works because multi-threaded xz compression splits up the stream
into many independently compressed blocks.  The stream ends with an
index of blocks.  So, to seek to an offset, we find the block containing
that offset in the index and then decompress and throw away data until
we reach the offset within the block.  We can then decompress the
desired amount of data, possibly from subsequent blocks.  There's no
high-level API in liblzma to do this, but we can do it by stitching
together a few low-level APIs.

Since we now save the uncompressed offset and size of each archive file,
pass that information down so we can do the optimized extraction when
applicable.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx | 431 --
 1 file changed, 417 insertions(+), 14 deletions(-)

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 95a7d941..c3822be3 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -1971,6 +1971,16 @@ handle_buildid_f_match (bool internal_req_t,
 
 
 #ifdef USE_LZMA
+struct lzma_exception: public reportable_exception
+{
+  lzma_exception(int rc, const string& msg):
+// liblzma doesn't have a lzma_ret -> string conversion function, so just
+// report the value.
+reportable_exception(string ("lzma error: ") + msg + ": error " + 
to_string(rc)) {
+  inc_metric("error_count","lzma",to_string(rc));
+}
+};
+
 // Neither RPM nor deb files support seeking to a specific file in the package.
 // Instead, to extract a specific file, we normally need to read the archive
 // sequentially until we find the file.  This is very slow for files at the end
@@ -2094,12 +2104,352 @@ is_seekable_archive (const string& rps, struct 
archive* a)
   // The file is only seekable if it has more than one Block.
   return num_records > 1;
 }
+
+// Read the Index at the end of an xz file.
+static lzma_index*
+read_xz_index (int fd)
+{
+  off_t footer_pos = -LZMA_STREAM_HEADER_SIZE;
+  if (lseek (fd, footer_pos, SEEK_END) == -1)
+throw libc_exception (errno, "lseek");
+
+  uint8_t footer[LZMA_STREAM_HEADER_SIZE];
+  size_t footer_read = 0;
+  while (footer_read < sizeof (footer))
+{
+  ssize_t bytes_read = read (fd, footer + footer_read,
+ sizeof (footer) - footer_read);
+  if (bytes_read < 0)
+{
+  if (errno == EINTR)
+continue;
+  throw libc_exception (errno, "read");
+}
+  if (bytes_read == 0)
+throw reportable_exception ("truncated file");
+  footer_read += bytes_read;
+}
+
+  lzma_stream_flags stream_flags;
+  lzma_ret ret = lzma_stream_footer_decode (&stream_flags, footer);
+  if (ret != LZMA_OK)
+throw lzma_exception (ret, "lzma_stream_footer_decode");
+
+  if (lseek (fd, footer_pos - stream_flags.backward_size, SEEK_END) == -1)
+throw libc_exception (errno, "lseek");
+
+  lzma_stream strm = LZMA_STREAM_INIT;
+  lzma_index* index = NULL;
+  ret = lzma_index_decoder (&strm, &index, UINT64_MAX);
+  if (ret != LZMA_OK)
+throw lzma_exception (ret, "lzma_index_decoder");
+  defer_dtor strm_ender (&strm, lzma_end);
+
+  uint8_t in_buf[4096];
+  while (true)
+{
+  if (strm.avail_in == 0)
+{
+  ssize_t bytes_read = read (fd, in_buf, sizeof (in_buf));
+  if (bytes_read < 0)
+{
+  if (errno == EINTR)
+continue;
+  throw libc_exception (errno, "read");
+}
+  if (bytes_read == 0)
+throw reportable_exception ("truncated file");
+  strm.avail_in = bytes_read;
+  strm.next_in = in_buf;
+}
+
+ret = lzma_code (&strm, LZMA_RUN);
+if (ret == LZMA_STREAM_END)
+  break;
+else if (ret != LZMA_OK)
+  throw lzma_exception (ret, "lzma_code index");
+}
+
+  ret = lzma_index_stream_flags (index, &stream_flags);
+  if (ret != LZMA_OK)
+{
+  lzma_index_end (index, NULL);
+  throw lzma_exception (ret, "lzma_index_stream_flags");
+}
+  return index;
+}
+
+static void
+my_lzma_index_end (lzma_index* index)
+{
+  lzma_index_end (index, NULL);
+}
+
+static void
+free_lzma_block_filter_options (lzma_block* block)
+{
+  for (int i = 0; i < LZMA_FILTERS_MAX; i++)
+{
+  free (block->filters[i].options);
+  block->filters[i].options = NULL;
+  

Re: [PATCH 0/3] debuginfod: speed up extraction from kernel debuginfo packages by 200x

[Omar Sandoval]

On Thu, Jul 11, 2024 at 04:16:25PM -0400, Frank Ch. Eigler wrote:
> Hi, Omar -
> 
> Thanks.  I wish this sort of amazing kludge weren't necessary, but
> given that it helps, so be it.
> 
> I'd like to commend you on the effort needed to match your code up
> with the stylistic idiosyncracies of the debuginfod c++ code.  It
> looks just like the other code.  My only reservation is the schema
> change.  Reindexing some of our large repos takes WEEKS.  Here's a
> possible way to avoid that:
> 
> - Preserve the current BUILDID schema id and tables as is.
> 
> - Add a new table for the intra-archive coordinates.  Think of it like a 
> cache.
>   Index it with archive-file-name and content-file-name (source0, source1 
> IIRC).
> 
> - During a fetch out of the archive-file-name, check whether the new
>   table has a record for that file.  If yes, cache hit, go through to
>   the xz extraction stuff, winner!
> 
> - If not, try the is_seekable() check on the archive.  If it is true, we have 
> an
>   archive that should be seekable, but we don't have it in the intra-archive 
> cache.
>   So take this opportunity to index that archive (only), populate the cache 
> table,
>   as the archive is being extracted.  (No need to use the new cache data 
> then, since
>   we've just paid the effort of decompressing/reading the whole thing 
> already.)
> 
> - Need to confirm that during grooming, a disappeared
>   archive-file-name would also drop the corresponding intra-archive
>   rows.
> 
> - Heck, during grooming or scanning, maybe the tool could preemptively
>   do the intra-archive coordinate cache thing if it's not already
>   done, just to defeat the latency of doing it on demand.
> 
> 
> What do you think?

Hi, Frank,

I didn't realize how expensive reindexing could be, thank you for
pointing that out.  Your proposal makes sense to me, I'll rework this.

Thanks,
Omar

[PATCH v2 0/5] debuginfod: speed up extraction from kernel debuginfo packages by 200x

[Omar Sandoval]

From: Omar Sandoval 

This is v2 of my patch series optimizing debuginfod for kernel
debuginfo.  v1 is here [7].

The main change from v1 is reworking the database changes to be backward
compatible and therefore not require reindexing.

Patch 1 is a preparatory refactor.  Patch 2 makes the schema changes.
Patch 3 implements the seekable xz extraction.  Patch 4 populates the
table of seekable entries at scan time.  Patch 5 does it for
pre-existing files at request time.

Here is the background copied and pasted from v1:

drgn [1] currently uses debuginfod with great success for debugging
userspace processes.  However, for debugging the Linux kernel (drgn's
main use case), we have had some performance issues with debuginfod, so
we intentionally avoid using it.  Specifically, it sometimes takes over
a minute for debuginfod to respond to queries for vmlinux and kernel
modules (not including the actual download time).

The reason for the slowness is that Linux kernel debuginfo packages are
very large and contain lots of files.  To respond to a query for a Linux
kernel debuginfo file, debuginfod has to decompress and iterate through
the whole package until it finds that file.  If the file is towards the
end of the package, this can take a very long time.  This was previously
reported for vdso files [2][3], which debuginfod was able to mitigate
with improved caching and prefetching.  However, kernel modules are far
greater in number, vary drastically by hardware and workload, and can be
spread all over the package, so in practice I've still been seeing long
delays.  This was also discussed on the drgn issue tracker [4].

The fundamental limitation is that Linux packages, which are essentially
compressed archives with extra metadata headers, don't support random
access to specific files.  However, the multi-threaded xz compression
format does actually support random access.  And, luckily, the kernel
debuginfo packages on Fedora, Debian, and Ubuntu all happen to use
multi-threaded xz compression!

debuginfod can take advantage of this: when it scans a package, if it is
a seekable xz archive, it can save the uncompressed offset and size of
each file.  Then, when it needs a file, it can seek to that offset and
extract it from there.  This requires some understanding of the xz
format and low-level liblzma code, but the speedup is massive: where the
worst case was previously about 50 seconds just to find a file in a
kernel debuginfo package, with this change the worst case is 0.25
seconds, a ~200x improvement! This works for both .rpm and .deb files.

I tested this by requesting and verifying the digest of every file from
a few kernel debuginfo rpms and debs [5].

P.S. The biggest downside of this change is that it depends on a very
specific compression format that is only used by kernel packages
incidentally.  I think this is something we should formalize with Linux
distributions: large debuginfo packages should use a seekable format.
Currently, xz in multi-threaded mode is the only option, but Zstandard
also has an experimental seekable format that is worth looking into [6].

Thanks,
Omar

1: https://github.com/osandov/drgn
2: https://sourceware.org/bugzilla/show_bug.cgi?id=29478
3: https://bugzilla.redhat.com/show_bug.cgi?id=1970578
4: https://github.com/osandov/drgn/pull/380
5: https://gist.github.com/osandov/89d521fdc6c9a07aa8bb0ebf91974346
6: https://github.com/facebook/zstd/tree/dev/contrib/seekable_format
7: https://sourceware.org/pipermail/elfutils-devel/2024q3/007191.html

Omar Sandoval (5):
  debuginfod: factor out common code for responding from an archive
  debugifod: add new table and views for seekable archives
  debuginfod: optimize extraction from seekable xz archives
  debuginfod: populate _r_seekable on scan
  debuginfod: populate _r_seekable on request

 configure.ac  |   5 +
 debuginfod/Makefile.am|   2 +-
 debuginfod/debuginfod.cxx | 921 --
 3 files changed, 788 insertions(+), 140 deletions(-)

-- 
2.45.2

[PATCH v2 2/5] debugifod: add new table and views for seekable archives

[Omar Sandoval]

From: Omar Sandoval 

In order to extract a file from a seekable archive, we need to know
where in the uncompressed archive the file data starts and its size.
Additionally, in order to populate the response headers, we need the
file modification time (since we won't be able to get it from the
archive metadata).  Add a new table, _r_seekable, keyed on the archive
file id and entry file id and containing the size, offset, and mtime.
It also contains the compression type just in case new seekable formats
are supported in the future.

In order to search this table when we get a request, we need the file
ids available.  Add the ids to the _query_d and _query_e views, and
rename them to _query_d2 and _query_e2.

This schema change is backward compatible and doesn't require
reindexing.  _query_d2 and _query_e2 can be renamed back the next time
BUILDIDS needs to be bumped.

Before this change, the database for a single kernel debuginfo RPM
(kernel-debuginfo-6.9.6-200.fc40.x86_64.rpm) was about 15MB.  This
change increases that by about 70kB, only a 0.5% increase.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx | 34 --
 1 file changed, 24 insertions(+), 10 deletions(-)

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 2d709026..81512fec 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -265,25 +265,39 @@ static const char DEBUGINFOD_SQLITE_DDL[] =
   "foreign key (content) references " BUILDIDS "_files(id) on update 
cascade on delete cascade,\n"
   "primary key (content, file, mtime)\n"
   ") " WITHOUT_ROWID ";\n"
+  "create table if not exists " BUILDIDS "_r_seekable (\n" // seekable rpm 
contents
+  "file integer not null,\n"
+  "content integer not null,\n"
+  "type text not null,\n"
+  "size integer not null,\n"
+  "offset integer not null,\n"
+  "mtime integer not null,\n"
+  "foreign key (file) references " BUILDIDS "_files(id) on update 
cascade on delete cascade,\n"
+  "foreign key (content) references " BUILDIDS "_files(id) on update 
cascade on delete cascade,\n"
+  "primary key (file, content)\n"
+  ") " WITHOUT_ROWID ";\n"
   // create views to glue together some of the above tables, for webapi D 
queries
-  "create view if not exists " BUILDIDS "_query_d as \n"
+  // NB: _query_d2 and _query_e2 were added to replace _query_d and _query_e
+  // without updating BUILDIDS.  They can be renamed back the next time 
BUILDIDS
+  // is updated.
+  "create view if not exists " BUILDIDS "_query_d2 as \n"
   "select\n"
-  "b.hex as buildid, n.mtime, 'F' as sourcetype, f0.name as source0, 
n.mtime as mtime, null as source1\n"
+  "b.hex as buildid, 'F' as sourcetype, n.file as id0, f0.name as 
source0, n.mtime as mtime, null as id1, null as source1\n"
   "from " BUILDIDS "_buildids b, " BUILDIDS "_files_v f0, " BUILDIDS 
"_f_de n\n"
   "where b.id = n.buildid and f0.id = n.file and n.debuginfo_p = 1\n"
   "union all select\n"
-  "b.hex as buildid, n.mtime, 'R' as sourcetype, f0.name as source0, 
n.mtime as mtime, f1.name as source1\n"
+  "b.hex as buildid, 'R' as sourcetype, n.file as id0, f0.name as 
source0, n.mtime as mtime, n.content as id1, f1.name as source1\n"
   "from " BUILDIDS "_buildids b, " BUILDIDS "_files_v f0, " BUILDIDS 
"_files_v f1, " BUILDIDS "_r_de n\n"
   "where b.id = n.buildid and f0.id = n.file and f1.id = n.content and 
n.debuginfo_p = 1\n"
   ";"
   // ... and for E queries
-  "create view if not exists " BUILDIDS "_query_e as \n"
+  "create view if not exists " BUILDIDS "_query_e2 as \n"
   "select\n"
-  "b.hex as buildid, n.mtime, 'F' as sourcetype, f0.name as source0, 
n.mtime as mtime, null as source1\n"
+  "b.hex as buildid, 'F' as sourcetype, n.file as id0, f0.name as 
source0, n.mtime as mtime, null as id1, null as source1\n"
   "from " BUILDIDS "_buildids b, " BUILDIDS "_files_v f0, " BUILDIDS 
"_f_de n\n"
   "where b.id = n.buildid and f0.id = n.file and n.executable_p = 1\n"
   "union all select\n"
-  "b.hex as buildid, n.mtime, 'R' as sourcetype, f0.name as source0, 
n.mtime as mtime, f1.name as source1\n"
+  "b.hex as buildid, 'R

[PATCH v2 1/5] debuginfod: factor out common code for responding from an archive

[Omar Sandoval]

From: Omar Sandoval 

handle_buildid_r_match has two very similar branches where it optionally
extracts a section and then creates a microhttpd response.  In
preparation for adding a third one, factor it out into a function.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx | 213 +-
 1 file changed, 96 insertions(+), 117 deletions(-)

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 305edde8..2d709026 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -1965,6 +1965,81 @@ string canonicalized_archive_entry_pathname(struct 
archive_entry *e)
 }
 
 
+// NB: takes ownership of, and may reassign, fd.
+static struct MHD_Response*
+create_buildid_r_response (int64_t b_mtime0,
+   const string& b_source0,
+   const string& b_source1,
+   const string& section,
+   const string& ima_sig,
+   const char* tmppath,
+   int& fd,
+   off_t size,
+   time_t mtime,
+   const string& metric,
+   const struct timespec& extract_begin)
+{
+  if (tmppath != NULL)
+{
+  struct timespec extract_end;
+  clock_gettime (CLOCK_MONOTONIC, &extract_end);
+  double extract_time = (extract_end.tv_sec - extract_begin.tv_sec)
++ (extract_end.tv_nsec - extract_begin.tv_nsec)/1.e9;
+  fdcache.intern(b_source0, b_source1, tmppath, size, true, extract_time);
+}
+
+  if (!section.empty ())
+{
+  int scn_fd = extract_section (fd, b_mtime0,
+b_source0 + ":" + b_source1,
+section, extract_begin);
+  close (fd);
+  if (scn_fd >= 0)
+fd = scn_fd;
+  else
+{
+  if (verbose)
+obatched (clog) << "cannot find section " << section
+<< " for archive " << b_source0
+<< " file " << b_source1 << endl;
+  return 0;
+}
+
+  struct stat fs;
+  if (fstat (fd, &fs) < 0)
+{
+  close (fd);
+  throw libc_exception (errno,
+string ("fstat ") + b_source0 + string (" ") + section);
+}
+  size = fs.st_size;
+}
+
+  struct MHD_Response* r = MHD_create_response_from_fd (size, fd);
+  if (r == 0)
+{
+  if (verbose)
+obatched(clog) << "cannot create fd-response for " << b_source0 << 
endl;
+  close(fd);
+}
+  else
+{
+  inc_metric ("http_responses_total","result",metric);
+  add_mhd_response_header (r, "Content-Type", "application/octet-stream");
+  add_mhd_response_header (r, "X-DEBUGINFOD-SIZE", 
to_string(size).c_str());
+  add_mhd_response_header (r, "X-DEBUGINFOD-ARCHIVE", b_source0.c_str());
+  add_mhd_response_header (r, "X-DEBUGINFOD-FILE", b_source1.c_str());
+  if(!ima_sig.empty()) add_mhd_response_header(r, 
"X-DEBUGINFOD-IMASIGNATURE", ima_sig.c_str());
+  add_mhd_last_modified (r, mtime);
+  if (verbose > 1)
+obatched(clog) << "serving " << metric << " " << b_source0
+   << " file " << b_source1
+   << " section=" << section
+   << " IMA signature=" << ima_sig << endl;
+  /* libmicrohttpd will close fd. */
+}
+  return r;
+}
 
 static struct MHD_Response*
 handle_buildid_r_match (bool internal_req_p,
@@ -2142,57 +2217,15 @@ handle_buildid_r_match (bool internal_req_p,
   break; // branch out of if "loop", to try new libarchive fetch 
attempt
 }
 
-  if (!section.empty ())
-   {
- int scn_fd = extract_section (fd, fs.st_mtime,
-   b_source0 + ":" + b_source1,
-   section, extract_begin);
- close (fd);
- if (scn_fd >= 0)
-   fd = scn_fd;
- else
-   {
- if (verbose)
-   obatched (clog) << "cannot find section " << section
-   << " for archive " << b_source0
-   << " file " << b_source1 << endl;
- return 0;
-   }
-
- rc = fstat(fd, &fs);
- if (rc < 0)
-   {
- close (fd);
- throw libc_exception (errno,
-  

[PATCH v2 3/5] debuginfod: optimize extraction from seekable xz archives

[Omar Sandoval]

From: Omar Sandoval 

The kernel debuginfo packages on Fedora, Debian, and Ubuntu, and many of
their downstreams, are all compressed with xz in multi-threaded mode,
which allows random access.  We can use this to bypass the full archive
extraction and dramatically speed up kernel debuginfo requests (from ~50
seconds in the worst case to < 0.25 seconds).

This works because multi-threaded xz compression splits up the stream
into many independently compressed blocks.  The stream ends with an
index of blocks.  So, to seek to an offset, we find the block containing
that offset in the index and then decompress and throw away data until
we reach the offset within the block.  We can then decompress the
desired amount of data, possibly from subsequent blocks.  There's no
high-level API in liblzma to do this, but we can do it by stitching
together a few low-level APIs.

We need to pass down the file ids then look up the size, uncompressed
offset, and mtime in the _r_seekable table.  Note that this table is not
yet populated, so this commit has no functional change on its own.

Signed-off-by: Omar Sandoval 
---
 configure.ac  |   5 +
 debuginfod/Makefile.am|   2 +-
 debuginfod/debuginfod.cxx | 452 +-
 3 files changed, 453 insertions(+), 6 deletions(-)

diff --git a/configure.ac b/configure.ac
index 24e68d94..9c5f7e51 100644
--- a/configure.ac
+++ b/configure.ac
@@ -441,8 +441,13 @@ eu_ZIPLIB(bzlib,BZLIB,bz2,BZ2_bzdopen,bzip2)
 # We need this since bzip2 doesn't have a pkgconfig file.
 BZ2_LIB="$LIBS"
 AC_SUBST([BZ2_LIB])
+save_LIBS="$LIBS"
+LIBS=
 eu_ZIPLIB(lzma,LZMA,lzma,lzma_auto_decoder,[LZMA (xz)])
+lzma_LIBS="$LIBS"
+LIBS="$lzma_LIBS $save_LIBS"
 AS_IF([test "x$with_lzma" = xyes], [LIBLZMA="liblzma"], [LIBLZMA=""])
+AC_SUBST([lzma_LIBS])
 AC_SUBST([LIBLZMA])
 eu_ZIPLIB(zstd,ZSTD,zstd,ZSTD_decompress,[ZSTD (zst)])
 AS_IF([test "x$with_zstd" = xyes], [LIBZSTD="libzstd"], [LIBLZSTD=""])
diff --git a/debuginfod/Makefile.am b/debuginfod/Makefile.am
index b74e3673..e199dc0c 100644
--- a/debuginfod/Makefile.am
+++ b/debuginfod/Makefile.am
@@ -70,7 +70,7 @@ bin_PROGRAMS += debuginfod-find
 endif
 
 debuginfod_SOURCES = debuginfod.cxx
-debuginfod_LDADD = $(libdw) $(libelf) $(libeu) $(libdebuginfod) $(argp_LDADD) 
$(fts_LIBS) $(libmicrohttpd_LIBS) $(sqlite3_LIBS) $(libarchive_LIBS) 
$(rpm_LIBS) $(jsonc_LIBS) $(libcurl_LIBS) -lpthread -ldl
+debuginfod_LDADD = $(libdw) $(libelf) $(libeu) $(libdebuginfod) $(argp_LDADD) 
$(fts_LIBS) $(libmicrohttpd_LIBS) $(sqlite3_LIBS) $(libarchive_LIBS) 
$(rpm_LIBS) $(jsonc_LIBS) $(libcurl_LIBS) $(lzma_LIBS) -lpthread -ldl
 
 debuginfod_find_SOURCES = debuginfod-find.c
 debuginfod_find_LDADD = $(libdw) $(libelf) $(libeu) $(libdebuginfod) 
$(argp_LDADD) $(fts_LIBS) $(jsonc_LIBS)
diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 81512fec..a9cbd7cc 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -63,6 +63,10 @@ extern "C" {
 #undef __attribute__ /* glibc bug - rhbz 1763325 */
 #endif
 
+#ifdef USE_LZMA
+#include 
+#endif
+
 #include 
 #include 
 #include 
@@ -1961,6 +1965,382 @@ handle_buildid_f_match (bool internal_req_t,
   return r;
 }
 
+
+#ifdef USE_LZMA
+struct lzma_exception: public reportable_exception
+{
+  lzma_exception(int rc, const string& msg):
+// liblzma doesn't have a lzma_ret -> string conversion function, so just
+// report the value.
+reportable_exception(string ("lzma error: ") + msg + ": error " + 
to_string(rc)) {
+  inc_metric("error_count","lzma",to_string(rc));
+}
+};
+
+// Neither RPM nor deb files support seeking to a specific file in the package.
+// Instead, to extract a specific file, we normally need to read the archive
+// sequentially until we find the file.  This is very slow for files at the end
+// of a large package with lots of files, like kernel debuginfo.
+//
+// However, if the compression format used in the archive supports seeking, we
+// can accelerate this.  As of July 2024, xz is the only widely-used format 
that
+// supports seeking, and usually only in multi-threaded mode.  Luckily, the
+// kernel-debuginfo package in Fedora and its downstreams, and the
+// linux-image-*-dbg package in Debian and its downstreams, all happen to use
+// this.
+//
+// The xz format [1] ends with an index of independently compressed blocks in
+// the stream.  In RPM and deb files, the xz stream is the last thing in the
+// file, so we assume that the xz Stream Footer is at the end of the package
+// file and do everything relative to that.  For each file in the archive, we
+// remember the size and offset of the file data in the uncompressed xz stream,
+// then we use the index to seek to that offset when we need that file.
+//
+// 1: https://xz.tukaani.org/format/xz

[PATCH v2 4/5] debuginfod: populate _r_seekable on scan

[Omar Sandoval]

From: Omar Sandoval 

Whenever a new archive is scanned, check if it is seekable with a little
liblzma magic, and populate _r_seekable if so.  With this, newly scanned
seekable archives will used the optimized extraction path added in the
previous commit.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx | 150 +-
 1 file changed, 147 insertions(+), 3 deletions(-)

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index a9cbd7cc..f120dc90 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -1998,6 +1998,109 @@ struct lzma_exception: public reportable_exception
 //
 // 1: https://xz.tukaani.org/format/xz-file-format.txt
 
+// Return whether an archive supports seeking.
+static bool
+is_seekable_archive (const string& rps, struct archive* a)
+{
+  // Only xz supports seeking.
+  if (archive_filter_code (a, 0) != ARCHIVE_FILTER_XZ)
+return false;
+
+  int fd = open (rps.c_str(), O_RDONLY);
+  if (fd < 0)
+return false;
+  defer_dtor fd_closer (fd, close);
+
+  // Seek to the xz Stream Footer.  We assume that it's the last thing in the
+  // file, which is true for RPM and deb files.
+  off_t footer_pos = -LZMA_STREAM_HEADER_SIZE;
+  if (lseek (fd, footer_pos, SEEK_END) == -1)
+return false;
+
+  // Decode the Stream Footer.
+  uint8_t footer[LZMA_STREAM_HEADER_SIZE];
+  size_t footer_read = 0;
+  while (footer_read < sizeof (footer))
+{
+  ssize_t bytes_read = read (fd, footer + footer_read,
+ sizeof (footer) - footer_read);
+  if (bytes_read < 0)
+{
+  if (errno == EINTR)
+continue;
+  return false;
+}
+  if (bytes_read == 0)
+return false;
+  footer_read += bytes_read;
+}
+
+  lzma_stream_flags stream_flags;
+  lzma_ret ret = lzma_stream_footer_decode (&stream_flags, footer);
+  if (ret != LZMA_OK)
+return false;
+
+  // Seek to the xz Index.
+  if (lseek (fd, footer_pos - stream_flags.backward_size, SEEK_END) == -1)
+return false;
+
+  // Decode the Number of Records in the Index.  liblzma doesn't have an API 
for
+  // this if you don't want to decode the whole Index, so we have to do it
+  // ourselves.
+  //
+  // We need 1 byte for the Index Indicator plus 1-9 bytes for the
+  // variable-length integer Number of Records.
+  uint8_t index[10];
+  size_t index_read = 0;
+  while (index_read == 0) {
+  ssize_t bytes_read = read (fd, index, sizeof (index));
+  if (bytes_read < 0)
+{
+  if (errno == EINTR)
+continue;
+  return false;
+}
+  if (bytes_read == 0)
+return false;
+  index_read += bytes_read;
+  }
+  // The Index Indicator must be 0.
+  if (index[0] != 0)
+return false;
+
+  lzma_vli num_records;
+  size_t pos = 0;
+  size_t in_pos = 1;
+  while (true)
+{
+  if (in_pos >= index_read)
+{
+  ssize_t bytes_read = read (fd, index, sizeof (index));
+  if (bytes_read < 0)
+  {
+if (errno == EINTR)
+  continue;
+return false;
+  }
+  if (bytes_read == 0)
+return false;
+  index_read = bytes_read;
+  in_pos = 0;
+}
+  ret = lzma_vli_decode (&num_records, &pos, index, &in_pos, index_read);
+  if (ret == LZMA_STREAM_END)
+break;
+  else if (ret != LZMA_OK)
+return false;
+}
+
+  if (verbose > 3)
+obatched(clog) << rps << " has " << num_records << " xz Blocks" << endl;
+
+  // The file is only seekable if it has more than one Block.
+  return num_records > 1;
+}
+
 // Read the Index at the end of an xz file.
 static lzma_index*
 read_xz_index (int fd)
@@ -2330,6 +2433,11 @@ extract_from_seekable_archive (const string& srcpath,
 }
 }
 #else
+static bool
+is_seekable_archive (const string& rps, struct archive* a)
+{
+  return false;
+}
 static int
 extract_from_seekable_archive (const string& srcpath,
char* tmppath,
@@ -4277,6 +4385,7 @@ archive_classify (const string& rps, string& 
archive_extension, int64_t archivei
   sqlite_ps& ps_upsert_buildids, sqlite_ps& 
ps_upsert_fileparts, sqlite_ps& ps_upsert_file,
   sqlite_ps& ps_lookup_file,
   sqlite_ps& ps_upsert_de, sqlite_ps& ps_upsert_sref, 
sqlite_ps& ps_upsert_sdef,
+  sqlite_ps& ps_upsert_seekable,
   time_t mtime,
   unsigned& fts_executable, unsigned& fts_debuginfo, unsigned& 
fts_sref, unsigned& fts_sdef,
   bool& fts_sref_complete_p)
@@ -4331,6 +4440,10 @@ archive_classify (const string& rps, string& 
archive_extension, int64_t archivei
   if (verbose > 3)
 obatc

[PATCH v2 5/5] debuginfod: populate _r_seekable on request

[Omar Sandoval]

From: Omar Sandoval 

Since the schema change adding _r_seekable was done in a backward
compatible way, seekable archives that were previously scanned will not
be in _r_seekable.  Whenever an archive is going to be extracted to
satisfy a request, check if it is seekable.  If so, populate _r_seekable
while extracting it so that future requests use the optimized path.

The next time that BUILDIDS is bumped, all archives will be checked at
scan time.  At that point, checking again will be unnecessary and this
commit can be reverted.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx | 76 +++
 1 file changed, 70 insertions(+), 6 deletions(-)

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index f120dc90..6fb4627c 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -2737,6 +2737,7 @@ handle_buildid_r_match (bool internal_req_p,
 }
 
   // no match ... look for a seekable entry
+  bool populate_seekable = true;
   unique_ptr pp (new sqlite_ps (db, "rpm-seekable-query",
"select type, size, offset, mtime 
from " BUILDIDS "_r_seekable "
"where file = ? and content = ?"));
@@ -2745,6 +2746,9 @@ handle_buildid_r_match (bool internal_req_p,
 {
   if (rc != SQLITE_ROW)
 throw sqlite_exception(rc, "step");
+  // if we found a match in _r_seekable but we fail to extract it, don't
+  // bother populating it again
+  populate_seekable = false;
   const char* seekable_type = (const char*) sqlite3_column_text (*pp, 0);
   if (seekable_type != NULL && strcmp (seekable_type, "xz") == 0)
 {
@@ -2836,16 +2840,39 @@ handle_buildid_r_match (bool internal_req_p,
   throw archive_exception(a, "cannot open archive from pipe");
 }
 
-  // archive traversal is in three stages, no, four stages:
-  // 1) skip entries whose names do not match the requested one
-  // 2) extract the matching entry name (set r = result)
-  // 3) extract some number of prefetched entries (just into fdcache)
-  // 4) abort any further processing
+  // If the archive was scanned in a version without _r_seekable, then we may
+  // need to populate _r_seekable now.  This can be removed the next time
+  // BUILDIDS is updated.
+  if (populate_seekable)
+{
+  populate_seekable = is_seekable_archive (b_source0, a);
+  if (populate_seekable)
+{
+  // NB: the names are already interned
+  pp.reset(new sqlite_ps (db, "rpm-seekable-insert2",
+  "insert or ignore into " BUILDIDS 
"_r_seekable (file, content, type, size, offset, mtime) "
+  "values (?, "
+  "(select id from " BUILDIDS "_files "
+  "where dirname = (select id from " BUILDIDS 
"_fileparts where name = ?) "
+  "and basename = (select id from " BUILDIDS 
"_fileparts where name = ?) "
+  "), 'xz', ?, ?, ?)"));
+}
+}
+
+  // archive traversal is in five stages:
+  // 1) before we find a matching entry, insert it into _r_seekable if needed 
or
+  //skip it otherwise
+  // 2) extract the matching entry (set r = result).  Also insert it into
+  //_r_seekable if needed
+  // 3) extract some number of prefetched entries (just into fdcache).  Also
+  //insert them into _r_seekable if needed
+  // 4) if needed, insert all of the remaining entries into _r_seekable
+  // 5) abort any further processing
   struct MHD_Response* r = 0; // will set in stage 2
   unsigned prefetch_count =
 internal_req_p ? 0 : fdcache_prefetch;// will decrement in stage 3
 
-  while(r == 0 || prefetch_count > 0) // stage 1, 2, or 3
+  while(r == 0 || prefetch_count > 0 || populate_seekable) // stage 1-4
 {
   if (interrupted)
 break;
@@ -2859,6 +2886,43 @@ handle_buildid_r_match (bool internal_req_p,
 continue;
 
   string fn = canonicalized_archive_entry_pathname (e);
+
+  if (populate_seekable)
+{
+  string dn, bn;
+  size_t slash = fn.rfind('/');
+  if (slash == std::string::npos) {
+dn = "";
+bn = fn;
+  } else {
+dn = fn.substr(0, slash);
+bn = fn.substr(slash + 1);
+  }
+
+  int64_t seekable_size = archive_entry_size (e);
+  int64_t seekable_offset = archive_filter_bytes (a, 0);
+  time_t seekable_mtime = archive_entry_mtime (e);
+
+  pp->reset();
+  pp->bind(1, b_id0);
+  pp->bind(2, dn);
+  pp->bind(3, bn);
+   

Re: [PATCH v2 0/5] debuginfod: speed up extraction from kernel debuginfo packages by 200x

[Omar Sandoval]

On Tue, Jul 16, 2024 at 04:16:01PM -0400, Frank Ch. Eigler wrote:
> Hi -
> 
> > This is v2 of my patch series optimizing debuginfod for kernel
> > debuginfo.  v1 is here [7].
> 
> This generally looks great to me.  I'll send it through the testsuite
> trybots here.

Great, thank you!

> But there isn't an xz-y test case yet.  Is there a
> smallish seekable-xz rpm file that you have or could make that we
> could jam into the elfutils testsuite?

Good point, I'll add one.

> (A prometheus metric counting
> seeked-xz attempts/successes would be cool to assist testing and to
> monitor it on deployed servers.)

I do have:

  inc_metric("error_count","lzma",to_string(rc));

in lzma_exception and:

  inc_metric ("http_responses_total","result",metric);

in create_buildid_r_response, which gets called with
metric = "seekable archive".  But I'll add another metric for attempts,
and name the existing one "seekable xz archive" to be more explicit.

Thanks,
Omar

Re: [PATCH v2 0/5] debuginfod: speed up extraction from kernel debuginfo packages by 200x

[Omar Sandoval]

On Tue, Jul 16, 2024 at 06:15:16PM -0400, Frank Ch. Eigler wrote:
> Hi -
> 
> > [...] I'll send it through the testsuite
> > trybots here.  [...]
> 
> There was some success and there was some failure. :-)
> 
> all 11 runs:
> 
> https://builder.sourceware.org/testruns/?commitishes=&has_expfile_glob=&has_trsfile_glob=&has_keyvalue_k=testrun.git_describe&has_keyvalue_op=glob&has_keyvalue_v=*&has_keyvalue2_k=source.gitbranch&has_keyvalue2_op=glob&has_keyvalue2_v=*users%2Ffche%2Ftry-xz*&order_by=testrun.authored.time&order=desc
> 
> in a grid view:
> 
> https://builder.sourceware.org/r_grid_testcase/?trid=9b0340db2a771c5b6483132afd75139699c4f8e5&toplevel=True&vertical=source.gitdescribe&v_limit=5&horizontal=uname-m&h_limit=10&opt_keyword_key=source.gitbranch&opt_keyword_value=*fche%2Ftry-xz*
> 
> e.g. failure:
> 
> https://builder.sourceware.org/testrun/2550ae9a06868d7e7f4c62176960c04f4534d8d0?filename=tests%2Frun-debuginfod-extraction-passive.sh.log#line657

Yup, that was a goof, and I don't know why I missed it on my local test
run.  This is the fix:

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 6fb4627c..08114f2e 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -2737,8 +2737,9 @@ handle_buildid_r_match (bool internal_req_p,
 }
 
   // no match ... look for a seekable entry
-  bool populate_seekable = true;
-  unique_ptr pp (new sqlite_ps (db, "rpm-seekable-query",
+  bool populate_seekable = ! passive_p;
+  unique_ptr pp (new sqlite_ps (internal_req_p ? db : dbq,
+   "rpm-seekable-query",
"select type, size, offset, mtime 
from " BUILDIDS "_r_seekable "
"where file = ? and content = ?"));
   rc = pp->reset().bind(1, b_id0).bind(2, b_id1).step();


I'll fold that in and include it in the next version.

[PATCH v3 0/7] debuginfod: speed up extraction from kernel debuginfo packages by 200x

[Omar Sandoval]

From: Omar Sandoval 

This is v3 of my patch series optimizing debuginfod for kernel
debuginfo.  v1 is here [7], v2 is here [8].  This version fixes a couple
of minor bugs and adds test cases.

Changes from v2 to v3:

- Added a test case with seekable rpm and deb files.
- Added a couple of independent fixes uncovered while adding tests.
- Added a few more prometheus metrics.
- Fixed passive mode.

Patches 1 and 2 fix existing bugs that the were uncovered by adding new
test package files.  Patch 3 is a preparatory refactor.  Patch 4 makes
the schema changes. Patch 5 implements the seekable xz extraction.
Patch 6 populates the table of seekable entries at scan time and adds a
test.  Patch 7 does it for pre-existing files at request time.

Here is the background copied and pasted from v1:

drgn [1] currently uses debuginfod with great success for debugging
userspace processes.  However, for debugging the Linux kernel (drgn's
main use case), we have had some performance issues with debuginfod, so
we intentionally avoid using it.  Specifically, it sometimes takes over
a minute for debuginfod to respond to queries for vmlinux and kernel
modules (not including the actual download time).

The reason for the slowness is that Linux kernel debuginfo packages are
very large and contain lots of files.  To respond to a query for a Linux
kernel debuginfo file, debuginfod has to decompress and iterate through
the whole package until it finds that file.  If the file is towards the
end of the package, this can take a very long time.  This was previously
reported for vdso files [2][3], which debuginfod was able to mitigate
with improved caching and prefetching.  However, kernel modules are far
greater in number, vary drastically by hardware and workload, and can be
spread all over the package, so in practice I've still been seeing long
delays.  This was also discussed on the drgn issue tracker [4].

The fundamental limitation is that Linux packages, which are essentially
compressed archives with extra metadata headers, don't support random
access to specific files.  However, the multi-threaded xz compression
format does actually support random access.  And, luckily, the kernel
debuginfo packages on Fedora, Debian, and Ubuntu all happen to use
multi-threaded xz compression!

debuginfod can take advantage of this: when it scans a package, if it is
a seekable xz archive, it can save the uncompressed offset and size of
each file.  Then, when it needs a file, it can seek to that offset and
extract it from there.  This requires some understanding of the xz
format and low-level liblzma code, but the speedup is massive: where the
worst case was previously about 50 seconds just to find a file in a
kernel debuginfo package, with this change the worst case is 0.25
seconds, a ~200x improvement! This works for both .rpm and .deb files.

I tested this by requesting and verifying the digest of every file from
a few kernel debuginfo rpms and debs [5].

P.S. The biggest downside of this change is that it depends on a very
specific compression format that is only used by kernel packages
incidentally.  I think this is something we should formalize with Linux
distributions: large debuginfo packages should use a seekable format.
Currently, xz in multi-threaded mode is the only option, but Zstandard
also has an experimental seekable format that is worth looking into [6].

Thanks,
Omar

1: https://github.com/osandov/drgn
2: https://sourceware.org/bugzilla/show_bug.cgi?id=29478
3: https://bugzilla.redhat.com/show_bug.cgi?id=1970578
4: https://github.com/osandov/drgn/pull/380
5: https://gist.github.com/osandov/89d521fdc6c9a07aa8bb0ebf91974346
6: https://github.com/facebook/zstd/tree/dev/contrib/seekable_format
7: https://sourceware.org/pipermail/elfutils-devel/2024q3/007191.html
8: https://sourceware.org/pipermail/elfutils-devel/2024q3/007208.html

Omar Sandoval (7):
  debuginfod: fix skipping  source file
  tests/run-debuginfod-fd-prefetch-caches.sh: disable fdcache limit
check
  debuginfod: factor out common code for responding from an archive
  debugifod: add new table and views for seekable archives
  debuginfod: optimize extraction from seekable xz archives
  debuginfod: populate _r_seekable on scan
  debuginfod: populate _r_seekable on request

 configure.ac  |   5 +
 debuginfod/Makefile.am|   2 +-
 debuginfod/debuginfod.cxx | 928 +++---
 tests/Makefile.am |   4 +-
 ...pressme-seekable-xz-dbgsym_1.0-1_amd64.deb | Bin 0 -> 6288 bytes
 ...compressme-seekable-xz_1.0-1.debian.tar.xz | Bin 0 -> 1440 bytes
 .../compressme-seekable-xz_1.0-1.dsc  |  19 +
 .../compressme-seekable-xz_1.0-1_amd64.deb| Bin 0 -> 6208 bytes
 .../compressme-seekable-xz_1.0.orig.tar.xz| Bin 0 -> 7160 bytes
 .../compressme-seekable-xz-1.0-1.src.rpm  | Bin 0 -> 15880 bytes
 .../compressme-seekable-xz-1.0-1

[PATCH v3 4/7] debugifod: add new table and views for seekable archives

[Omar Sandoval]

From: Omar Sandoval 

In order to extract a file from a seekable archive, we need to know
where in the uncompressed archive the file data starts and its size.
Additionally, in order to populate the response headers, we need the
file modification time (since we won't be able to get it from the
archive metadata).  Add a new table, _r_seekable, keyed on the archive
file id and entry file id and containing the size, offset, and mtime.
It also contains the compression type just in case new seekable formats
are supported in the future.

In order to search this table when we get a request, we need the file
ids available.  Add the ids to the _query_d and _query_e views, and
rename them to _query_d2 and _query_e2.

This schema change is backward compatible and doesn't require
reindexing.  _query_d2 and _query_e2 can be renamed back the next time
BUILDIDS needs to be bumped.

Before this change, the database for a single kernel debuginfo RPM
(kernel-debuginfo-6.9.6-200.fc40.x86_64.rpm) was about 15MB.  This
change increases that by about 70kB, only a 0.5% increase.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx | 34 --
 1 file changed, 24 insertions(+), 10 deletions(-)

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 24702c23..b3d80090 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -265,25 +265,39 @@ static const char DEBUGINFOD_SQLITE_DDL[] =
   "foreign key (content) references " BUILDIDS "_files(id) on update 
cascade on delete cascade,\n"
   "primary key (content, file, mtime)\n"
   ") " WITHOUT_ROWID ";\n"
+  "create table if not exists " BUILDIDS "_r_seekable (\n" // seekable rpm 
contents
+  "file integer not null,\n"
+  "content integer not null,\n"
+  "type text not null,\n"
+  "size integer not null,\n"
+  "offset integer not null,\n"
+  "mtime integer not null,\n"
+  "foreign key (file) references " BUILDIDS "_files(id) on update 
cascade on delete cascade,\n"
+  "foreign key (content) references " BUILDIDS "_files(id) on update 
cascade on delete cascade,\n"
+  "primary key (file, content)\n"
+  ") " WITHOUT_ROWID ";\n"
   // create views to glue together some of the above tables, for webapi D 
queries
-  "create view if not exists " BUILDIDS "_query_d as \n"
+  // NB: _query_d2 and _query_e2 were added to replace _query_d and _query_e
+  // without updating BUILDIDS.  They can be renamed back the next time 
BUILDIDS
+  // is updated.
+  "create view if not exists " BUILDIDS "_query_d2 as \n"
   "select\n"
-  "b.hex as buildid, n.mtime, 'F' as sourcetype, f0.name as source0, 
n.mtime as mtime, null as source1\n"
+  "b.hex as buildid, 'F' as sourcetype, n.file as id0, f0.name as 
source0, n.mtime as mtime, null as id1, null as source1\n"
   "from " BUILDIDS "_buildids b, " BUILDIDS "_files_v f0, " BUILDIDS 
"_f_de n\n"
   "where b.id = n.buildid and f0.id = n.file and n.debuginfo_p = 1\n"
   "union all select\n"
-  "b.hex as buildid, n.mtime, 'R' as sourcetype, f0.name as source0, 
n.mtime as mtime, f1.name as source1\n"
+  "b.hex as buildid, 'R' as sourcetype, n.file as id0, f0.name as 
source0, n.mtime as mtime, n.content as id1, f1.name as source1\n"
   "from " BUILDIDS "_buildids b, " BUILDIDS "_files_v f0, " BUILDIDS 
"_files_v f1, " BUILDIDS "_r_de n\n"
   "where b.id = n.buildid and f0.id = n.file and f1.id = n.content and 
n.debuginfo_p = 1\n"
   ";"
   // ... and for E queries
-  "create view if not exists " BUILDIDS "_query_e as \n"
+  "create view if not exists " BUILDIDS "_query_e2 as \n"
   "select\n"
-  "b.hex as buildid, n.mtime, 'F' as sourcetype, f0.name as source0, 
n.mtime as mtime, null as source1\n"
+  "b.hex as buildid, 'F' as sourcetype, n.file as id0, f0.name as 
source0, n.mtime as mtime, null as id1, null as source1\n"
   "from " BUILDIDS "_buildids b, " BUILDIDS "_files_v f0, " BUILDIDS 
"_f_de n\n"
   "where b.id = n.buildid and f0.id = n.file and n.executable_p = 1\n"
   "union all select\n"
-  "b.hex as buildid, n.mtime, 'R' as sourcetype, f0.name as source0, 
n.mtime as mtime, f1.name as source1\n"
+  "b.hex as buildid, 'R

[PATCH v3 5/7] debuginfod: optimize extraction from seekable xz archives

[Omar Sandoval]

From: Omar Sandoval 

The kernel debuginfo packages on Fedora, Debian, and Ubuntu, and many of
their downstreams, are all compressed with xz in multi-threaded mode,
which allows random access.  We can use this to bypass the full archive
extraction and dramatically speed up kernel debuginfo requests (from ~50
seconds in the worst case to < 0.25 seconds).

This works because multi-threaded xz compression splits up the stream
into many independently compressed blocks.  The stream ends with an
index of blocks.  So, to seek to an offset, we find the block containing
that offset in the index and then decompress and throw away data until
we reach the offset within the block.  We can then decompress the
desired amount of data, possibly from subsequent blocks.  There's no
high-level API in liblzma to do this, but we can do it by stitching
together a few low-level APIs.

We need to pass down the file ids then look up the size, uncompressed
offset, and mtime in the _r_seekable table.  Note that this table is not
yet populated, so this commit has no functional change on its own.

Signed-off-by: Omar Sandoval 
---
 configure.ac  |   5 +
 debuginfod/Makefile.am|   2 +-
 debuginfod/debuginfod.cxx | 456 +-
 3 files changed, 457 insertions(+), 6 deletions(-)

diff --git a/configure.ac b/configure.ac
index 24e68d94..9c5f7e51 100644
--- a/configure.ac
+++ b/configure.ac
@@ -441,8 +441,13 @@ eu_ZIPLIB(bzlib,BZLIB,bz2,BZ2_bzdopen,bzip2)
 # We need this since bzip2 doesn't have a pkgconfig file.
 BZ2_LIB="$LIBS"
 AC_SUBST([BZ2_LIB])
+save_LIBS="$LIBS"
+LIBS=
 eu_ZIPLIB(lzma,LZMA,lzma,lzma_auto_decoder,[LZMA (xz)])
+lzma_LIBS="$LIBS"
+LIBS="$lzma_LIBS $save_LIBS"
 AS_IF([test "x$with_lzma" = xyes], [LIBLZMA="liblzma"], [LIBLZMA=""])
+AC_SUBST([lzma_LIBS])
 AC_SUBST([LIBLZMA])
 eu_ZIPLIB(zstd,ZSTD,zstd,ZSTD_decompress,[ZSTD (zst)])
 AS_IF([test "x$with_zstd" = xyes], [LIBZSTD="libzstd"], [LIBLZSTD=""])
diff --git a/debuginfod/Makefile.am b/debuginfod/Makefile.am
index b74e3673..e199dc0c 100644
--- a/debuginfod/Makefile.am
+++ b/debuginfod/Makefile.am
@@ -70,7 +70,7 @@ bin_PROGRAMS += debuginfod-find
 endif
 
 debuginfod_SOURCES = debuginfod.cxx
-debuginfod_LDADD = $(libdw) $(libelf) $(libeu) $(libdebuginfod) $(argp_LDADD) 
$(fts_LIBS) $(libmicrohttpd_LIBS) $(sqlite3_LIBS) $(libarchive_LIBS) 
$(rpm_LIBS) $(jsonc_LIBS) $(libcurl_LIBS) -lpthread -ldl
+debuginfod_LDADD = $(libdw) $(libelf) $(libeu) $(libdebuginfod) $(argp_LDADD) 
$(fts_LIBS) $(libmicrohttpd_LIBS) $(sqlite3_LIBS) $(libarchive_LIBS) 
$(rpm_LIBS) $(jsonc_LIBS) $(libcurl_LIBS) $(lzma_LIBS) -lpthread -ldl
 
 debuginfod_find_SOURCES = debuginfod-find.c
 debuginfod_find_LDADD = $(libdw) $(libelf) $(libeu) $(libdebuginfod) 
$(argp_LDADD) $(fts_LIBS) $(jsonc_LIBS)
diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index b3d80090..cf7f48ab 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -63,6 +63,10 @@ extern "C" {
 #undef __attribute__ /* glibc bug - rhbz 1763325 */
 #endif
 
+#ifdef USE_LZMA
+#include 
+#endif
+
 #include 
 #include 
 #include 
@@ -1961,6 +1965,385 @@ handle_buildid_f_match (bool internal_req_t,
   return r;
 }
 
+
+#ifdef USE_LZMA
+struct lzma_exception: public reportable_exception
+{
+  lzma_exception(int rc, const string& msg):
+// liblzma doesn't have a lzma_ret -> string conversion function, so just
+// report the value.
+reportable_exception(string ("lzma error: ") + msg + ": error " + 
to_string(rc)) {
+  inc_metric("error_count","lzma",to_string(rc));
+}
+};
+
+// Neither RPM nor deb files support seeking to a specific file in the package.
+// Instead, to extract a specific file, we normally need to read the archive
+// sequentially until we find the file.  This is very slow for files at the end
+// of a large package with lots of files, like kernel debuginfo.
+//
+// However, if the compression format used in the archive supports seeking, we
+// can accelerate this.  As of July 2024, xz is the only widely-used format 
that
+// supports seeking, and usually only in multi-threaded mode.  Luckily, the
+// kernel-debuginfo package in Fedora and its downstreams, and the
+// linux-image-*-dbg package in Debian and its downstreams, all happen to use
+// this.
+//
+// The xz format [1] ends with an index of independently compressed blocks in
+// the stream.  In RPM and deb files, the xz stream is the last thing in the
+// file, so we assume that the xz Stream Footer is at the end of the package
+// file and do everything relative to that.  For each file in the archive, we
+// remember the size and offset of the file data in the uncompressed xz stream,
+// then we use the index to seek to that offset when we need that file.
+//
+// 1: https://xz.tukaani.org/format/xz

[PATCH v3 7/7] debuginfod: populate _r_seekable on request

[Omar Sandoval]

From: Omar Sandoval 

Since the schema change adding _r_seekable was done in a backward
compatible way, seekable archives that were previously scanned will not
be in _r_seekable.  Whenever an archive is going to be extracted to
satisfy a request, check if it is seekable.  If so, populate _r_seekable
while extracting it so that future requests use the optimized path.

The next time that BUILDIDS is bumped, all archives will be checked at
scan time.  At that point, checking again will be unnecessary and this
commit (including the test case modification) can be reverted.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx| 76 +---
 tests/run-debuginfod-seekable.sh | 45 +++
 2 files changed, 115 insertions(+), 6 deletions(-)

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 677eca30..d8a02fb5 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -2740,6 +2740,7 @@ handle_buildid_r_match (bool internal_req_p,
 }
 
   // no match ... look for a seekable entry
+  bool populate_seekable = ! passive_p;
   unique_ptr pp (new sqlite_ps (internal_req_p ? db : dbq,
"rpm-seekable-query",
"select type, size, offset, mtime 
from " BUILDIDS "_r_seekable "
@@ -2749,6 +2750,9 @@ handle_buildid_r_match (bool internal_req_p,
 {
   if (rc != SQLITE_ROW)
 throw sqlite_exception(rc, "step");
+  // if we found a match in _r_seekable but we fail to extract it, don't
+  // bother populating it again
+  populate_seekable = false;
   const char* seekable_type = (const char*) sqlite3_column_text (*pp, 0);
   if (seekable_type != NULL && strcmp (seekable_type, "xz") == 0)
 {
@@ -2840,16 +2844,39 @@ handle_buildid_r_match (bool internal_req_p,
   throw archive_exception(a, "cannot open archive from pipe");
 }
 
-  // archive traversal is in three stages, no, four stages:
-  // 1) skip entries whose names do not match the requested one
-  // 2) extract the matching entry name (set r = result)
-  // 3) extract some number of prefetched entries (just into fdcache)
-  // 4) abort any further processing
+  // If the archive was scanned in a version without _r_seekable, then we may
+  // need to populate _r_seekable now.  This can be removed the next time
+  // BUILDIDS is updated.
+  if (populate_seekable)
+{
+  populate_seekable = is_seekable_archive (b_source0, a);
+  if (populate_seekable)
+{
+  // NB: the names are already interned
+  pp.reset(new sqlite_ps (db, "rpm-seekable-insert2",
+  "insert or ignore into " BUILDIDS 
"_r_seekable (file, content, type, size, offset, mtime) "
+  "values (?, "
+  "(select id from " BUILDIDS "_files "
+  "where dirname = (select id from " BUILDIDS 
"_fileparts where name = ?) "
+  "and basename = (select id from " BUILDIDS 
"_fileparts where name = ?) "
+  "), 'xz', ?, ?, ?)"));
+}
+}
+
+  // archive traversal is in five stages:
+  // 1) before we find a matching entry, insert it into _r_seekable if needed 
or
+  //skip it otherwise
+  // 2) extract the matching entry (set r = result).  Also insert it into
+  //_r_seekable if needed
+  // 3) extract some number of prefetched entries (just into fdcache).  Also
+  //insert them into _r_seekable if needed
+  // 4) if needed, insert all of the remaining entries into _r_seekable
+  // 5) abort any further processing
   struct MHD_Response* r = 0; // will set in stage 2
   unsigned prefetch_count =
 internal_req_p ? 0 : fdcache_prefetch;// will decrement in stage 3
 
-  while(r == 0 || prefetch_count > 0) // stage 1, 2, or 3
+  while(r == 0 || prefetch_count > 0 || populate_seekable) // stage 1-4
 {
   if (interrupted)
 break;
@@ -2863,6 +2890,43 @@ handle_buildid_r_match (bool internal_req_p,
 continue;
 
   string fn = canonicalized_archive_entry_pathname (e);
+
+  if (populate_seekable)
+{
+  string dn, bn;
+  size_t slash = fn.rfind('/');
+  if (slash == std::string::npos) {
+dn = "";
+bn = fn;
+  } else {
+dn = fn.substr(0, slash);
+bn = fn.substr(slash + 1);
+  }
+
+  int64_t seekable_size = archive_entry_size (e);
+  int64_t seekable_offset = archive_filter_bytes (a, 0);
+  time_t seekable_mtime = archive_entry_mtime (e);
+
+  pp->reset();
+

[PATCH v3 6/7] debuginfod: populate _r_seekable on scan

[Omar Sandoval]

From: Omar Sandoval 

Whenever a new archive is scanned, check if it is seekable with a little
liblzma magic, and populate _r_seekable if so.  With this, newly scanned
seekable archives will used the optimized extraction path added in the
previous commit.  Also add a test case using some artificial packages.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx | 150 +-
 tests/Makefile.am |   4 +-
 ...pressme-seekable-xz-dbgsym_1.0-1_amd64.deb | Bin 0 -> 6288 bytes
 ...compressme-seekable-xz_1.0-1.debian.tar.xz | Bin 0 -> 1440 bytes
 .../compressme-seekable-xz_1.0-1.dsc  |  19 +++
 .../compressme-seekable-xz_1.0-1_amd64.deb| Bin 0 -> 6208 bytes
 .../compressme-seekable-xz_1.0.orig.tar.xz| Bin 0 -> 7160 bytes
 .../compressme-seekable-xz-1.0-1.src.rpm  | Bin 0 -> 15880 bytes
 .../compressme-seekable-xz-1.0-1.x86_64.rpm   | Bin 0 -> 31873 bytes
 ...sme-seekable-xz-debuginfo-1.0-1.x86_64.rpm | Bin 0 -> 21917 bytes
 ...e-seekable-xz-debugsource-1.0-1.x86_64.rpm | Bin 0 -> 7961 bytes
 tests/run-debuginfod-seekable.sh  | 141 
 12 files changed, 309 insertions(+), 5 deletions(-)
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz-dbgsym_1.0-1_amd64.deb
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0-1.debian.tar.xz
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0-1.dsc
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0-1_amd64.deb
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0.orig.tar.xz
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-1.0-1.src.rpm
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-1.0-1.x86_64.rpm
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-debuginfo-1.0-1.x86_64.rpm
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-debugsource-1.0-1.x86_64.rpm
 create mode 100755 tests/run-debuginfod-seekable.sh

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index cf7f48ab..677eca30 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -1998,6 +1998,109 @@ struct lzma_exception: public reportable_exception
 //
 // 1: https://xz.tukaani.org/format/xz-file-format.txt
 
+// Return whether an archive supports seeking.
+static bool
+is_seekable_archive (const string& rps, struct archive* a)
+{
+  // Only xz supports seeking.
+  if (archive_filter_code (a, 0) != ARCHIVE_FILTER_XZ)
+return false;
+
+  int fd = open (rps.c_str(), O_RDONLY);
+  if (fd < 0)
+return false;
+  defer_dtor fd_closer (fd, close);
+
+  // Seek to the xz Stream Footer.  We assume that it's the last thing in the
+  // file, which is true for RPM and deb files.
+  off_t footer_pos = -LZMA_STREAM_HEADER_SIZE;
+  if (lseek (fd, footer_pos, SEEK_END) == -1)
+return false;
+
+  // Decode the Stream Footer.
+  uint8_t footer[LZMA_STREAM_HEADER_SIZE];
+  size_t footer_read = 0;
+  while (footer_read < sizeof (footer))
+{
+  ssize_t bytes_read = read (fd, footer + footer_read,
+ sizeof (footer) - footer_read);
+  if (bytes_read < 0)
+{
+  if (errno == EINTR)
+continue;
+  return false;
+}
+  if (bytes_read == 0)
+return false;
+  footer_read += bytes_read;
+}
+
+  lzma_stream_flags stream_flags;
+  lzma_ret ret = lzma_stream_footer_decode (&stream_flags, footer);
+  if (ret != LZMA_OK)
+return false;
+
+  // Seek to the xz Index.
+  if (lseek (fd, footer_pos - stream_flags.backward_size, SEEK_END) == -1)
+return false;
+
+  // Decode the Number of Records in the Index.  liblzma doesn't have an API 
for
+  // this if you don't want to decode the whole Index, so we have to do it
+  // ourselves.
+  //
+  // We need 1 byte for the Index Indicator plus 1-9 bytes for the
+  // variable-length integer Number of Records.
+  uint8_t index[10];
+  size_t index_read = 0;
+  while (index_read == 0) {
+  ssize_t bytes_read = read (fd, index, sizeof (index));
+  if (bytes_read < 0)
+{
+  if (errno == EINTR)
+continue;
+  return false;
+}
+  if (bytes_read == 0)
+return false;
+  index_read += bytes_read;
+  }
+  // The Index Indicator must be 0.
+  if (index[0] != 0)
+return false;
+
+  lzma_vli num_records;
+  size_t pos = 0;
+  size_t in_pos = 1;
+  while (true)
+{
+  if (in_pos >= index_read)
+{
+  ssize_t bytes_read = read (fd, index, sizeof (index));
+  if (bytes_read < 0)
+  {
+if (errno == EINTR)
+  continue;
+return false;
+  }
+  if (bytes_read == 0)
+return

[PATCH v3 1/7] debuginfod: fix skipping source file

[Omar Sandoval]

From: Omar Sandoval 

dwarf_extract_source_paths explicitly skips source files that equal
"", but dwarf_filesrc may return a path like "dir/".
Check for and skip that case, too.

In particular, the test debuginfod RPMs have paths like this.  However,
the test cases didn't catch this because they have a bug, too: they
follow symlinks, which results in double-counting every file.  Fix that,
too.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx | 3 ++-
 tests/run-debuginfod-archive-groom.sh | 2 +-
 tests/run-debuginfod-extraction.sh| 2 +-
 3 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 305edde8..92022f3d 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -3446,7 +3446,8 @@ dwarf_extract_source_paths (Elf *elf, set& 
debug_sourcefiles)
   if (hat == NULL)
 continue;
 
-  if (string(hat) == "") // gcc intrinsics, don't bother 
record
+  if (string(hat) == ""
+  || string_endswith(hat, "")) // gcc intrinsics, don't 
bother record
 continue;
 
   string waldo;
diff --git a/tests/run-debuginfod-archive-groom.sh 
b/tests/run-debuginfod-archive-groom.sh
index e2c394ef..0131158f 100755
--- a/tests/run-debuginfod-archive-groom.sh
+++ b/tests/run-debuginfod-archive-groom.sh
@@ -109,7 +109,7 @@ for i in $newrpms; do
 rpm2cpio ../$i | cpio -ivd;
 cd ..;
 done
-sourcefiles=$(find -name \*\\.debug \
+sourcefiles=$(find -name \*\\.debug -type f \
   | env LD_LIBRARY_PATH=$ldpath xargs \
 ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
   | grep mtime: | wc --lines)
diff --git a/tests/run-debuginfod-extraction.sh 
b/tests/run-debuginfod-extraction.sh
index da6b25cf..f49dc6f6 100755
--- a/tests/run-debuginfod-extraction.sh
+++ b/tests/run-debuginfod-extraction.sh
@@ -94,7 +94,7 @@ for i in $newrpms; do
 rpm2cpio ../$i | cpio -ivd;
 cd ..;
 done
-sourcefiles=$(find -name \*\\.debug \
+sourcefiles=$(find -name \*\\.debug -type f \
   | env LD_LIBRARY_PATH=$ldpath xargs \
 ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
   | grep mtime: | wc --lines)
-- 
2.45.2

[PATCH v3 3/7] debuginfod: factor out common code for responding from an archive

[Omar Sandoval]

From: Omar Sandoval 

handle_buildid_r_match has two very similar branches where it optionally
extracts a section and then creates a microhttpd response.  In
preparation for adding a third one, factor it out into a function.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx | 213 +-
 1 file changed, 96 insertions(+), 117 deletions(-)

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 92022f3d..24702c23 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -1965,6 +1965,81 @@ string canonicalized_archive_entry_pathname(struct 
archive_entry *e)
 }
 
 
+// NB: takes ownership of, and may reassign, fd.
+static struct MHD_Response*
+create_buildid_r_response (int64_t b_mtime0,
+   const string& b_source0,
+   const string& b_source1,
+   const string& section,
+   const string& ima_sig,
+   const char* tmppath,
+   int& fd,
+   off_t size,
+   time_t mtime,
+   const string& metric,
+   const struct timespec& extract_begin)
+{
+  if (tmppath != NULL)
+{
+  struct timespec extract_end;
+  clock_gettime (CLOCK_MONOTONIC, &extract_end);
+  double extract_time = (extract_end.tv_sec - extract_begin.tv_sec)
++ (extract_end.tv_nsec - extract_begin.tv_nsec)/1.e9;
+  fdcache.intern(b_source0, b_source1, tmppath, size, true, extract_time);
+}
+
+  if (!section.empty ())
+{
+  int scn_fd = extract_section (fd, b_mtime0,
+b_source0 + ":" + b_source1,
+section, extract_begin);
+  close (fd);
+  if (scn_fd >= 0)
+fd = scn_fd;
+  else
+{
+  if (verbose)
+obatched (clog) << "cannot find section " << section
+<< " for archive " << b_source0
+<< " file " << b_source1 << endl;
+  return 0;
+}
+
+  struct stat fs;
+  if (fstat (fd, &fs) < 0)
+{
+  close (fd);
+  throw libc_exception (errno,
+string ("fstat ") + b_source0 + string (" ") + section);
+}
+  size = fs.st_size;
+}
+
+  struct MHD_Response* r = MHD_create_response_from_fd (size, fd);
+  if (r == 0)
+{
+  if (verbose)
+obatched(clog) << "cannot create fd-response for " << b_source0 << 
endl;
+  close(fd);
+}
+  else
+{
+  inc_metric ("http_responses_total","result",metric);
+  add_mhd_response_header (r, "Content-Type", "application/octet-stream");
+  add_mhd_response_header (r, "X-DEBUGINFOD-SIZE", 
to_string(size).c_str());
+  add_mhd_response_header (r, "X-DEBUGINFOD-ARCHIVE", b_source0.c_str());
+  add_mhd_response_header (r, "X-DEBUGINFOD-FILE", b_source1.c_str());
+  if(!ima_sig.empty()) add_mhd_response_header(r, 
"X-DEBUGINFOD-IMASIGNATURE", ima_sig.c_str());
+  add_mhd_last_modified (r, mtime);
+  if (verbose > 1)
+obatched(clog) << "serving " << metric << " " << b_source0
+   << " file " << b_source1
+   << " section=" << section
+   << " IMA signature=" << ima_sig << endl;
+  /* libmicrohttpd will close fd. */
+}
+  return r;
+}
 
 static struct MHD_Response*
 handle_buildid_r_match (bool internal_req_p,
@@ -2142,57 +2217,15 @@ handle_buildid_r_match (bool internal_req_p,
   break; // branch out of if "loop", to try new libarchive fetch 
attempt
 }
 
-  if (!section.empty ())
-   {
- int scn_fd = extract_section (fd, fs.st_mtime,
-   b_source0 + ":" + b_source1,
-   section, extract_begin);
- close (fd);
- if (scn_fd >= 0)
-   fd = scn_fd;
- else
-   {
- if (verbose)
-   obatched (clog) << "cannot find section " << section
-   << " for archive " << b_source0
-   << " file " << b_source1 << endl;
- return 0;
-   }
-
- rc = fstat(fd, &fs);
- if (rc < 0)
-   {
- close (fd);
- throw libc_exception (errno,
-  

[PATCH v3 2/7] tests/run-debuginfod-fd-prefetch-caches.sh: disable fdcache limit check

[Omar Sandoval]

From: Omar Sandoval 

Since commit acd9525e93d7 ("PR31265 - rework debuginfod archive-extract
fdcache"), the fdcache limit is only applied when a new file is interned
and it has been at least 10 seconds since the limit was last applied.
This means that the fdcache can go over the limit temporarily.

run-debuginfod-fd-prefetch-caches.sh happens to avoid tripping over this
because of lucky sizes of the files used in the test.  However, adding
new files for an upcoming test exposed this failure.

Disable this part of the test for now.

Signed-off-by: Omar Sandoval 
---
 tests/run-debuginfod-fd-prefetch-caches.sh | 4 
 1 file changed, 4 insertions(+)

diff --git a/tests/run-debuginfod-fd-prefetch-caches.sh 
b/tests/run-debuginfod-fd-prefetch-caches.sh
index 3db78ade..90730555 100755
--- a/tests/run-debuginfod-fd-prefetch-caches.sh
+++ b/tests/run-debuginfod-fd-prefetch-caches.sh
@@ -99,6 +99,9 @@ kill $PID1
 wait $PID1
 PID1=0
 
+# Since we now only limit the fd cache every 10 seconds, it can temporarily go
+# over the limit.  That makes this part of the test unreliable.
+if false; then
 #
 # Test mb limit on fd cache
 #
@@ -148,3 +151,4 @@ kill $PID1
 wait $PID1
 PID1=0
 exit 0
+fi
-- 
2.45.2

Re: [PATCH v3 0/7] debuginfod: speed up extraction from kernel debuginfo packages by 200x

[Omar Sandoval]

On Fri, Jul 19, 2024 at 01:34:48PM -0400, Frank Ch. Eigler wrote:
> Hi -
> 
> > This is v3 of my patch series optimizing debuginfod for kernel
> > debuginfo.  v1 is here [7], v2 is here [8].  This version fixes a couple
> > of minor bugs and adds test cases. [...]
> 
> Thanks, LGTM, running through try-buildbots to make sure.

Sorry about the distcheck failures, looks like I forgot to add the new
test files to EXTRA_DIST.  I'll be sure to run distcheck next time.

I'll send v4 shortly.

Thanks,
Omar

[PATCH v4 0/7] debuginfod: speed up extraction from kernel debuginfo packages by 200x

[Omar Sandoval]

From: Omar Sandoval 

This is v4 of my patch series optimizing debuginfod for kernel
debuginfo.  v1 is here [1], v2 is here [2], v3 is here [3].  The only
changes from v3 in this version are fixing a bogus maybe-uninitialized
error on the Debian build and adding the new test files to EXTRA_DIST so
that make distcheck passes.

Thanks,
Omar

1: https://sourceware.org/pipermail/elfutils-devel/2024q3/007191.html
2: https://sourceware.org/pipermail/elfutils-devel/2024q3/007208.html
3: https://sourceware.org/pipermail/elfutils-devel/2024q3/007243.html

Omar Sandoval (7):
  debuginfod: fix skipping  source file
  tests/run-debuginfod-fd-prefetch-caches.sh: disable fdcache limit
check
  debuginfod: factor out common code for responding from an archive
  debugifod: add new table and views for seekable archives
  debuginfod: optimize extraction from seekable xz archives
  debuginfod: populate _r_seekable on scan
  debuginfod: populate _r_seekable on request

 configure.ac  |   5 +
 debuginfod/Makefile.am|   2 +-
 debuginfod/debuginfod.cxx | 923 +++---
 tests/Makefile.am |  13 +-
 ...pressme-seekable-xz-dbgsym_1.0-1_amd64.deb | Bin 0 -> 6288 bytes
 ...compressme-seekable-xz_1.0-1.debian.tar.xz | Bin 0 -> 1440 bytes
 .../compressme-seekable-xz_1.0-1.dsc  |  19 +
 .../compressme-seekable-xz_1.0-1_amd64.deb| Bin 0 -> 6208 bytes
 .../compressme-seekable-xz_1.0.orig.tar.xz| Bin 0 -> 7160 bytes
 .../compressme-seekable-xz-1.0-1.src.rpm  | Bin 0 -> 15880 bytes
 .../compressme-seekable-xz-1.0-1.x86_64.rpm   | Bin 0 -> 31873 bytes
 ...sme-seekable-xz-debuginfo-1.0-1.x86_64.rpm | Bin 0 -> 21917 bytes
 ...e-seekable-xz-debugsource-1.0-1.x86_64.rpm | Bin 0 -> 7961 bytes
 tests/run-debuginfod-archive-groom.sh |   2 +-
 tests/run-debuginfod-extraction.sh|   2 +-
 tests/run-debuginfod-fd-prefetch-caches.sh|   4 +
 tests/run-debuginfod-seekable.sh  | 186 
 17 files changed, 1011 insertions(+), 145 deletions(-)
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz-dbgsym_1.0-1_amd64.deb
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0-1.debian.tar.xz
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0-1.dsc
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0-1_amd64.deb
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0.orig.tar.xz
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-1.0-1.src.rpm
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-1.0-1.x86_64.rpm
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-debuginfo-1.0-1.x86_64.rpm
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-debugsource-1.0-1.x86_64.rpm
 create mode 100755 tests/run-debuginfod-seekable.sh

-- 
2.45.2

[PATCH v4 1/7] debuginfod: fix skipping source file

[Omar Sandoval]

From: Omar Sandoval 

dwarf_extract_source_paths explicitly skips source files that equal
"", but dwarf_filesrc may return a path like "dir/".
Check for and skip that case, too.

In particular, the test debuginfod RPMs have paths like this.  However,
the test cases didn't catch this because they have a bug, too: they
follow symlinks, which results in double-counting every file.  Fix that,
too.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx | 3 ++-
 tests/run-debuginfod-archive-groom.sh | 2 +-
 tests/run-debuginfod-extraction.sh| 2 +-
 3 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 305edde8..92022f3d 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -3446,7 +3446,8 @@ dwarf_extract_source_paths (Elf *elf, set& 
debug_sourcefiles)
   if (hat == NULL)
 continue;
 
-  if (string(hat) == "") // gcc intrinsics, don't bother 
record
+  if (string(hat) == ""
+  || string_endswith(hat, "")) // gcc intrinsics, don't 
bother record
 continue;
 
   string waldo;
diff --git a/tests/run-debuginfod-archive-groom.sh 
b/tests/run-debuginfod-archive-groom.sh
index e2c394ef..0131158f 100755
--- a/tests/run-debuginfod-archive-groom.sh
+++ b/tests/run-debuginfod-archive-groom.sh
@@ -109,7 +109,7 @@ for i in $newrpms; do
 rpm2cpio ../$i | cpio -ivd;
 cd ..;
 done
-sourcefiles=$(find -name \*\\.debug \
+sourcefiles=$(find -name \*\\.debug -type f \
   | env LD_LIBRARY_PATH=$ldpath xargs \
 ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
   | grep mtime: | wc --lines)
diff --git a/tests/run-debuginfod-extraction.sh 
b/tests/run-debuginfod-extraction.sh
index da6b25cf..f49dc6f6 100755
--- a/tests/run-debuginfod-extraction.sh
+++ b/tests/run-debuginfod-extraction.sh
@@ -94,7 +94,7 @@ for i in $newrpms; do
 rpm2cpio ../$i | cpio -ivd;
 cd ..;
 done
-sourcefiles=$(find -name \*\\.debug \
+sourcefiles=$(find -name \*\\.debug -type f \
   | env LD_LIBRARY_PATH=$ldpath xargs \
 ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
   | grep mtime: | wc --lines)
-- 
2.45.2

[PATCH v4 2/7] tests/run-debuginfod-fd-prefetch-caches.sh: disable fdcache limit check

[Omar Sandoval]

From: Omar Sandoval 

Since commit acd9525e93d7 ("PR31265 - rework debuginfod archive-extract
fdcache"), the fdcache limit is only applied when a new file is interned
and it has been at least 10 seconds since the limit was last applied.
This means that the fdcache can go over the limit temporarily.

run-debuginfod-fd-prefetch-caches.sh happens to avoid tripping over this
because of lucky sizes of the files used in the test.  However, adding
new files for an upcoming test exposed this failure.

Disable this part of the test for now.

Signed-off-by: Omar Sandoval 
---
 tests/run-debuginfod-fd-prefetch-caches.sh | 4 
 1 file changed, 4 insertions(+)

diff --git a/tests/run-debuginfod-fd-prefetch-caches.sh 
b/tests/run-debuginfod-fd-prefetch-caches.sh
index 3db78ade..90730555 100755
--- a/tests/run-debuginfod-fd-prefetch-caches.sh
+++ b/tests/run-debuginfod-fd-prefetch-caches.sh
@@ -99,6 +99,9 @@ kill $PID1
 wait $PID1
 PID1=0
 
+# Since we now only limit the fd cache every 10 seconds, it can temporarily go
+# over the limit.  That makes this part of the test unreliable.
+if false; then
 #
 # Test mb limit on fd cache
 #
@@ -148,3 +151,4 @@ kill $PID1
 wait $PID1
 PID1=0
 exit 0
+fi
-- 
2.45.2

[PATCH v4 4/7] debugifod: add new table and views for seekable archives

[Omar Sandoval]

From: Omar Sandoval 

In order to extract a file from a seekable archive, we need to know
where in the uncompressed archive the file data starts and its size.
Additionally, in order to populate the response headers, we need the
file modification time (since we won't be able to get it from the
archive metadata).  Add a new table, _r_seekable, keyed on the archive
file id and entry file id and containing the size, offset, and mtime.
It also contains the compression type just in case new seekable formats
are supported in the future.

In order to search this table when we get a request, we need the file
ids available.  Add the ids to the _query_d and _query_e views, and
rename them to _query_d2 and _query_e2.

This schema change is backward compatible and doesn't require
reindexing.  _query_d2 and _query_e2 can be renamed back the next time
BUILDIDS needs to be bumped.

Before this change, the database for a single kernel debuginfo RPM
(kernel-debuginfo-6.9.6-200.fc40.x86_64.rpm) was about 15MB.  This
change increases that by about 70kB, only a 0.5% increase.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx | 34 --
 1 file changed, 24 insertions(+), 10 deletions(-)

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 24702c23..b3d80090 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -265,25 +265,39 @@ static const char DEBUGINFOD_SQLITE_DDL[] =
   "foreign key (content) references " BUILDIDS "_files(id) on update 
cascade on delete cascade,\n"
   "primary key (content, file, mtime)\n"
   ") " WITHOUT_ROWID ";\n"
+  "create table if not exists " BUILDIDS "_r_seekable (\n" // seekable rpm 
contents
+  "file integer not null,\n"
+  "content integer not null,\n"
+  "type text not null,\n"
+  "size integer not null,\n"
+  "offset integer not null,\n"
+  "mtime integer not null,\n"
+  "foreign key (file) references " BUILDIDS "_files(id) on update 
cascade on delete cascade,\n"
+  "foreign key (content) references " BUILDIDS "_files(id) on update 
cascade on delete cascade,\n"
+  "primary key (file, content)\n"
+  ") " WITHOUT_ROWID ";\n"
   // create views to glue together some of the above tables, for webapi D 
queries
-  "create view if not exists " BUILDIDS "_query_d as \n"
+  // NB: _query_d2 and _query_e2 were added to replace _query_d and _query_e
+  // without updating BUILDIDS.  They can be renamed back the next time 
BUILDIDS
+  // is updated.
+  "create view if not exists " BUILDIDS "_query_d2 as \n"
   "select\n"
-  "b.hex as buildid, n.mtime, 'F' as sourcetype, f0.name as source0, 
n.mtime as mtime, null as source1\n"
+  "b.hex as buildid, 'F' as sourcetype, n.file as id0, f0.name as 
source0, n.mtime as mtime, null as id1, null as source1\n"
   "from " BUILDIDS "_buildids b, " BUILDIDS "_files_v f0, " BUILDIDS 
"_f_de n\n"
   "where b.id = n.buildid and f0.id = n.file and n.debuginfo_p = 1\n"
   "union all select\n"
-  "b.hex as buildid, n.mtime, 'R' as sourcetype, f0.name as source0, 
n.mtime as mtime, f1.name as source1\n"
+  "b.hex as buildid, 'R' as sourcetype, n.file as id0, f0.name as 
source0, n.mtime as mtime, n.content as id1, f1.name as source1\n"
   "from " BUILDIDS "_buildids b, " BUILDIDS "_files_v f0, " BUILDIDS 
"_files_v f1, " BUILDIDS "_r_de n\n"
   "where b.id = n.buildid and f0.id = n.file and f1.id = n.content and 
n.debuginfo_p = 1\n"
   ";"
   // ... and for E queries
-  "create view if not exists " BUILDIDS "_query_e as \n"
+  "create view if not exists " BUILDIDS "_query_e2 as \n"
   "select\n"
-  "b.hex as buildid, n.mtime, 'F' as sourcetype, f0.name as source0, 
n.mtime as mtime, null as source1\n"
+  "b.hex as buildid, 'F' as sourcetype, n.file as id0, f0.name as 
source0, n.mtime as mtime, null as id1, null as source1\n"
   "from " BUILDIDS "_buildids b, " BUILDIDS "_files_v f0, " BUILDIDS 
"_f_de n\n"
   "where b.id = n.buildid and f0.id = n.file and n.executable_p = 1\n"
   "union all select\n"
-  "b.hex as buildid, n.mtime, 'R' as sourcetype, f0.name as source0, 
n.mtime as mtime, f1.name as source1\n"
+  "b.hex as buildid, 'R

[PATCH v4 5/7] debuginfod: optimize extraction from seekable xz archives

[Omar Sandoval]

From: Omar Sandoval 

The kernel debuginfo packages on Fedora, Debian, and Ubuntu, and many of
their downstreams, are all compressed with xz in multi-threaded mode,
which allows random access.  We can use this to bypass the full archive
extraction and dramatically speed up kernel debuginfo requests (from ~50
seconds in the worst case to < 0.25 seconds).

This works because multi-threaded xz compression splits up the stream
into many independently compressed blocks.  The stream ends with an
index of blocks.  So, to seek to an offset, we find the block containing
that offset in the index and then decompress and throw away data until
we reach the offset within the block.  We can then decompress the
desired amount of data, possibly from subsequent blocks.  There's no
high-level API in liblzma to do this, but we can do it by stitching
together a few low-level APIs.

We need to pass down the file ids then look up the size, uncompressed
offset, and mtime in the _r_seekable table.  Note that this table is not
yet populated, so this commit has no functional change on its own.

Signed-off-by: Omar Sandoval 
---
 configure.ac  |   5 +
 debuginfod/Makefile.am|   2 +-
 debuginfod/debuginfod.cxx | 456 +-
 3 files changed, 457 insertions(+), 6 deletions(-)

diff --git a/configure.ac b/configure.ac
index 24e68d94..9c5f7e51 100644
--- a/configure.ac
+++ b/configure.ac
@@ -441,8 +441,13 @@ eu_ZIPLIB(bzlib,BZLIB,bz2,BZ2_bzdopen,bzip2)
 # We need this since bzip2 doesn't have a pkgconfig file.
 BZ2_LIB="$LIBS"
 AC_SUBST([BZ2_LIB])
+save_LIBS="$LIBS"
+LIBS=
 eu_ZIPLIB(lzma,LZMA,lzma,lzma_auto_decoder,[LZMA (xz)])
+lzma_LIBS="$LIBS"
+LIBS="$lzma_LIBS $save_LIBS"
 AS_IF([test "x$with_lzma" = xyes], [LIBLZMA="liblzma"], [LIBLZMA=""])
+AC_SUBST([lzma_LIBS])
 AC_SUBST([LIBLZMA])
 eu_ZIPLIB(zstd,ZSTD,zstd,ZSTD_decompress,[ZSTD (zst)])
 AS_IF([test "x$with_zstd" = xyes], [LIBZSTD="libzstd"], [LIBLZSTD=""])
diff --git a/debuginfod/Makefile.am b/debuginfod/Makefile.am
index b74e3673..e199dc0c 100644
--- a/debuginfod/Makefile.am
+++ b/debuginfod/Makefile.am
@@ -70,7 +70,7 @@ bin_PROGRAMS += debuginfod-find
 endif
 
 debuginfod_SOURCES = debuginfod.cxx
-debuginfod_LDADD = $(libdw) $(libelf) $(libeu) $(libdebuginfod) $(argp_LDADD) 
$(fts_LIBS) $(libmicrohttpd_LIBS) $(sqlite3_LIBS) $(libarchive_LIBS) 
$(rpm_LIBS) $(jsonc_LIBS) $(libcurl_LIBS) -lpthread -ldl
+debuginfod_LDADD = $(libdw) $(libelf) $(libeu) $(libdebuginfod) $(argp_LDADD) 
$(fts_LIBS) $(libmicrohttpd_LIBS) $(sqlite3_LIBS) $(libarchive_LIBS) 
$(rpm_LIBS) $(jsonc_LIBS) $(libcurl_LIBS) $(lzma_LIBS) -lpthread -ldl
 
 debuginfod_find_SOURCES = debuginfod-find.c
 debuginfod_find_LDADD = $(libdw) $(libelf) $(libeu) $(libdebuginfod) 
$(argp_LDADD) $(fts_LIBS) $(jsonc_LIBS)
diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index b3d80090..cf7f48ab 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -63,6 +63,10 @@ extern "C" {
 #undef __attribute__ /* glibc bug - rhbz 1763325 */
 #endif
 
+#ifdef USE_LZMA
+#include 
+#endif
+
 #include 
 #include 
 #include 
@@ -1961,6 +1965,385 @@ handle_buildid_f_match (bool internal_req_t,
   return r;
 }
 
+
+#ifdef USE_LZMA
+struct lzma_exception: public reportable_exception
+{
+  lzma_exception(int rc, const string& msg):
+// liblzma doesn't have a lzma_ret -> string conversion function, so just
+// report the value.
+reportable_exception(string ("lzma error: ") + msg + ": error " + 
to_string(rc)) {
+  inc_metric("error_count","lzma",to_string(rc));
+}
+};
+
+// Neither RPM nor deb files support seeking to a specific file in the package.
+// Instead, to extract a specific file, we normally need to read the archive
+// sequentially until we find the file.  This is very slow for files at the end
+// of a large package with lots of files, like kernel debuginfo.
+//
+// However, if the compression format used in the archive supports seeking, we
+// can accelerate this.  As of July 2024, xz is the only widely-used format 
that
+// supports seeking, and usually only in multi-threaded mode.  Luckily, the
+// kernel-debuginfo package in Fedora and its downstreams, and the
+// linux-image-*-dbg package in Debian and its downstreams, all happen to use
+// this.
+//
+// The xz format [1] ends with an index of independently compressed blocks in
+// the stream.  In RPM and deb files, the xz stream is the last thing in the
+// file, so we assume that the xz Stream Footer is at the end of the package
+// file and do everything relative to that.  For each file in the archive, we
+// remember the size and offset of the file data in the uncompressed xz stream,
+// then we use the index to seek to that offset when we need that file.
+//
+// 1: https://xz.tukaani.org/format/xz

[PATCH v4 3/7] debuginfod: factor out common code for responding from an archive

[Omar Sandoval]

From: Omar Sandoval 

handle_buildid_r_match has two very similar branches where it optionally
extracts a section and then creates a microhttpd response.  In
preparation for adding a third one, factor it out into a function.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx | 213 +-
 1 file changed, 96 insertions(+), 117 deletions(-)

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 92022f3d..24702c23 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -1965,6 +1965,81 @@ string canonicalized_archive_entry_pathname(struct 
archive_entry *e)
 }
 
 
+// NB: takes ownership of, and may reassign, fd.
+static struct MHD_Response*
+create_buildid_r_response (int64_t b_mtime0,
+   const string& b_source0,
+   const string& b_source1,
+   const string& section,
+   const string& ima_sig,
+   const char* tmppath,
+   int& fd,
+   off_t size,
+   time_t mtime,
+   const string& metric,
+   const struct timespec& extract_begin)
+{
+  if (tmppath != NULL)
+{
+  struct timespec extract_end;
+  clock_gettime (CLOCK_MONOTONIC, &extract_end);
+  double extract_time = (extract_end.tv_sec - extract_begin.tv_sec)
++ (extract_end.tv_nsec - extract_begin.tv_nsec)/1.e9;
+  fdcache.intern(b_source0, b_source1, tmppath, size, true, extract_time);
+}
+
+  if (!section.empty ())
+{
+  int scn_fd = extract_section (fd, b_mtime0,
+b_source0 + ":" + b_source1,
+section, extract_begin);
+  close (fd);
+  if (scn_fd >= 0)
+fd = scn_fd;
+  else
+{
+  if (verbose)
+obatched (clog) << "cannot find section " << section
+<< " for archive " << b_source0
+<< " file " << b_source1 << endl;
+  return 0;
+}
+
+  struct stat fs;
+  if (fstat (fd, &fs) < 0)
+{
+  close (fd);
+  throw libc_exception (errno,
+string ("fstat ") + b_source0 + string (" ") + section);
+}
+  size = fs.st_size;
+}
+
+  struct MHD_Response* r = MHD_create_response_from_fd (size, fd);
+  if (r == 0)
+{
+  if (verbose)
+obatched(clog) << "cannot create fd-response for " << b_source0 << 
endl;
+  close(fd);
+}
+  else
+{
+  inc_metric ("http_responses_total","result",metric);
+  add_mhd_response_header (r, "Content-Type", "application/octet-stream");
+  add_mhd_response_header (r, "X-DEBUGINFOD-SIZE", 
to_string(size).c_str());
+  add_mhd_response_header (r, "X-DEBUGINFOD-ARCHIVE", b_source0.c_str());
+  add_mhd_response_header (r, "X-DEBUGINFOD-FILE", b_source1.c_str());
+  if(!ima_sig.empty()) add_mhd_response_header(r, 
"X-DEBUGINFOD-IMASIGNATURE", ima_sig.c_str());
+  add_mhd_last_modified (r, mtime);
+  if (verbose > 1)
+obatched(clog) << "serving " << metric << " " << b_source0
+   << " file " << b_source1
+   << " section=" << section
+   << " IMA signature=" << ima_sig << endl;
+  /* libmicrohttpd will close fd. */
+}
+  return r;
+}
 
 static struct MHD_Response*
 handle_buildid_r_match (bool internal_req_p,
@@ -2142,57 +2217,15 @@ handle_buildid_r_match (bool internal_req_p,
   break; // branch out of if "loop", to try new libarchive fetch 
attempt
 }
 
-  if (!section.empty ())
-   {
- int scn_fd = extract_section (fd, fs.st_mtime,
-   b_source0 + ":" + b_source1,
-   section, extract_begin);
- close (fd);
- if (scn_fd >= 0)
-   fd = scn_fd;
- else
-   {
- if (verbose)
-   obatched (clog) << "cannot find section " << section
-   << " for archive " << b_source0
-   << " file " << b_source1 << endl;
- return 0;
-   }
-
- rc = fstat(fd, &fs);
- if (rc < 0)
-   {
- close (fd);
- throw libc_exception (errno,
-  

[PATCH v4 7/7] debuginfod: populate _r_seekable on request

[Omar Sandoval]

From: Omar Sandoval 

Since the schema change adding _r_seekable was done in a backward
compatible way, seekable archives that were previously scanned will not
be in _r_seekable.  Whenever an archive is going to be extracted to
satisfy a request, check if it is seekable.  If so, populate _r_seekable
while extracting it so that future requests use the optimized path.

The next time that BUILDIDS is bumped, all archives will be checked at
scan time.  At that point, checking again will be unnecessary and this
commit (including the test case modification) can be reverted.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx| 76 +---
 tests/run-debuginfod-seekable.sh | 45 +++
 2 files changed, 115 insertions(+), 6 deletions(-)

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 5fe2db0c..fb7873ae 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -2740,6 +2740,7 @@ handle_buildid_r_match (bool internal_req_p,
 }
 
   // no match ... look for a seekable entry
+  bool populate_seekable = ! passive_p;
   unique_ptr pp (new sqlite_ps (internal_req_p ? db : dbq,
"rpm-seekable-query",
"select type, size, offset, mtime 
from " BUILDIDS "_r_seekable "
@@ -2749,6 +2750,9 @@ handle_buildid_r_match (bool internal_req_p,
 {
   if (rc != SQLITE_ROW)
 throw sqlite_exception(rc, "step");
+  // if we found a match in _r_seekable but we fail to extract it, don't
+  // bother populating it again
+  populate_seekable = false;
   const char* seekable_type = (const char*) sqlite3_column_text (*pp, 0);
   if (seekable_type != NULL && strcmp (seekable_type, "xz") == 0)
 {
@@ -2840,16 +2844,39 @@ handle_buildid_r_match (bool internal_req_p,
   throw archive_exception(a, "cannot open archive from pipe");
 }
 
-  // archive traversal is in three stages, no, four stages:
-  // 1) skip entries whose names do not match the requested one
-  // 2) extract the matching entry name (set r = result)
-  // 3) extract some number of prefetched entries (just into fdcache)
-  // 4) abort any further processing
+  // If the archive was scanned in a version without _r_seekable, then we may
+  // need to populate _r_seekable now.  This can be removed the next time
+  // BUILDIDS is updated.
+  if (populate_seekable)
+{
+  populate_seekable = is_seekable_archive (b_source0, a);
+  if (populate_seekable)
+{
+  // NB: the names are already interned
+  pp.reset(new sqlite_ps (db, "rpm-seekable-insert2",
+  "insert or ignore into " BUILDIDS 
"_r_seekable (file, content, type, size, offset, mtime) "
+  "values (?, "
+  "(select id from " BUILDIDS "_files "
+  "where dirname = (select id from " BUILDIDS 
"_fileparts where name = ?) "
+  "and basename = (select id from " BUILDIDS 
"_fileparts where name = ?) "
+  "), 'xz', ?, ?, ?)"));
+}
+}
+
+  // archive traversal is in five stages:
+  // 1) before we find a matching entry, insert it into _r_seekable if needed 
or
+  //skip it otherwise
+  // 2) extract the matching entry (set r = result).  Also insert it into
+  //_r_seekable if needed
+  // 3) extract some number of prefetched entries (just into fdcache).  Also
+  //insert them into _r_seekable if needed
+  // 4) if needed, insert all of the remaining entries into _r_seekable
+  // 5) abort any further processing
   struct MHD_Response* r = 0; // will set in stage 2
   unsigned prefetch_count =
 internal_req_p ? 0 : fdcache_prefetch;// will decrement in stage 3
 
-  while(r == 0 || prefetch_count > 0) // stage 1, 2, or 3
+  while(r == 0 || prefetch_count > 0 || populate_seekable) // stage 1-4
 {
   if (interrupted)
 break;
@@ -2863,6 +2890,43 @@ handle_buildid_r_match (bool internal_req_p,
 continue;
 
   string fn = canonicalized_archive_entry_pathname (e);
+
+  if (populate_seekable)
+{
+  string dn, bn;
+  size_t slash = fn.rfind('/');
+  if (slash == std::string::npos) {
+dn = "";
+bn = fn;
+  } else {
+dn = fn.substr(0, slash);
+bn = fn.substr(slash + 1);
+  }
+
+  int64_t seekable_size = archive_entry_size (e);
+  int64_t seekable_offset = archive_filter_bytes (a, 0);
+  time_t seekable_mtime = archive_entry_mtime (e);
+
+  pp->reset();
+

[PATCH v4 6/7] debuginfod: populate _r_seekable on scan

[Omar Sandoval]

From: Omar Sandoval 

Whenever a new archive is scanned, check if it is seekable with a little
liblzma magic, and populate _r_seekable if so.  With this, newly scanned
seekable archives will used the optimized extraction path added in the
previous commit.  Also add a test case using some artificial packages.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx | 145 +-
 tests/Makefile.am |  13 +-
 ...pressme-seekable-xz-dbgsym_1.0-1_amd64.deb | Bin 0 -> 6288 bytes
 ...compressme-seekable-xz_1.0-1.debian.tar.xz | Bin 0 -> 1440 bytes
 .../compressme-seekable-xz_1.0-1.dsc  |  19 +++
 .../compressme-seekable-xz_1.0-1_amd64.deb| Bin 0 -> 6208 bytes
 .../compressme-seekable-xz_1.0.orig.tar.xz| Bin 0 -> 7160 bytes
 .../compressme-seekable-xz-1.0-1.src.rpm  | Bin 0 -> 15880 bytes
 .../compressme-seekable-xz-1.0-1.x86_64.rpm   | Bin 0 -> 31873 bytes
 ...sme-seekable-xz-debuginfo-1.0-1.x86_64.rpm | Bin 0 -> 21917 bytes
 ...e-seekable-xz-debugsource-1.0-1.x86_64.rpm | Bin 0 -> 7961 bytes
 tests/run-debuginfod-seekable.sh  | 141 +
 12 files changed, 313 insertions(+), 5 deletions(-)
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz-dbgsym_1.0-1_amd64.deb
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0-1.debian.tar.xz
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0-1.dsc
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0-1_amd64.deb
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0.orig.tar.xz
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-1.0-1.src.rpm
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-1.0-1.x86_64.rpm
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-debuginfo-1.0-1.x86_64.rpm
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-debugsource-1.0-1.x86_64.rpm
 create mode 100755 tests/run-debuginfod-seekable.sh

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index cf7f48ab..5fe2db0c 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -1998,6 +1998,109 @@ struct lzma_exception: public reportable_exception
 //
 // 1: https://xz.tukaani.org/format/xz-file-format.txt
 
+// Return whether an archive supports seeking.
+static bool
+is_seekable_archive (const string& rps, struct archive* a)
+{
+  // Only xz supports seeking.
+  if (archive_filter_code (a, 0) != ARCHIVE_FILTER_XZ)
+return false;
+
+  int fd = open (rps.c_str(), O_RDONLY);
+  if (fd < 0)
+return false;
+  defer_dtor fd_closer (fd, close);
+
+  // Seek to the xz Stream Footer.  We assume that it's the last thing in the
+  // file, which is true for RPM and deb files.
+  off_t footer_pos = -LZMA_STREAM_HEADER_SIZE;
+  if (lseek (fd, footer_pos, SEEK_END) == -1)
+return false;
+
+  // Decode the Stream Footer.
+  uint8_t footer[LZMA_STREAM_HEADER_SIZE];
+  size_t footer_read = 0;
+  while (footer_read < sizeof (footer))
+{
+  ssize_t bytes_read = read (fd, footer + footer_read,
+ sizeof (footer) - footer_read);
+  if (bytes_read < 0)
+{
+  if (errno == EINTR)
+continue;
+  return false;
+}
+  if (bytes_read == 0)
+return false;
+  footer_read += bytes_read;
+}
+
+  lzma_stream_flags stream_flags;
+  lzma_ret ret = lzma_stream_footer_decode (&stream_flags, footer);
+  if (ret != LZMA_OK)
+return false;
+
+  // Seek to the xz Index.
+  if (lseek (fd, footer_pos - stream_flags.backward_size, SEEK_END) == -1)
+return false;
+
+  // Decode the Number of Records in the Index.  liblzma doesn't have an API 
for
+  // this if you don't want to decode the whole Index, so we have to do it
+  // ourselves.
+  //
+  // We need 1 byte for the Index Indicator plus 1-9 bytes for the
+  // variable-length integer Number of Records.
+  uint8_t index[10];
+  size_t index_read = 0;
+  while (index_read == 0) {
+  ssize_t bytes_read = read (fd, index, sizeof (index));
+  if (bytes_read < 0)
+{
+  if (errno == EINTR)
+continue;
+  return false;
+}
+  if (bytes_read == 0)
+return false;
+  index_read += bytes_read;
+  }
+  // The Index Indicator must be 0.
+  if (index[0] != 0)
+return false;
+
+  lzma_vli num_records;
+  size_t pos = 0;
+  size_t in_pos = 1;
+  while (true)
+{
+  if (in_pos >= index_read)
+{
+  ssize_t bytes_read = read (fd, index, sizeof (index));
+  if (bytes_read < 0)
+  {
+if (errno == EINTR)
+  continue;
+return false;
+  }
+  if (bytes_read == 0)
+return

Re: [PATCH v4 0/7] debuginfod: speed up extraction from kernel debuginfo packages by 200x

[Omar Sandoval]

On Tue, Jul 23, 2024 at 05:47:50PM -0400, Aaron Merey wrote:
> Hi Omar,
> 
> On Fri, Jul 19, 2024 at 2:24 PM Omar Sandoval  wrote:
> >
> > From: Omar Sandoval 
> >
> > This is v4 of my patch series optimizing debuginfod for kernel
> > debuginfo.  v1 is here [1], v2 is here [2], v3 is here [3].  The only
> > changes from v3 in this version are fixing a bogus maybe-uninitialized
> > error on the Debian build and adding the new test files to EXTRA_DIST so
> > that make distcheck passes.
> >
> > Thanks,
> > Omar
> >
> > 1: https://sourceware.org/pipermail/elfutils-devel/2024q3/007191.html
> > 2: https://sourceware.org/pipermail/elfutils-devel/2024q3/007208.html
> > 3: https://sourceware.org/pipermail/elfutils-devel/2024q3/007243.html
> 
> Thanks for working on these patches.  I ran v4 on the sourceware buildbots.
> The new testcase fails on debian-ppc64 [1], debian-i386 [2] and
> fedora-s390x [3].  There was a 4th failing buildbot but AFAICT the failure
> is unrelated to your patches.
> 
> Do you mind taking a look at this?  The patches otherwise LGTM and I'll
> merge them once this is passing on the buildbots.

Thanks for the test run, and sorry for the churn.  It looks like these
are all failing with "error: no fdcache hits" because every addition to
the fdcache results in "fdcache emergency flush for filling tmpdir".  I
see that run-debuginfod-fd-prefetch-caches.sh has a workaround for this
which I'll copy.

Thanks,
Omar

[PATCH v5 1/7] debuginfod: fix skipping source file

[Omar Sandoval]

From: Omar Sandoval 

dwarf_extract_source_paths explicitly skips source files that equal
"", but dwarf_filesrc may return a path like "dir/".
Check for and skip that case, too.

In particular, the test debuginfod RPMs have paths like this.  However,
the test cases didn't catch this because they have a bug, too: they
follow symlinks, which results in double-counting every file.  Fix that,
too.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx | 3 ++-
 tests/run-debuginfod-archive-groom.sh | 2 +-
 tests/run-debuginfod-extraction.sh| 2 +-
 3 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 305edde8..92022f3d 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -3446,7 +3446,8 @@ dwarf_extract_source_paths (Elf *elf, set& 
debug_sourcefiles)
   if (hat == NULL)
 continue;
 
-  if (string(hat) == "") // gcc intrinsics, don't bother 
record
+  if (string(hat) == ""
+  || string_endswith(hat, "")) // gcc intrinsics, don't 
bother record
 continue;
 
   string waldo;
diff --git a/tests/run-debuginfod-archive-groom.sh 
b/tests/run-debuginfod-archive-groom.sh
index e2c394ef..0131158f 100755
--- a/tests/run-debuginfod-archive-groom.sh
+++ b/tests/run-debuginfod-archive-groom.sh
@@ -109,7 +109,7 @@ for i in $newrpms; do
 rpm2cpio ../$i | cpio -ivd;
 cd ..;
 done
-sourcefiles=$(find -name \*\\.debug \
+sourcefiles=$(find -name \*\\.debug -type f \
   | env LD_LIBRARY_PATH=$ldpath xargs \
 ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
   | grep mtime: | wc --lines)
diff --git a/tests/run-debuginfod-extraction.sh 
b/tests/run-debuginfod-extraction.sh
index da6b25cf..f49dc6f6 100755
--- a/tests/run-debuginfod-extraction.sh
+++ b/tests/run-debuginfod-extraction.sh
@@ -94,7 +94,7 @@ for i in $newrpms; do
 rpm2cpio ../$i | cpio -ivd;
 cd ..;
 done
-sourcefiles=$(find -name \*\\.debug \
+sourcefiles=$(find -name \*\\.debug -type f \
   | env LD_LIBRARY_PATH=$ldpath xargs \
 ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
   | grep mtime: | wc --lines)
-- 
2.45.2

[PATCH v5 0/7] debuginfod: speed up extraction from kernel debuginfo packages by 200x

[Omar Sandoval]

From: Omar Sandoval 

This is v4 of my patch series optimizing debuginfod for kernel
debuginfo.  v1 is here [1], v2 is here [2], v3 is here [3], v4 is here
[4].  The only change from v4 in this version is adding --fdcache-mbs
and --fdcache-mintmp to the new test to fix some sporadic test failures.
Hopefully this version finally gets a clean test run.

Thanks,
Omar

1: https://sourceware.org/pipermail/elfutils-devel/2024q3/007191.html
2: https://sourceware.org/pipermail/elfutils-devel/2024q3/007208.html
3: https://sourceware.org/pipermail/elfutils-devel/2024q3/007243.html
4: https://sourceware.org/pipermail/elfutils-devel/2024q3/007255.html

Omar Sandoval (7):
  debuginfod: fix skipping  source file
  tests/run-debuginfod-fd-prefetch-caches.sh: disable fdcache limit
check
  debuginfod: factor out common code for responding from an archive
  debugifod: add new table and views for seekable archives
  debuginfod: optimize extraction from seekable xz archives
  debuginfod: populate _r_seekable on scan
  debuginfod: populate _r_seekable on request

 configure.ac  |   5 +
 debuginfod/Makefile.am|   2 +-
 debuginfod/debuginfod.cxx | 923 +++---
 tests/Makefile.am |  13 +-
 ...pressme-seekable-xz-dbgsym_1.0-1_amd64.deb | Bin 0 -> 6288 bytes
 ...compressme-seekable-xz_1.0-1.debian.tar.xz | Bin 0 -> 1440 bytes
 .../compressme-seekable-xz_1.0-1.dsc  |  19 +
 .../compressme-seekable-xz_1.0-1_amd64.deb| Bin 0 -> 6208 bytes
 .../compressme-seekable-xz_1.0.orig.tar.xz| Bin 0 -> 7160 bytes
 .../compressme-seekable-xz-1.0-1.src.rpm  | Bin 0 -> 15880 bytes
 .../compressme-seekable-xz-1.0-1.x86_64.rpm   | Bin 0 -> 31873 bytes
 ...sme-seekable-xz-debuginfo-1.0-1.x86_64.rpm | Bin 0 -> 21917 bytes
 ...e-seekable-xz-debugsource-1.0-1.x86_64.rpm | Bin 0 -> 7961 bytes
 tests/run-debuginfod-archive-groom.sh |   2 +-
 tests/run-debuginfod-extraction.sh|   2 +-
 tests/run-debuginfod-fd-prefetch-caches.sh|   4 +
 tests/run-debuginfod-seekable.sh  | 192 
 17 files changed, 1017 insertions(+), 145 deletions(-)
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz-dbgsym_1.0-1_amd64.deb
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0-1.debian.tar.xz
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0-1.dsc
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0-1_amd64.deb
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0.orig.tar.xz
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-1.0-1.src.rpm
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-1.0-1.x86_64.rpm
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-debuginfo-1.0-1.x86_64.rpm
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-debugsource-1.0-1.x86_64.rpm
 create mode 100755 tests/run-debuginfod-seekable.sh

-- 
2.45.2

[PATCH v5 2/7] tests/run-debuginfod-fd-prefetch-caches.sh: disable fdcache limit check

[Omar Sandoval]

From: Omar Sandoval 

Since commit acd9525e93d7 ("PR31265 - rework debuginfod archive-extract
fdcache"), the fdcache limit is only applied when a new file is interned
and it has been at least 10 seconds since the limit was last applied.
This means that the fdcache can go over the limit temporarily.

run-debuginfod-fd-prefetch-caches.sh happens to avoid tripping over this
because of lucky sizes of the files used in the test.  However, adding
new files for an upcoming test exposed this failure.

Disable this part of the test for now.

Signed-off-by: Omar Sandoval 
---
 tests/run-debuginfod-fd-prefetch-caches.sh | 4 
 1 file changed, 4 insertions(+)

diff --git a/tests/run-debuginfod-fd-prefetch-caches.sh 
b/tests/run-debuginfod-fd-prefetch-caches.sh
index 3db78ade..90730555 100755
--- a/tests/run-debuginfod-fd-prefetch-caches.sh
+++ b/tests/run-debuginfod-fd-prefetch-caches.sh
@@ -99,6 +99,9 @@ kill $PID1
 wait $PID1
 PID1=0
 
+# Since we now only limit the fd cache every 10 seconds, it can temporarily go
+# over the limit.  That makes this part of the test unreliable.
+if false; then
 #
 # Test mb limit on fd cache
 #
@@ -148,3 +151,4 @@ kill $PID1
 wait $PID1
 PID1=0
 exit 0
+fi
-- 
2.45.2

[PATCH v5 7/7] debuginfod: populate _r_seekable on request

[Omar Sandoval]

From: Omar Sandoval 

Since the schema change adding _r_seekable was done in a backward
compatible way, seekable archives that were previously scanned will not
be in _r_seekable.  Whenever an archive is going to be extracted to
satisfy a request, check if it is seekable.  If so, populate _r_seekable
while extracting it so that future requests use the optimized path.

The next time that BUILDIDS is bumped, all archives will be checked at
scan time.  At that point, checking again will be unnecessary and this
commit (including the test case modification) can be reverted.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx| 76 +---
 tests/run-debuginfod-seekable.sh | 48 
 2 files changed, 118 insertions(+), 6 deletions(-)

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 5fe2db0c..fb7873ae 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -2740,6 +2740,7 @@ handle_buildid_r_match (bool internal_req_p,
 }
 
   // no match ... look for a seekable entry
+  bool populate_seekable = ! passive_p;
   unique_ptr pp (new sqlite_ps (internal_req_p ? db : dbq,
"rpm-seekable-query",
"select type, size, offset, mtime 
from " BUILDIDS "_r_seekable "
@@ -2749,6 +2750,9 @@ handle_buildid_r_match (bool internal_req_p,
 {
   if (rc != SQLITE_ROW)
 throw sqlite_exception(rc, "step");
+  // if we found a match in _r_seekable but we fail to extract it, don't
+  // bother populating it again
+  populate_seekable = false;
   const char* seekable_type = (const char*) sqlite3_column_text (*pp, 0);
   if (seekable_type != NULL && strcmp (seekable_type, "xz") == 0)
 {
@@ -2840,16 +2844,39 @@ handle_buildid_r_match (bool internal_req_p,
   throw archive_exception(a, "cannot open archive from pipe");
 }
 
-  // archive traversal is in three stages, no, four stages:
-  // 1) skip entries whose names do not match the requested one
-  // 2) extract the matching entry name (set r = result)
-  // 3) extract some number of prefetched entries (just into fdcache)
-  // 4) abort any further processing
+  // If the archive was scanned in a version without _r_seekable, then we may
+  // need to populate _r_seekable now.  This can be removed the next time
+  // BUILDIDS is updated.
+  if (populate_seekable)
+{
+  populate_seekable = is_seekable_archive (b_source0, a);
+  if (populate_seekable)
+{
+  // NB: the names are already interned
+  pp.reset(new sqlite_ps (db, "rpm-seekable-insert2",
+  "insert or ignore into " BUILDIDS 
"_r_seekable (file, content, type, size, offset, mtime) "
+  "values (?, "
+  "(select id from " BUILDIDS "_files "
+  "where dirname = (select id from " BUILDIDS 
"_fileparts where name = ?) "
+  "and basename = (select id from " BUILDIDS 
"_fileparts where name = ?) "
+  "), 'xz', ?, ?, ?)"));
+}
+}
+
+  // archive traversal is in five stages:
+  // 1) before we find a matching entry, insert it into _r_seekable if needed 
or
+  //skip it otherwise
+  // 2) extract the matching entry (set r = result).  Also insert it into
+  //_r_seekable if needed
+  // 3) extract some number of prefetched entries (just into fdcache).  Also
+  //insert them into _r_seekable if needed
+  // 4) if needed, insert all of the remaining entries into _r_seekable
+  // 5) abort any further processing
   struct MHD_Response* r = 0; // will set in stage 2
   unsigned prefetch_count =
 internal_req_p ? 0 : fdcache_prefetch;// will decrement in stage 3
 
-  while(r == 0 || prefetch_count > 0) // stage 1, 2, or 3
+  while(r == 0 || prefetch_count > 0 || populate_seekable) // stage 1-4
 {
   if (interrupted)
 break;
@@ -2863,6 +2890,43 @@ handle_buildid_r_match (bool internal_req_p,
 continue;
 
   string fn = canonicalized_archive_entry_pathname (e);
+
+  if (populate_seekable)
+{
+  string dn, bn;
+  size_t slash = fn.rfind('/');
+  if (slash == std::string::npos) {
+dn = "";
+bn = fn;
+  } else {
+dn = fn.substr(0, slash);
+bn = fn.substr(slash + 1);
+  }
+
+  int64_t seekable_size = archive_entry_size (e);
+  int64_t seekable_offset = archive_filter_bytes (a, 0);
+  time_t seekable_mtime = archive_entry_mtime (e);
+
+  pp->reset();
+

[PATCH v5 3/7] debuginfod: factor out common code for responding from an archive

[Omar Sandoval]

From: Omar Sandoval 

handle_buildid_r_match has two very similar branches where it optionally
extracts a section and then creates a microhttpd response.  In
preparation for adding a third one, factor it out into a function.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx | 213 +-
 1 file changed, 96 insertions(+), 117 deletions(-)

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 92022f3d..24702c23 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -1965,6 +1965,81 @@ string canonicalized_archive_entry_pathname(struct 
archive_entry *e)
 }
 
 
+// NB: takes ownership of, and may reassign, fd.
+static struct MHD_Response*
+create_buildid_r_response (int64_t b_mtime0,
+   const string& b_source0,
+   const string& b_source1,
+   const string& section,
+   const string& ima_sig,
+   const char* tmppath,
+   int& fd,
+   off_t size,
+   time_t mtime,
+   const string& metric,
+   const struct timespec& extract_begin)
+{
+  if (tmppath != NULL)
+{
+  struct timespec extract_end;
+  clock_gettime (CLOCK_MONOTONIC, &extract_end);
+  double extract_time = (extract_end.tv_sec - extract_begin.tv_sec)
++ (extract_end.tv_nsec - extract_begin.tv_nsec)/1.e9;
+  fdcache.intern(b_source0, b_source1, tmppath, size, true, extract_time);
+}
+
+  if (!section.empty ())
+{
+  int scn_fd = extract_section (fd, b_mtime0,
+b_source0 + ":" + b_source1,
+section, extract_begin);
+  close (fd);
+  if (scn_fd >= 0)
+fd = scn_fd;
+  else
+{
+  if (verbose)
+obatched (clog) << "cannot find section " << section
+<< " for archive " << b_source0
+<< " file " << b_source1 << endl;
+  return 0;
+}
+
+  struct stat fs;
+  if (fstat (fd, &fs) < 0)
+{
+  close (fd);
+  throw libc_exception (errno,
+string ("fstat ") + b_source0 + string (" ") + section);
+}
+  size = fs.st_size;
+}
+
+  struct MHD_Response* r = MHD_create_response_from_fd (size, fd);
+  if (r == 0)
+{
+  if (verbose)
+obatched(clog) << "cannot create fd-response for " << b_source0 << 
endl;
+  close(fd);
+}
+  else
+{
+  inc_metric ("http_responses_total","result",metric);
+  add_mhd_response_header (r, "Content-Type", "application/octet-stream");
+  add_mhd_response_header (r, "X-DEBUGINFOD-SIZE", 
to_string(size).c_str());
+  add_mhd_response_header (r, "X-DEBUGINFOD-ARCHIVE", b_source0.c_str());
+  add_mhd_response_header (r, "X-DEBUGINFOD-FILE", b_source1.c_str());
+  if(!ima_sig.empty()) add_mhd_response_header(r, 
"X-DEBUGINFOD-IMASIGNATURE", ima_sig.c_str());
+  add_mhd_last_modified (r, mtime);
+  if (verbose > 1)
+obatched(clog) << "serving " << metric << " " << b_source0
+   << " file " << b_source1
+   << " section=" << section
+   << " IMA signature=" << ima_sig << endl;
+  /* libmicrohttpd will close fd. */
+}
+  return r;
+}
 
 static struct MHD_Response*
 handle_buildid_r_match (bool internal_req_p,
@@ -2142,57 +2217,15 @@ handle_buildid_r_match (bool internal_req_p,
   break; // branch out of if "loop", to try new libarchive fetch 
attempt
 }
 
-  if (!section.empty ())
-   {
- int scn_fd = extract_section (fd, fs.st_mtime,
-   b_source0 + ":" + b_source1,
-   section, extract_begin);
- close (fd);
- if (scn_fd >= 0)
-   fd = scn_fd;
- else
-   {
- if (verbose)
-   obatched (clog) << "cannot find section " << section
-   << " for archive " << b_source0
-   << " file " << b_source1 << endl;
- return 0;
-   }
-
- rc = fstat(fd, &fs);
- if (rc < 0)
-   {
- close (fd);
- throw libc_exception (errno,
-  

[PATCH v5 6/7] debuginfod: populate _r_seekable on scan

[Omar Sandoval]

From: Omar Sandoval 

Whenever a new archive is scanned, check if it is seekable with a little
liblzma magic, and populate _r_seekable if so.  With this, newly scanned
seekable archives will used the optimized extraction path added in the
previous commit.  Also add a test case using some artificial packages.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx | 145 +-
 tests/Makefile.am |  13 +-
 ...pressme-seekable-xz-dbgsym_1.0-1_amd64.deb | Bin 0 -> 6288 bytes
 ...compressme-seekable-xz_1.0-1.debian.tar.xz | Bin 0 -> 1440 bytes
 .../compressme-seekable-xz_1.0-1.dsc  |  19 +++
 .../compressme-seekable-xz_1.0-1_amd64.deb| Bin 0 -> 6208 bytes
 .../compressme-seekable-xz_1.0.orig.tar.xz| Bin 0 -> 7160 bytes
 .../compressme-seekable-xz-1.0-1.src.rpm  | Bin 0 -> 15880 bytes
 .../compressme-seekable-xz-1.0-1.x86_64.rpm   | Bin 0 -> 31873 bytes
 ...sme-seekable-xz-debuginfo-1.0-1.x86_64.rpm | Bin 0 -> 21917 bytes
 ...e-seekable-xz-debugsource-1.0-1.x86_64.rpm | Bin 0 -> 7961 bytes
 tests/run-debuginfod-seekable.sh  | 144 +
 12 files changed, 316 insertions(+), 5 deletions(-)
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz-dbgsym_1.0-1_amd64.deb
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0-1.debian.tar.xz
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0-1.dsc
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0-1_amd64.deb
 create mode 100644 
tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0.orig.tar.xz
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-1.0-1.src.rpm
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-1.0-1.x86_64.rpm
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-debuginfo-1.0-1.x86_64.rpm
 create mode 100644 
tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-debugsource-1.0-1.x86_64.rpm
 create mode 100755 tests/run-debuginfod-seekable.sh

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index cf7f48ab..5fe2db0c 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -1998,6 +1998,109 @@ struct lzma_exception: public reportable_exception
 //
 // 1: https://xz.tukaani.org/format/xz-file-format.txt
 
+// Return whether an archive supports seeking.
+static bool
+is_seekable_archive (const string& rps, struct archive* a)
+{
+  // Only xz supports seeking.
+  if (archive_filter_code (a, 0) != ARCHIVE_FILTER_XZ)
+return false;
+
+  int fd = open (rps.c_str(), O_RDONLY);
+  if (fd < 0)
+return false;
+  defer_dtor fd_closer (fd, close);
+
+  // Seek to the xz Stream Footer.  We assume that it's the last thing in the
+  // file, which is true for RPM and deb files.
+  off_t footer_pos = -LZMA_STREAM_HEADER_SIZE;
+  if (lseek (fd, footer_pos, SEEK_END) == -1)
+return false;
+
+  // Decode the Stream Footer.
+  uint8_t footer[LZMA_STREAM_HEADER_SIZE];
+  size_t footer_read = 0;
+  while (footer_read < sizeof (footer))
+{
+  ssize_t bytes_read = read (fd, footer + footer_read,
+ sizeof (footer) - footer_read);
+  if (bytes_read < 0)
+{
+  if (errno == EINTR)
+continue;
+  return false;
+}
+  if (bytes_read == 0)
+return false;
+  footer_read += bytes_read;
+}
+
+  lzma_stream_flags stream_flags;
+  lzma_ret ret = lzma_stream_footer_decode (&stream_flags, footer);
+  if (ret != LZMA_OK)
+return false;
+
+  // Seek to the xz Index.
+  if (lseek (fd, footer_pos - stream_flags.backward_size, SEEK_END) == -1)
+return false;
+
+  // Decode the Number of Records in the Index.  liblzma doesn't have an API 
for
+  // this if you don't want to decode the whole Index, so we have to do it
+  // ourselves.
+  //
+  // We need 1 byte for the Index Indicator plus 1-9 bytes for the
+  // variable-length integer Number of Records.
+  uint8_t index[10];
+  size_t index_read = 0;
+  while (index_read == 0) {
+  ssize_t bytes_read = read (fd, index, sizeof (index));
+  if (bytes_read < 0)
+{
+  if (errno == EINTR)
+continue;
+  return false;
+}
+  if (bytes_read == 0)
+return false;
+  index_read += bytes_read;
+  }
+  // The Index Indicator must be 0.
+  if (index[0] != 0)
+return false;
+
+  lzma_vli num_records;
+  size_t pos = 0;
+  size_t in_pos = 1;
+  while (true)
+{
+  if (in_pos >= index_read)
+{
+  ssize_t bytes_read = read (fd, index, sizeof (index));
+  if (bytes_read < 0)
+  {
+if (errno == EINTR)
+  continue;
+return false;
+  }
+  if (bytes_read == 0)
+return

[PATCH v5 4/7] debugifod: add new table and views for seekable archives

[Omar Sandoval]

From: Omar Sandoval 

In order to extract a file from a seekable archive, we need to know
where in the uncompressed archive the file data starts and its size.
Additionally, in order to populate the response headers, we need the
file modification time (since we won't be able to get it from the
archive metadata).  Add a new table, _r_seekable, keyed on the archive
file id and entry file id and containing the size, offset, and mtime.
It also contains the compression type just in case new seekable formats
are supported in the future.

In order to search this table when we get a request, we need the file
ids available.  Add the ids to the _query_d and _query_e views, and
rename them to _query_d2 and _query_e2.

This schema change is backward compatible and doesn't require
reindexing.  _query_d2 and _query_e2 can be renamed back the next time
BUILDIDS needs to be bumped.

Before this change, the database for a single kernel debuginfo RPM
(kernel-debuginfo-6.9.6-200.fc40.x86_64.rpm) was about 15MB.  This
change increases that by about 70kB, only a 0.5% increase.

Signed-off-by: Omar Sandoval 
---
 debuginfod/debuginfod.cxx | 34 --
 1 file changed, 24 insertions(+), 10 deletions(-)

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 24702c23..b3d80090 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -265,25 +265,39 @@ static const char DEBUGINFOD_SQLITE_DDL[] =
   "foreign key (content) references " BUILDIDS "_files(id) on update 
cascade on delete cascade,\n"
   "primary key (content, file, mtime)\n"
   ") " WITHOUT_ROWID ";\n"
+  "create table if not exists " BUILDIDS "_r_seekable (\n" // seekable rpm 
contents
+  "file integer not null,\n"
+  "content integer not null,\n"
+  "type text not null,\n"
+  "size integer not null,\n"
+  "offset integer not null,\n"
+  "mtime integer not null,\n"
+  "foreign key (file) references " BUILDIDS "_files(id) on update 
cascade on delete cascade,\n"
+  "foreign key (content) references " BUILDIDS "_files(id) on update 
cascade on delete cascade,\n"
+  "primary key (file, content)\n"
+  ") " WITHOUT_ROWID ";\n"
   // create views to glue together some of the above tables, for webapi D 
queries
-  "create view if not exists " BUILDIDS "_query_d as \n"
+  // NB: _query_d2 and _query_e2 were added to replace _query_d and _query_e
+  // without updating BUILDIDS.  They can be renamed back the next time 
BUILDIDS
+  // is updated.
+  "create view if not exists " BUILDIDS "_query_d2 as \n"
   "select\n"
-  "b.hex as buildid, n.mtime, 'F' as sourcetype, f0.name as source0, 
n.mtime as mtime, null as source1\n"
+  "b.hex as buildid, 'F' as sourcetype, n.file as id0, f0.name as 
source0, n.mtime as mtime, null as id1, null as source1\n"
   "from " BUILDIDS "_buildids b, " BUILDIDS "_files_v f0, " BUILDIDS 
"_f_de n\n"
   "where b.id = n.buildid and f0.id = n.file and n.debuginfo_p = 1\n"
   "union all select\n"
-  "b.hex as buildid, n.mtime, 'R' as sourcetype, f0.name as source0, 
n.mtime as mtime, f1.name as source1\n"
+  "b.hex as buildid, 'R' as sourcetype, n.file as id0, f0.name as 
source0, n.mtime as mtime, n.content as id1, f1.name as source1\n"
   "from " BUILDIDS "_buildids b, " BUILDIDS "_files_v f0, " BUILDIDS 
"_files_v f1, " BUILDIDS "_r_de n\n"
   "where b.id = n.buildid and f0.id = n.file and f1.id = n.content and 
n.debuginfo_p = 1\n"
   ";"
   // ... and for E queries
-  "create view if not exists " BUILDIDS "_query_e as \n"
+  "create view if not exists " BUILDIDS "_query_e2 as \n"
   "select\n"
-  "b.hex as buildid, n.mtime, 'F' as sourcetype, f0.name as source0, 
n.mtime as mtime, null as source1\n"
+  "b.hex as buildid, 'F' as sourcetype, n.file as id0, f0.name as 
source0, n.mtime as mtime, null as id1, null as source1\n"
   "from " BUILDIDS "_buildids b, " BUILDIDS "_files_v f0, " BUILDIDS 
"_f_de n\n"
   "where b.id = n.buildid and f0.id = n.file and n.executable_p = 1\n"
   "union all select\n"
-  "b.hex as buildid, n.mtime, 'R' as sourcetype, f0.name as source0, 
n.mtime as mtime, f1.name as source1\n"
+  "b.hex as buildid, 'R

[PATCH v5 5/7] debuginfod: optimize extraction from seekable xz archives

[Omar Sandoval]

From: Omar Sandoval 

The kernel debuginfo packages on Fedora, Debian, and Ubuntu, and many of
their downstreams, are all compressed with xz in multi-threaded mode,
which allows random access.  We can use this to bypass the full archive
extraction and dramatically speed up kernel debuginfo requests (from ~50
seconds in the worst case to < 0.25 seconds).

This works because multi-threaded xz compression splits up the stream
into many independently compressed blocks.  The stream ends with an
index of blocks.  So, to seek to an offset, we find the block containing
that offset in the index and then decompress and throw away data until
we reach the offset within the block.  We can then decompress the
desired amount of data, possibly from subsequent blocks.  There's no
high-level API in liblzma to do this, but we can do it by stitching
together a few low-level APIs.

We need to pass down the file ids then look up the size, uncompressed
offset, and mtime in the _r_seekable table.  Note that this table is not
yet populated, so this commit has no functional change on its own.

Signed-off-by: Omar Sandoval 
---
 configure.ac  |   5 +
 debuginfod/Makefile.am|   2 +-
 debuginfod/debuginfod.cxx | 456 +-
 3 files changed, 457 insertions(+), 6 deletions(-)

diff --git a/configure.ac b/configure.ac
index 24e68d94..9c5f7e51 100644
--- a/configure.ac
+++ b/configure.ac
@@ -441,8 +441,13 @@ eu_ZIPLIB(bzlib,BZLIB,bz2,BZ2_bzdopen,bzip2)
 # We need this since bzip2 doesn't have a pkgconfig file.
 BZ2_LIB="$LIBS"
 AC_SUBST([BZ2_LIB])
+save_LIBS="$LIBS"
+LIBS=
 eu_ZIPLIB(lzma,LZMA,lzma,lzma_auto_decoder,[LZMA (xz)])
+lzma_LIBS="$LIBS"
+LIBS="$lzma_LIBS $save_LIBS"
 AS_IF([test "x$with_lzma" = xyes], [LIBLZMA="liblzma"], [LIBLZMA=""])
+AC_SUBST([lzma_LIBS])
 AC_SUBST([LIBLZMA])
 eu_ZIPLIB(zstd,ZSTD,zstd,ZSTD_decompress,[ZSTD (zst)])
 AS_IF([test "x$with_zstd" = xyes], [LIBZSTD="libzstd"], [LIBLZSTD=""])
diff --git a/debuginfod/Makefile.am b/debuginfod/Makefile.am
index b74e3673..e199dc0c 100644
--- a/debuginfod/Makefile.am
+++ b/debuginfod/Makefile.am
@@ -70,7 +70,7 @@ bin_PROGRAMS += debuginfod-find
 endif
 
 debuginfod_SOURCES = debuginfod.cxx
-debuginfod_LDADD = $(libdw) $(libelf) $(libeu) $(libdebuginfod) $(argp_LDADD) 
$(fts_LIBS) $(libmicrohttpd_LIBS) $(sqlite3_LIBS) $(libarchive_LIBS) 
$(rpm_LIBS) $(jsonc_LIBS) $(libcurl_LIBS) -lpthread -ldl
+debuginfod_LDADD = $(libdw) $(libelf) $(libeu) $(libdebuginfod) $(argp_LDADD) 
$(fts_LIBS) $(libmicrohttpd_LIBS) $(sqlite3_LIBS) $(libarchive_LIBS) 
$(rpm_LIBS) $(jsonc_LIBS) $(libcurl_LIBS) $(lzma_LIBS) -lpthread -ldl
 
 debuginfod_find_SOURCES = debuginfod-find.c
 debuginfod_find_LDADD = $(libdw) $(libelf) $(libeu) $(libdebuginfod) 
$(argp_LDADD) $(fts_LIBS) $(jsonc_LIBS)
diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index b3d80090..cf7f48ab 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -63,6 +63,10 @@ extern "C" {
 #undef __attribute__ /* glibc bug - rhbz 1763325 */
 #endif
 
+#ifdef USE_LZMA
+#include 
+#endif
+
 #include 
 #include 
 #include 
@@ -1961,6 +1965,385 @@ handle_buildid_f_match (bool internal_req_t,
   return r;
 }
 
+
+#ifdef USE_LZMA
+struct lzma_exception: public reportable_exception
+{
+  lzma_exception(int rc, const string& msg):
+// liblzma doesn't have a lzma_ret -> string conversion function, so just
+// report the value.
+reportable_exception(string ("lzma error: ") + msg + ": error " + 
to_string(rc)) {
+  inc_metric("error_count","lzma",to_string(rc));
+}
+};
+
+// Neither RPM nor deb files support seeking to a specific file in the package.
+// Instead, to extract a specific file, we normally need to read the archive
+// sequentially until we find the file.  This is very slow for files at the end
+// of a large package with lots of files, like kernel debuginfo.
+//
+// However, if the compression format used in the archive supports seeking, we
+// can accelerate this.  As of July 2024, xz is the only widely-used format 
that
+// supports seeking, and usually only in multi-threaded mode.  Luckily, the
+// kernel-debuginfo package in Fedora and its downstreams, and the
+// linux-image-*-dbg package in Debian and its downstreams, all happen to use
+// this.
+//
+// The xz format [1] ends with an index of independently compressed blocks in
+// the stream.  In RPM and deb files, the xz stream is the last thing in the
+// file, so we assume that the xz Stream Footer is at the end of the package
+// file and do everything relative to that.  For each file in the archive, we
+// remember the size and offset of the file data in the uncompressed xz stream,
+// then we use the index to seek to that offset when we need that file.
+//
+// 1: https://xz.tukaani.org/format/xz

Re: [PATCH v5 0/7] debuginfod: speed up extraction from kernel debuginfo packages by 200x

[Omar Sandoval]

On Wed, Jul 24, 2024 at 06:20:21PM -0400, Aaron Merey wrote:
> On Tue, Jul 23, 2024 at 6:40 PM Omar Sandoval  wrote:
> >
> > From: Omar Sandoval 
> >
> > This is v4 of my patch series optimizing debuginfod for kernel
> > debuginfo.  v1 is here [1], v2 is here [2], v3 is here [3], v4 is here
> > [4].  The only change from v4 in this version is adding --fdcache-mbs
> > and --fdcache-mintmp to the new test to fix some sporadic test failures.
> > Hopefully this version finally gets a clean test run.
> 
> Thanks Omar. I've gone ahead and merged these patches.

Thanks so much!

Omar

[RFC PATCH 1/2] libcpu: merge libcpu_{i386,x86_64,bpf} into one library

[Omar Sandoval]

From: Omar Sandoval 

In preparation for combining the libebl backend modules, combine all of
the libcpu backends into libcpu.a.

Signed-off-by: Omar Sandoval 
---
 backends/ChangeLog   |  4 
 backends/Makefile.am | 12 +++-
 libcpu/ChangeLog |  4 
 libcpu/Makefile.am   | 12 +---
 4 files changed, 16 insertions(+), 16 deletions(-)

diff --git a/backends/ChangeLog b/backends/ChangeLog
index 6c2b47a9..b40a4c8e 100644
--- a/backends/ChangeLog
+++ b/backends/ChangeLog
@@ -1,3 +1,7 @@
+2019-07-02  Omar Sandoval  
+
+   * Makefile.am: Use combined libcpu.a library.
+
 2019-04-14  Mark Wielaard  
 
* riscv_cfi.c: Fix BACKEND define.
diff --git a/backends/Makefile.am b/backends/Makefile.am
index 2126a2ec..0307da07 100644
--- a/backends/Makefile.am
+++ b/backends/Makefile.am
@@ -43,6 +43,7 @@ noinst_LIBRARIES = $(libebl_pic)
 noinst_DATA = $(libebl_pic:_pic.a=.so)
 
 
+libcpu = ../libcpu/libcpu.a
 libelf = ../libelf/libelf.so
 libdw = ../libdw/libdw.so
 libeu = ../lib/libeu.a
@@ -50,7 +51,6 @@ libeu = ../lib/libeu.a
 i386_SRCS = i386_init.c i386_symbol.c i386_corenote.c i386_cfi.c \
i386_retval.c i386_regs.c i386_auxv.c i386_syscall.c \
i386_initreg.c i386_unwind.c
-cpu_i386 = ../libcpu/libcpu_i386.a
 libebl_i386_pic_a_SOURCES = $(i386_SRCS)
 am_libebl_i386_pic_a_OBJECTS = $(i386_SRCS:.c=.os)
 
@@ -61,7 +61,6 @@ am_libebl_sh_pic_a_OBJECTS = $(sh_SRCS:.c=.os)
 x86_64_SRCS = x86_64_init.c x86_64_symbol.c x86_64_corenote.c x86_64_cfi.c \
  x86_64_retval.c x86_64_regs.c i386_auxv.c x86_64_syscall.c \
  x86_64_initreg.c x86_64_unwind.c x32_corenote.c
-cpu_x86_64 = ../libcpu/libcpu_x86_64.a
 libebl_x86_64_pic_a_SOURCES = $(x86_64_SRCS)
 am_libebl_x86_64_pic_a_OBJECTS = $(x86_64_SRCS:.c=.os)
 
@@ -127,7 +126,6 @@ am_libebl_m68k_pic_a_OBJECTS = $(m68k_SRCS:.c=.os)
 m68k_corenote_no_Wpacked_not_aligned = yes
 
 bpf_SRCS = bpf_init.c bpf_regs.c bpf_symbol.c
-cpu_bpf = ../libcpu/libcpu_bpf.a
 libebl_bpf_pic_a_SOURCES = $(bpf_SRCS)
 am_libebl_bpf_pic_a_OBJECTS = $(bpf_SRCS:.c=.os)
 
@@ -137,20 +135,16 @@ libebl_riscv_pic_a_SOURCES = $(riscv_SRCS)
 am_libebl_riscv_pic_a_OBJECTS = $(riscv_SRCS:.c=.os)
 
 
-libebl_%.so libebl_%.map: libebl_%_pic.a $(libelf) $(libdw) $(libeu)
+libebl_%.so libebl_%.map: libebl_%_pic.a $(libcpu) $(libelf) $(libdw) $(libeu)
@rm -f $(@:.so=.map)
$(AM_V_at)echo 'ELFUTILS_$(PACKAGE_VERSION) { global: $*_init; local: 
*; };' \
  > $(@:.so=.map)
$(AM_V_CCLD)$(LINK) $(dso_LDFLAGS) -o $(@:.map=.so) \
-   -Wl,--whole-archive $< $(cpu_$*) -Wl,--no-whole-archive \
+   -Wl,--whole-archive $< $(libcpu) -Wl,--no-whole-archive \
-Wl,--version-script,$(@:.so=.map),--no-undefined \
-Wl,--as-needed $(libelf) $(libdw) $(libeu)
@$(textrel_check)
 
-libebl_i386.so: $(cpu_i386)
-libebl_x86_64.so: $(cpu_x86_64)
-libebl_bpf.so: $(cpu_bpf)
-
 install: install-am install-ebl-modules
 install-ebl-modules:
$(mkinstalldirs) $(DESTDIR)$(libdir)/$(LIBEBL_SUBDIR)
diff --git a/libcpu/ChangeLog b/libcpu/ChangeLog
index adebbef8..0075a793 100644
--- a/libcpu/ChangeLog
+++ b/libcpu/ChangeLog
@@ -1,3 +1,7 @@
+2019-07-02  Omar Sandoval  
+
+   * Makefile.am: Combine backends into libcpu.a.
+
 2018-11-04  Mark Wielaard  
 
* bpf_disasm.c (bpf_disasm): Recognize BPF_JLT, BPF_JLE, BPF_JSLT
diff --git a/libcpu/Makefile.am b/libcpu/Makefile.am
index 4c8778d1..a7d9f6fd 100644
--- a/libcpu/Makefile.am
+++ b/libcpu/Makefile.am
@@ -35,20 +35,16 @@ LEXCOMPILE = $(LEX) $(LFLAGS) $(AM_LFLAGS) -P$( $@T
$(AM_V_at)mv -f $@T $@
@@ -86,6 +82,8 @@ i386_gendis_LDADD = $(libeu) -lm
 
 i386_parse.h: i386_parse.c ;
 
+bpf_disasm_CFLAGS = -Wno-format-nonliteral
+
 EXTRA_DIST = defs/i386
 
 CLEANFILES += $(foreach P,i386 x86_64,$P_defs $P.mnemonics)
-- 
2.22.0

[RFC PATCH 0/2] elfutils: don't use dlopen() for libebl modules

[Omar Sandoval]

From: Omar Sandoval 

Hi,

I'm developing an application which uses libdwfl. When I tested it in
our production environment, the application hit DWARF parsing errors for
Linux kernel modules. I tracked it down to an issue that ELF relocations
were silently not being applied because the proper libebl_CPU.so could
not be found. That appears to be a bug in relocate_sections() or
check_badreltype() in libdwfl, but that got me looking at getting rid of
the dlopen() entirely. IMO, it simplifies things nicely, and it makes it
possible to use the elfutils libraries in standalone static binaries.

I came across a previous discussion [1] where Mark suggested a hybrid
approach, where one could configure which modules to compile in and
which ones to keep separate. I tried to implement this, but it quickly
turned into automake dependency hell: the backends subdirectory needs to
depend on the libdw subdirectory in order to link libebl_CPU.so against
libdw.so, but the libdw subdirectory needs to depend on the backends
subdirectory in order to include the static backends in libdw.so.

My understanding of the benefit of separate modules is that we don't
need to link all backend modules into every elfutils binary. I did some
measurements to that end:

Dynamic backends (status quo):
44K ./libasm/libasm.so
380K./libdw/libdw.so
120K./libelf/libelf.so
72K ./src/nm
28K ./src/size
80K ./src/strip
140K./src/elflint
28K ./src/findtextrel
36K ./src/addr2line
52K ./src/elfcmp
56K ./src/objdump
32K ./src/strings
48K ./src/ar
60K ./src/unstrip
36K ./src/stack
40K ./src/elfcompress
32K ./src/ranlib
272K./src/readelf
68K backends/libebl_i386.so
24K backends/libebl_sh.so
76K backends/libebl_x86_64.so
28K backends/libebl_ia64.so
24K backends/libebl_alpha.so
36K backends/libebl_arm.so
44K backends/libebl_aarch64.so
36K backends/libebl_sparc.so
36K backends/libebl_ppc.so
36K backends/libebl_ppc64.so
32K backends/libebl_s390.so
24K backends/libebl_tilegx.so
20K backends/libebl_m68k.so
24K backends/libebl_bpf.so
32K backends/libebl_riscv.so
2.1Mtotal

Static backends (after this series):
44K ./libasm/libasm.so
668K./libdw/libdw.so
120K./libelf/libelf.so
356K./src/nm
28K ./src/size
368K./src/strip
428K./src/elflint
28K ./src/findtextrel
36K ./src/addr2line
340K./src/elfcmp
348K./src/objdump
32K ./src/strings
48K ./src/ar
60K ./src/unstrip
36K ./src/stack
40K ./src/elfcompress
32K ./src/ranlib
560K./src/readelf
3.5Mtotal

If the size increase is a deal-breaker, one easy solution may be to
export the libebl symbols from libdw so that libebl.a doesn't need to be
linked into the elfutils binaries. Considering that libebl.a is already
shipped by distributions, this doesn't increase the surface area of
exported APIs (and the warning in libebl.h should make it clear that
these APIs are not intended to be used).

Thoughts? Is the size increase palatable? Should I add a third patch to
export the libebl symbols in libdw in order to mitigate the size
increase? Or is this a dead end?

Thanks!

1: https://sourceware.org/ml/elfutils-devel/2018-q2/msg00171.html

Omar Sandoval (2):
  libcpu: merge libcpu_{i386,x86_64,bpf} into one library
  Don't use dlopen() for libebl modules

 ChangeLog|   6 +
 Makefile.am  |   4 +-
 backends/ChangeLog   |   5 +
 backends/Makefile.am |  99 ++-
 configure.ac |  12 +-
 libcpu/ChangeLog |   4 +
 libcpu/Makefile.am   |  12 +-
 libdw/ChangeLog  |   7 +
 libdw/Makefile.am|  11 +-
 libebl/ChangeLog |  11 ++
 libebl/Makefile.am   |   7 +-
 libebl/eblclosebackend.c |   4 -
 libebl/eblopenbackend.c  | 267 +++
 libebl/libeblP.h |   3 -
 src/ChangeLog|   7 +
 src/Makefile.am  |  18 +--
 tests/ChangeLog  |   5 +
 tests/Makefile.am|  34 ++---
 tests/test-subr.sh   |   4 +-
 19 files changed, 208 insertions(+), 312 deletions(-)

-- 
2.22.0

[RFC PATCH 2/2] Don't use dlopen() for libebl modules

[Omar Sandoval]

From: Omar Sandoval 

Currently, architecture-specific code for libebl exists in separate
libebl_$ARCH.so libraries which libebl loads with dlopen() at runtime.
This makes it impossible to have standalone, statically-linked binaries
which use libdwfl if they depend on any architecture-specific
functionality. Additionally, when these libraries cannot be found, the
failure modes are non-obvious. So, let's get rid of libebl_$arch.so and
move it all into libebl.a, which simplifies things considerably.

Signed-off-by: Omar Sandoval 
---
 ChangeLog|   6 +
 Makefile.am  |   4 +-
 backends/ChangeLog   |   1 +
 backends/Makefile.am |  93 ++
 configure.ac |  12 +-
 libdw/ChangeLog  |   7 +
 libdw/Makefile.am|  11 +-
 libebl/ChangeLog |  11 ++
 libebl/Makefile.am   |   7 +-
 libebl/eblclosebackend.c |   4 -
 libebl/eblopenbackend.c  | 267 +++
 libebl/libeblP.h |   3 -
 src/ChangeLog|   7 +
 src/Makefile.am  |  18 +--
 tests/ChangeLog  |   5 +
 tests/Makefile.am|  34 ++---
 tests/test-subr.sh   |   4 +-
 17 files changed, 195 insertions(+), 299 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 5c45cccf..ff12ee27 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,9 @@
+2019-07-02  Omar Sandoval  
+
+   * configure.ac: Get rid of --enable-libebl-subdir.
+   * Makefile.am: Reorder backends and libcpu before libebl in SUBDIRS to
+   satisfy build dependencies.
+
 2019-02-14  Mark Wielaard  
 
* configure.ac: Set version to 0.176.
diff --git a/Makefile.am b/Makefile.am
index 2ff444e7..51f54552 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -27,8 +27,8 @@ AM_MAKEFLAGS = --no-print-directory
 pkginclude_HEADERS = version.h
 
 # Add doc back when we have some real content.
-SUBDIRS = config m4 lib libelf libebl libdwelf libdwfl libdw libcpu libasm \
- backends src po tests
+SUBDIRS = config m4 lib libelf libcpu backends libebl libdwelf libdwfl libdw \
+ libasm src po tests
 
 EXTRA_DIST = elfutils.spec GPG-KEY NOTES CONTRIBUTING \
 COPYING COPYING-GPLV2 COPYING-LGPLV3
diff --git a/backends/ChangeLog b/backends/ChangeLog
index b40a4c8e..11f513c1 100644
--- a/backends/ChangeLog
+++ b/backends/ChangeLog
@@ -1,6 +1,7 @@
 2019-07-02  Omar Sandoval  
 
* Makefile.am: Use combined libcpu.a library.
+   Combine modules into libebl_backends.a.
 
 2019-04-14  Mark Wielaard  
 
diff --git a/backends/Makefile.am b/backends/Makefile.am
index 0307da07..733c8623 100644
--- a/backends/Makefile.am
+++ b/backends/Makefile.am
@@ -28,95 +28,58 @@
 ## the GNU Lesser General Public License along with this program.  If
 ## not, see <http://www.gnu.org/licenses/>.
 include $(top_srcdir)/config/eu.am
+AM_CFLAGS += $(fpic_CFLAGS)
 AM_CPPFLAGS += -I$(top_srcdir)/libebl -I$(top_srcdir)/libasm \
   -I$(top_srcdir)/libelf -I$(top_srcdir)/libdw
 
+noinst_LIBRARIES = libebl_backends.a
 
 modules = i386 sh x86_64 ia64 alpha arm aarch64 sparc ppc ppc64 s390 \
  tilegx m68k bpf riscv
-libebl_pic = libebl_i386_pic.a libebl_sh_pic.a libebl_x86_64_pic.a\
-libebl_ia64_pic.a libebl_alpha_pic.a libebl_arm_pic.a\
-libebl_aarch64_pic.a libebl_sparc_pic.a libebl_ppc_pic.a \
-libebl_ppc64_pic.a libebl_s390_pic.a libebl_tilegx_pic.a \
-libebl_m68k_pic.a libebl_bpf_pic.a libebl_riscv_pic.a
-noinst_LIBRARIES = $(libebl_pic)
-noinst_DATA = $(libebl_pic:_pic.a=.so)
-
-
-libcpu = ../libcpu/libcpu.a
-libelf = ../libelf/libelf.so
-libdw = ../libdw/libdw.so
-libeu = ../lib/libeu.a
 
 i386_SRCS = i386_init.c i386_symbol.c i386_corenote.c i386_cfi.c \
i386_retval.c i386_regs.c i386_auxv.c i386_syscall.c \
i386_initreg.c i386_unwind.c
-libebl_i386_pic_a_SOURCES = $(i386_SRCS)
-am_libebl_i386_pic_a_OBJECTS = $(i386_SRCS:.c=.os)
 
 sh_SRCS = sh_init.c sh_symbol.c sh_corenote.c sh_regs.c sh_retval.c
-libebl_sh_pic_a_SOURCES = $(sh_SRCS)
-am_libebl_sh_pic_a_OBJECTS = $(sh_SRCS:.c=.os)
 
 x86_64_SRCS = x86_64_init.c x86_64_symbol.c x86_64_corenote.c x86_64_cfi.c \
- x86_64_retval.c x86_64_regs.c i386_auxv.c x86_64_syscall.c \
- x86_64_initreg.c x86_64_unwind.c x32_corenote.c
-libebl_x86_64_pic_a_SOURCES = $(x86_64_SRCS)
-am_libebl_x86_64_pic_a_OBJECTS = $(x86_64_SRCS:.c=.os)
+ x86_64_retval.c x86_64_regs.c x86_64_syscall.c x86_64_initreg.c \
+ x86_64_unwind.c x32_corenote.c
+
 
 ia64_SRCS = ia64_init.c ia64_symbol.c ia64_regs.c ia64_retval.c
-libebl_ia64_pic_a_SOURCES = $(ia64_SRCS)
-am_libebl_ia64_pic_a_OBJECTS = $(ia64_SRCS:.c=.os)
 
 alpha_SRCS = alpha_init.c alpha_symbol.c alpha_retval.c alpha_regs.c \
 alpha_corenote.c alpha_auxv.c
-libebl_alpha_pic_a_SOURCES = $(alpha_SRCS)
-am_libebl_alpha_pic_a_OBJECTS = $(alpha_SRCS:.c=.os)
 
 arm_SRCS = arm_init.c arm_symbol.c arm_regs.c arm

Re: [RFC PATCH 0/2] elfutils: don't use dlopen() for libebl modules

[Omar Sandoval]

On Wed, Jul 03, 2019 at 05:33:42PM -0400, Frank Ch. Eigler wrote:
> Hi -
> 
> > [...]
> > My understanding of the benefit of separate modules is that we don't
> > need to link all backend modules into every elfutils binary. I did some
> > measurements to that end:
> > 
> > Dynamic backends (status quo):
> > 44K ./libasm/libasm.so
> > 380K./libdw/libdw.so
> > 120K./libelf/libelf.so
> > 56K ./src/objdump
> > 
> > Static backends (after this series):
> > 44K ./libasm/libasm.so
> > 668K./libdw/libdw.so
> > 120K./libelf/libelf.so
> > 348K./src/objdump
> 
> Assuming a dynamically linked objdump would be the same size as
> before, and that the libebl-* contents would only bloat libdw.so,
> and only by 300K, IMHO we should just go for it.

This is actually with dynamically linked binaries -- the size increase
is because we link libebl.a directly into all of the binaries in
addition to libdw.so:

objdump_LDADD  = $(libasm) $(libebl) $(libdw) $(libelf) $(libeu) $(argp_LDADD)

Re: [RFC PATCH 0/2] elfutils: don't use dlopen() for libebl modules

[Omar Sandoval]

On Wed, Jul 03, 2019 at 05:39:53PM -0400, Frank Ch. Eigler wrote:
> Hi - 
> 
> > This is actually with dynamically linked binaries -- the size increase
> > is because we link libebl.a directly into all of the binaries in
> > addition to libdw.so:
> > 
> > objdump_LDADD  = $(libasm) $(libebl) $(libdw) $(libelf) $(libeu) 
> > $(argp_LDADD
> 
> Is there some reason to keep doing that?
> 
> - FChE

Yes:

$ make -C src
make: Entering directory '/home/osandov/dev/elfutils/src'
  CCLD objdump
/bin/ld: objdump.o: in function `show_relocs_x':
/home/osandov/dev/elfutils/src/objdump.c:352: undefined reference to 
`ebl_reloc_type_name'
/bin/ld: objdump.o: in function `handle_elf':
/home/osandov/dev/elfutils/src/objdump.c:755: undefined reference to 
`ebl_openbackend'
/bin/ld: /home/osandov/dev/elfutils/src/objdump.c:757: undefined reference to 
`ebl_backend_name'
/bin/ld: /home/osandov/dev/elfutils/src/objdump.c:788: undefined reference to 
`ebl_closebackend'
collect2: error: ld returned 1 exit status
make: *** [Makefile:624: objdump] Error 1
make: Leaving directory '/home/osandov/dev/elfutils/src'

Some of the binaries use libebl, and although libebl is linked into libdw.so,
the libebl symbols are not exported by libdw. So, libebl is linked in
statically for the binaries.

This is why I suggested exporting those symbols from libdw.so. (I also
considered adding a libebl.so, but that would have a circular dependency with
libdw.so, so they might as well be combined).

[PATCH 1/5] libebl: remove unnecessary variable in Makefile.am

[Omar Sandoval]

From: Omar Sandoval 

gen_SOURCES is assigned to libebl_a_SOURCES and never used again. Get
rid of it.

Signed-off-by: Omar Sandoval 
---
 libebl/ChangeLog   |  4 
 libebl/Makefile.am | 39 +++
 2 files changed, 23 insertions(+), 20 deletions(-)

diff --git a/libebl/ChangeLog b/libebl/ChangeLog
index 8a7d177f..9510f9d5 100644
--- a/libebl/ChangeLog
+++ b/libebl/ChangeLog
@@ -1,3 +1,7 @@
+2019-07-05  Omar Sandoval  
+
+   * Makefile.am (gen_SOURCES): Remove.
+
 2019-05-30  Mark Wielaard  
 
* eblopenbackend.c (try_dlopen): New function extracted from
diff --git a/libebl/Makefile.am b/libebl/Makefile.am
index 737de6b0..ccc1a31a 100644
--- a/libebl/Makefile.am
+++ b/libebl/Makefile.am
@@ -37,26 +37,25 @@ lib_LIBRARIES = libebl.a
 
 pkginclude_HEADERS = libebl.h
 
-gen_SOURCES = eblopenbackend.c eblclosebackend.c \
- eblreloctypename.c eblsegmenttypename.c \
- eblsectiontypename.c eblmachineflagname.c \
- eblsymboltypename.c ebldynamictagname.c eblsectionname.c \
- eblsymbolbindingname.c eblbackendname.c eblosabiname.c \
- eblmachineflagcheck.c eblmachinesectionflagcheck.c \
- eblreloctypecheck.c eblrelocvaliduse.c eblrelocsimpletype.c \
- ebldynamictagcheck.c eblcorenotetypename.c eblobjnotetypename.c \
- eblcorenote.c eblobjnote.c ebldebugscnp.c \
- eblgotpcreloccheck.c eblcopyrelocp.c eblsectionstripp.c \
- eblelfclass.c eblelfdata.c eblelfmachine.c \
- ebl_check_special_symbol.c eblbsspltp.c eblretval.c \
- eblreginfo.c eblnonerelocp.c eblrelativerelocp.c \
- eblsysvhashentrysize.c eblauxvinfo.c eblcheckobjattr.c \
- ebl_check_special_section.c ebl_syscall_abi.c eblabicfi.c \
- eblstother.c eblinitreg.c ebldwarftoregno.c eblnormalizepc.c \
- eblunwind.c eblresolvesym.c eblcheckreloctargettype.c \
- ebl_data_marker_symbol.c
-
-libebl_a_SOURCES = $(gen_SOURCES)
+libebl_a_SOURCES = eblopenbackend.c eblclosebackend.c eblreloctypename.c \
+  eblsegmenttypename.c eblsectiontypename.c \
+  eblmachineflagname.c eblsymboltypename.c \
+  ebldynamictagname.c eblsectionname.c \
+  eblsymbolbindingname.c eblbackendname.c eblosabiname.c \
+  eblmachineflagcheck.c eblmachinesectionflagcheck.c \
+  eblreloctypecheck.c eblrelocvaliduse.c \
+  eblrelocsimpletype.c ebldynamictagcheck.c \
+  eblcorenotetypename.c eblobjnotetypename.c eblcorenote.c \
+  eblobjnote.c ebldebugscnp.c eblgotpcreloccheck.c \
+  eblcopyrelocp.c eblsectionstripp.c eblelfclass.c \
+  eblelfdata.c eblelfmachine.c ebl_check_special_symbol.c \
+  eblbsspltp.c eblretval.c eblreginfo.c eblnonerelocp.c \
+  eblrelativerelocp.c eblsysvhashentrysize.c eblauxvinfo.c \
+  eblcheckobjattr.c ebl_check_special_section.c \
+  ebl_syscall_abi.c eblabicfi.c eblstother.c eblinitreg.c \
+  ebldwarftoregno.c eblnormalizepc.c eblunwind.c \
+  eblresolvesym.c eblcheckreloctargettype.c \
+  ebl_data_marker_symbol.c
 
 noinst_HEADERS = libeblP.h ebl-hooks.h
 
-- 
2.22.0

[PATCH 0/5] elfutils: don't use dlopen() for libebl modules

[Omar Sandoval]

From: Omar Sandoval 

Hello,

This series is a followup to my RFC [1]. Compared to the RFC patches,
this series adds patch 1 as a trivial cleanup, patch 3 to fix an
oversight in the RFC w.r.t. PIC object files sneaking into libdw.a, and
patch 5 to implement my suggestion for reducing the size of the elfutils
binaries by exporting libebl symbols from libdw.so. After this series,
libebl no longer uses dlopen() to find backends, and the total size of
the package is actually smaller than before (2.1 MB):

44K ./libasm/libasm.so
672K./libdw/libdw.so
120K./libelf/libelf.so
52K ./src/nm
28K ./src/size
60K ./src/strip
116K./src/elflint
28K ./src/findtextrel
36K ./src/addr2line
32K ./src/elfcmp
40K ./src/objdump
32K ./src/strings
48K ./src/ar
60K ./src/unstrip
36K ./src/stack
40K ./src/elfcompress
32K ./src/ranlib
236K./src/readelf
1.7Mtotal

If there are no issues, I'd love to see this merged, as it greatly
simplifies using libdw in environments where dlopen() is not supported.

Thanks!

1: https://sourceware.org/ml/elfutils-devel/2019-q3/msg00010.html

Omar Sandoval (5):
  libebl: remove unnecessary variable in Makefile.am
  libcpu: merge libcpu_{i386,x86_64,bpf} into one library
  Add PIC and non-PIC variants of libcpu and libebl
  Don't use dlopen() for libebl modules
  libdw: export libebl symbols

 ChangeLog|   6 +
 Makefile.am  |   4 +-
 backends/ChangeLog   |   6 +
 backends/Makefile.am | 104 +++
 configure.ac |  12 +-
 libcpu/ChangeLog |   6 +
 libcpu/Makefile.am   |  24 ++--
 libdw/ChangeLog  |   9 ++
 libdw/Makefile.am|  21 ++-
 libdw/libdw.map  |  60 -
 libebl/ChangeLog |  12 ++
 libebl/Makefile.am   |  46 ---
 libebl/eblclosebackend.c |   4 -
 libebl/eblopenbackend.c  | 267 +++
 libebl/libebl.h  |   8 +-
 libebl/libeblP.h |   3 -
 src/ChangeLog|  17 +++
 src/Makefile.am  |  21 ++-
 tests/ChangeLog  |   5 +
 tests/Makefile.am|  36 +++---
 tests/test-subr.sh   |   4 +-
 21 files changed, 331 insertions(+), 344 deletions(-)

-- 
2.22.0

[PATCH 2/5] libcpu: merge libcpu_{i386,x86_64,bpf} into one library

[Omar Sandoval]

From: Omar Sandoval 

In preparation for combining the libebl backend modules, combine all of
the libcpu backends into libcpu.a.

Signed-off-by: Omar Sandoval 
---
 backends/ChangeLog   |  4 
 backends/Makefile.am | 12 +++-
 libcpu/ChangeLog |  4 
 libcpu/Makefile.am   | 12 +---
 4 files changed, 16 insertions(+), 16 deletions(-)

diff --git a/backends/ChangeLog b/backends/ChangeLog
index 6c2b47a9..219e0702 100644
--- a/backends/ChangeLog
+++ b/backends/ChangeLog
@@ -1,3 +1,7 @@
+2019-07-05  Omar Sandoval  
+
+   * Makefile.am: Replace libcpu_{i386,x86_64,bpf}.a with libcpu.a.
+
 2019-04-14  Mark Wielaard  
 
* riscv_cfi.c: Fix BACKEND define.
diff --git a/backends/Makefile.am b/backends/Makefile.am
index 2126a2ec..0307da07 100644
--- a/backends/Makefile.am
+++ b/backends/Makefile.am
@@ -43,6 +43,7 @@ noinst_LIBRARIES = $(libebl_pic)
 noinst_DATA = $(libebl_pic:_pic.a=.so)
 
 
+libcpu = ../libcpu/libcpu.a
 libelf = ../libelf/libelf.so
 libdw = ../libdw/libdw.so
 libeu = ../lib/libeu.a
@@ -50,7 +51,6 @@ libeu = ../lib/libeu.a
 i386_SRCS = i386_init.c i386_symbol.c i386_corenote.c i386_cfi.c \
i386_retval.c i386_regs.c i386_auxv.c i386_syscall.c \
i386_initreg.c i386_unwind.c
-cpu_i386 = ../libcpu/libcpu_i386.a
 libebl_i386_pic_a_SOURCES = $(i386_SRCS)
 am_libebl_i386_pic_a_OBJECTS = $(i386_SRCS:.c=.os)
 
@@ -61,7 +61,6 @@ am_libebl_sh_pic_a_OBJECTS = $(sh_SRCS:.c=.os)
 x86_64_SRCS = x86_64_init.c x86_64_symbol.c x86_64_corenote.c x86_64_cfi.c \
  x86_64_retval.c x86_64_regs.c i386_auxv.c x86_64_syscall.c \
  x86_64_initreg.c x86_64_unwind.c x32_corenote.c
-cpu_x86_64 = ../libcpu/libcpu_x86_64.a
 libebl_x86_64_pic_a_SOURCES = $(x86_64_SRCS)
 am_libebl_x86_64_pic_a_OBJECTS = $(x86_64_SRCS:.c=.os)
 
@@ -127,7 +126,6 @@ am_libebl_m68k_pic_a_OBJECTS = $(m68k_SRCS:.c=.os)
 m68k_corenote_no_Wpacked_not_aligned = yes
 
 bpf_SRCS = bpf_init.c bpf_regs.c bpf_symbol.c
-cpu_bpf = ../libcpu/libcpu_bpf.a
 libebl_bpf_pic_a_SOURCES = $(bpf_SRCS)
 am_libebl_bpf_pic_a_OBJECTS = $(bpf_SRCS:.c=.os)
 
@@ -137,20 +135,16 @@ libebl_riscv_pic_a_SOURCES = $(riscv_SRCS)
 am_libebl_riscv_pic_a_OBJECTS = $(riscv_SRCS:.c=.os)
 
 
-libebl_%.so libebl_%.map: libebl_%_pic.a $(libelf) $(libdw) $(libeu)
+libebl_%.so libebl_%.map: libebl_%_pic.a $(libcpu) $(libelf) $(libdw) $(libeu)
@rm -f $(@:.so=.map)
$(AM_V_at)echo 'ELFUTILS_$(PACKAGE_VERSION) { global: $*_init; local: 
*; };' \
  > $(@:.so=.map)
$(AM_V_CCLD)$(LINK) $(dso_LDFLAGS) -o $(@:.map=.so) \
-   -Wl,--whole-archive $< $(cpu_$*) -Wl,--no-whole-archive \
+   -Wl,--whole-archive $< $(libcpu) -Wl,--no-whole-archive \
-Wl,--version-script,$(@:.so=.map),--no-undefined \
-Wl,--as-needed $(libelf) $(libdw) $(libeu)
@$(textrel_check)
 
-libebl_i386.so: $(cpu_i386)
-libebl_x86_64.so: $(cpu_x86_64)
-libebl_bpf.so: $(cpu_bpf)
-
 install: install-am install-ebl-modules
 install-ebl-modules:
$(mkinstalldirs) $(DESTDIR)$(libdir)/$(LIBEBL_SUBDIR)
diff --git a/libcpu/ChangeLog b/libcpu/ChangeLog
index adebbef8..c0ea72ec 100644
--- a/libcpu/ChangeLog
+++ b/libcpu/ChangeLog
@@ -1,3 +1,7 @@
+2019-07-05  Omar Sandoval  
+
+   * Makefile.am: Combine libcpu_{i386,x86_64,bpf}.a into libcpu.a.
+
 2018-11-04  Mark Wielaard  
 
* bpf_disasm.c (bpf_disasm): Recognize BPF_JLT, BPF_JLE, BPF_JSLT
diff --git a/libcpu/Makefile.am b/libcpu/Makefile.am
index 4c8778d1..a7d9f6fd 100644
--- a/libcpu/Makefile.am
+++ b/libcpu/Makefile.am
@@ -35,20 +35,16 @@ LEXCOMPILE = $(LEX) $(LFLAGS) $(AM_LFLAGS) -P$( $@T
$(AM_V_at)mv -f $@T $@
@@ -86,6 +82,8 @@ i386_gendis_LDADD = $(libeu) -lm
 
 i386_parse.h: i386_parse.c ;
 
+bpf_disasm_CFLAGS = -Wno-format-nonliteral
+
 EXTRA_DIST = defs/i386
 
 CLEANFILES += $(foreach P,i386 x86_64,$P_defs $P.mnemonics)
-- 
2.22.0

[PATCH 3/5] Add PIC and non-PIC variants of libcpu and libebl

[Omar Sandoval]

From: Omar Sandoval 

Currently, libcpu and libebl are always compiled as PIC. An upcoming
change will add the objects from libcpu.a and libebl.a to libdw.a, which
should not be PIC unless configured that way. So, make libcpu.a and
libebl.a non-PIC and add libcpu_pic.a and libebl_pic.a.

Signed-off-by: Omar Sandoval 
---
 backends/ChangeLog   |  1 +
 backends/Makefile.am |  2 +-
 libcpu/ChangeLog |  2 ++
 libcpu/Makefile.am   | 18 +-
 libdw/ChangeLog  |  4 
 libdw/Makefile.am|  2 +-
 libebl/ChangeLog |  4 +++-
 libebl/Makefile.am   |  8 +++-
 8 files changed, 32 insertions(+), 9 deletions(-)

diff --git a/backends/ChangeLog b/backends/ChangeLog
index 219e0702..d631417f 100644
--- a/backends/ChangeLog
+++ b/backends/ChangeLog
@@ -1,6 +1,7 @@
 2019-07-05  Omar Sandoval  
 
* Makefile.am: Replace libcpu_{i386,x86_64,bpf}.a with libcpu.a.
+   Replace libcpu.a with libcpu_pic.a.
 
 2019-04-14  Mark Wielaard  
 
diff --git a/backends/Makefile.am b/backends/Makefile.am
index 0307da07..d1275206 100644
--- a/backends/Makefile.am
+++ b/backends/Makefile.am
@@ -43,7 +43,7 @@ noinst_LIBRARIES = $(libebl_pic)
 noinst_DATA = $(libebl_pic:_pic.a=.so)
 
 
-libcpu = ../libcpu/libcpu.a
+libcpu = ../libcpu/libcpu_pic.a
 libelf = ../libelf/libelf.so
 libdw = ../libdw/libdw.so
 libeu = ../lib/libeu.a
diff --git a/libcpu/ChangeLog b/libcpu/ChangeLog
index c0ea72ec..883896a2 100644
--- a/libcpu/ChangeLog
+++ b/libcpu/ChangeLog
@@ -1,6 +1,8 @@
 2019-07-05  Omar Sandoval  
 
* Makefile.am: Combine libcpu_{i386,x86_64,bpf}.a into libcpu.a.
+   Make libcpu.a non-PIC by default.
+   Add libcpu_pic.a.
 
 2018-11-04  Mark Wielaard  
 
diff --git a/libcpu/Makefile.am b/libcpu/Makefile.am
index a7d9f6fd..88717361 100644
--- a/libcpu/Makefile.am
+++ b/libcpu/Makefile.am
@@ -30,15 +30,22 @@
 include $(top_srcdir)/config/eu.am
 AM_CPPFLAGS += -I$(srcdir)/../libelf -I$(srcdir)/../libebl \
-I$(srcdir)/../libdw -I$(srcdir)/../libasm
-AM_CFLAGS += $(fpic_CFLAGS) -fdollars-in-identifiers
+if BUILD_STATIC
+AM_CFLAGS += $(fpic_CFLAGS)
+endif
+AM_CFLAGS += -fdollars-in-identifiers
 LEXCOMPILE = $(LEX) $(LFLAGS) $(AM_LFLAGS) -P$(
+
+   * Makefile.am (libdw_so_LIBS): Replace libebl.a with libebl_pic.a.
+
 2019-05-16  Mark Wielaard  
 
* dwarf.h: Add DW_AT_GNU_numerator, DW_AT_GNU_denominator and
diff --git a/libdw/Makefile.am b/libdw/Makefile.am
index 7a3d5322..7dc4cec0 100644
--- a/libdw/Makefile.am
+++ b/libdw/Makefile.am
@@ -106,7 +106,7 @@ libdw_pic_a_SOURCES =
 am_libdw_pic_a_OBJECTS = $(libdw_a_SOURCES:.c=.os)
 
 libdw_so_LIBS = libdw_pic.a ../libdwelf/libdwelf_pic.a \
- ../libdwfl/libdwfl_pic.a ../libebl/libebl.a
+ ../libdwfl/libdwfl_pic.a ../libebl/libebl_pic.a
 libdw_so_DEPS = ../lib/libeu.a ../libelf/libelf.so
 libdw_so_LDLIBS = $(libdw_so_DEPS) -ldl -lz $(argp_LDADD) $(zip_LIBS)
 libdw_so_SOURCES =
diff --git a/libebl/ChangeLog b/libebl/ChangeLog
index 9510f9d5..6fd83471 100644
--- a/libebl/ChangeLog
+++ b/libebl/ChangeLog
@@ -1,6 +1,8 @@
 2019-07-05  Omar Sandoval  
 
-   * Makefile.am (gen_SOURCES): Remove.
+   * Makefile.am: Make libebl.a non-PIC by default.
+   Add libebl_pic.a.
+   (gen_SOURCES): Remove.
 
 2019-05-30  Mark Wielaard  
 
diff --git a/libebl/Makefile.am b/libebl/Makefile.am
index ccc1a31a..b85ead01 100644
--- a/libebl/Makefile.am
+++ b/libebl/Makefile.am
@@ -28,12 +28,15 @@
 ## not, see <http://www.gnu.org/licenses/>.
 ##
 include $(top_srcdir)/config/eu.am
+if BUILD_STATIC
 AM_CFLAGS += $(fpic_CFLAGS)
+endif
 AM_CPPFLAGS += -I$(srcdir)/../libelf -I$(srcdir)/../libdw -I$(srcdir)/../libasm
 VERSION = 1
 LIBEBL_SUBDIR = @LIBEBL_SUBDIR@
 
 lib_LIBRARIES = libebl.a
+noinst_LIBRARIES = libebl_pic.a
 
 pkginclude_HEADERS = libebl.h
 
@@ -57,6 +60,9 @@ libebl_a_SOURCES = eblopenbackend.c eblclosebackend.c 
eblreloctypename.c \
   eblresolvesym.c eblcheckreloctargettype.c \
   ebl_data_marker_symbol.c
 
+libebl_pic_a_SOURCES =
+am_libebl_pic_a_OBJECTS = $(libebl_a_SOURCES:.c=.os)
+
 noinst_HEADERS = libeblP.h ebl-hooks.h
 
-CLEANFILES += $(am_libebl_pic_a_OBJECTS)
+MOSTLYCLEANFILES = $(am_libebl_pic_a_OBJECTS)
-- 
2.22.0

[PATCH 4/5] Don't use dlopen() for libebl modules

[Omar Sandoval]

From: Omar Sandoval 

Currently, architecture-specific code for libebl exists in separate
libebl_$ARCH.so libraries which libebl loads with dlopen() at runtime.
This makes it impossible to have standalone, statically-linked binaries
which use libdwfl if they depend on any architecture-specific
functionality. Additionally, when these libraries cannot be found, the
failure modes are non-obvious. So, let's get rid of libebl_$arch.so and
move it all into libdw.so/libdw.a, which simplifies things considerably.

Signed-off-by: Omar Sandoval 
---
 ChangeLog|   6 +
 Makefile.am  |   4 +-
 backends/ChangeLog   |   1 +
 backends/Makefile.am |  98 +++---
 configure.ac |  12 +-
 libdw/ChangeLog  |   4 +
 libdw/Makefile.am|  21 ++-
 libebl/ChangeLog |   6 +
 libebl/Makefile.am   |   1 -
 libebl/eblclosebackend.c |   4 -
 libebl/eblopenbackend.c  | 267 +++
 libebl/libebl.h  |   8 +-
 libebl/libeblP.h |   3 -
 src/ChangeLog|   7 +
 src/Makefile.am  |  20 +--
 tests/ChangeLog  |   5 +
 tests/Makefile.am|  36 +++---
 tests/test-subr.sh   |   4 +-
 18 files changed, 200 insertions(+), 307 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 5c45cccf..49e7c858 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,9 @@
+2017-07-05  Omar Sandoval  
+
+   * configure.ac: Get rid of --enable-libebl-subdir.
+   * Makefile.am (SUBDIRS): Reorder backends and libcpu before libebl to
+   satisfy build dependencies.
+
 2019-02-14  Mark Wielaard  
 
* configure.ac: Set version to 0.176.
diff --git a/Makefile.am b/Makefile.am
index 2ff444e7..51f54552 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -27,8 +27,8 @@ AM_MAKEFLAGS = --no-print-directory
 pkginclude_HEADERS = version.h
 
 # Add doc back when we have some real content.
-SUBDIRS = config m4 lib libelf libebl libdwelf libdwfl libdw libcpu libasm \
- backends src po tests
+SUBDIRS = config m4 lib libelf libcpu backends libebl libdwelf libdwfl libdw \
+ libasm src po tests
 
 EXTRA_DIST = elfutils.spec GPG-KEY NOTES CONTRIBUTING \
 COPYING COPYING-GPLV2 COPYING-LGPLV3
diff --git a/backends/ChangeLog b/backends/ChangeLog
index d631417f..f104e8ec 100644
--- a/backends/ChangeLog
+++ b/backends/ChangeLog
@@ -2,6 +2,7 @@
 
* Makefile.am: Replace libcpu_{i386,x86_64,bpf}.a with libcpu.a.
Replace libcpu.a with libcpu_pic.a.
+   Combine libebl_CPU.so modules into libebl_backends{,_pic}.a.
 
 2019-04-14  Mark Wielaard  
 
diff --git a/backends/Makefile.am b/backends/Makefile.am
index d1275206..f0ecc405 100644
--- a/backends/Makefile.am
+++ b/backends/Makefile.am
@@ -28,95 +28,60 @@
 ## the GNU Lesser General Public License along with this program.  If
 ## not, see <http://www.gnu.org/licenses/>.
 include $(top_srcdir)/config/eu.am
+if BUILD_STATIC
+AM_CFLAGS += $(fpic_CFLAGS)
+endif
 AM_CPPFLAGS += -I$(top_srcdir)/libebl -I$(top_srcdir)/libasm \
   -I$(top_srcdir)/libelf -I$(top_srcdir)/libdw
 
+noinst_LIBRARIES = libebl_backends.a libebl_backends_pic.a
 
 modules = i386 sh x86_64 ia64 alpha arm aarch64 sparc ppc ppc64 s390 \
  tilegx m68k bpf riscv
-libebl_pic = libebl_i386_pic.a libebl_sh_pic.a libebl_x86_64_pic.a\
-libebl_ia64_pic.a libebl_alpha_pic.a libebl_arm_pic.a\
-libebl_aarch64_pic.a libebl_sparc_pic.a libebl_ppc_pic.a \
-libebl_ppc64_pic.a libebl_s390_pic.a libebl_tilegx_pic.a \
-libebl_m68k_pic.a libebl_bpf_pic.a libebl_riscv_pic.a
-noinst_LIBRARIES = $(libebl_pic)
-noinst_DATA = $(libebl_pic:_pic.a=.so)
-
-
-libcpu = ../libcpu/libcpu_pic.a
-libelf = ../libelf/libelf.so
-libdw = ../libdw/libdw.so
-libeu = ../lib/libeu.a
 
 i386_SRCS = i386_init.c i386_symbol.c i386_corenote.c i386_cfi.c \
i386_retval.c i386_regs.c i386_auxv.c i386_syscall.c \
i386_initreg.c i386_unwind.c
-libebl_i386_pic_a_SOURCES = $(i386_SRCS)
-am_libebl_i386_pic_a_OBJECTS = $(i386_SRCS:.c=.os)
 
 sh_SRCS = sh_init.c sh_symbol.c sh_corenote.c sh_regs.c sh_retval.c
-libebl_sh_pic_a_SOURCES = $(sh_SRCS)
-am_libebl_sh_pic_a_OBJECTS = $(sh_SRCS:.c=.os)
 
 x86_64_SRCS = x86_64_init.c x86_64_symbol.c x86_64_corenote.c x86_64_cfi.c \
- x86_64_retval.c x86_64_regs.c i386_auxv.c x86_64_syscall.c \
- x86_64_initreg.c x86_64_unwind.c x32_corenote.c
-libebl_x86_64_pic_a_SOURCES = $(x86_64_SRCS)
-am_libebl_x86_64_pic_a_OBJECTS = $(x86_64_SRCS:.c=.os)
+ x86_64_retval.c x86_64_regs.c x86_64_syscall.c x86_64_initreg.c \
+ x86_64_unwind.c x32_corenote.c
+
 
 ia64_SRCS = ia64_init.c ia64_symbol.c ia64_regs.c ia64_retval.c
-libebl_ia64_pic_a_SOURCES = $(ia64_SRCS)
-am_libebl_ia64_pic_a_OBJECTS = $(ia64_SRCS:.c=.os)
 
 alpha_SRCS = alpha_init.c alpha_symbol.c alpha_retval.c alpha_regs.c \
 alpha_corenote.c a

[PATCH 5/5] libdw: export libebl symbols

[Omar Sandoval]

From: Omar Sandoval 

The main downside of the previous change to build in all libebl backend
modules statically is that the total installed size of elfutils
increased (from 2.1 MB to 3.5 MB in my case). This is because we have to
statically link libebl and its backends into every binary. Instead,
since libebl is already linked into libdw.so, we can simply export the
libebl symbols in libdw.so for the binaries to use. This shrinks the
total size to 1.7 MB, which is smaller than where we started.

This doesn't change the status of libebl: it is still considered
internal and the API/ABI are still subject to change.

Signed-off-by: Omar Sandoval 
---
 libdw/ChangeLog |  1 +
 libdw/libdw.map | 60 -
 src/ChangeLog   | 10 +
 src/Makefile.am | 19 
 4 files changed, 79 insertions(+), 11 deletions(-)

diff --git a/libdw/ChangeLog b/libdw/ChangeLog
index 4e088688..84e03699 100644
--- a/libdw/ChangeLog
+++ b/libdw/ChangeLog
@@ -5,6 +5,7 @@
(libdw_so_LDLIBS): Remove -ldl.
(libdw.so): Remove -rpath.
(libdw_a_LIBADD): Add libebl, libebl_backends, and libcpu objects.
+   * libdw.map (ELFUTILS_0.177): New section. Add libebl symbols.
 
 2019-05-16  Mark Wielaard  
 
diff --git a/libdw/libdw.map b/libdw/libdw.map
index 55482d58..c9f69d32 100644
--- a/libdw/libdw.map
+++ b/libdw/libdw.map
@@ -360,4 +360,62 @@ ELFUTILS_0.173 {
 ELFUTILS_0.175 {
   global:
 dwelf_elf_begin;
-} ELFUTILS_0.173;
\ No newline at end of file
+} ELFUTILS_0.173;
+
+ELFUITLS_0.177 {
+  global:
+ebl_openbackend;
+ebl_openbackend_machine;
+ebl_openbackend_emulation;
+ebl_closebackend;
+ebl_get_elfmachine;
+ebl_get_elfclass;
+ebl_get_elfdata;
+ebl_backend_name;
+ebl_reloc_type_name;
+ebl_reloc_type_check;
+ebl_reloc_valid_use;
+ebl_reloc_simple_type;
+ebl_gotpc_reloc_check;
+ebl_segment_type_name;
+ebl_section_type_name;
+ebl_section_name;
+ebl_machine_flag_name;
+ebl_machine_flag_check;
+ebl_machine_section_flag_check;
+ebl_check_special_section;
+ebl_symbol_type_name;
+ebl_symbol_binding_name;
+ebl_dynamic_tag_name;
+ebl_dynamic_tag_check;
+ebl_check_special_symbol;
+ebl_data_marker_symbol;
+ebl_check_st_other_bits;
+ebl_osabi_name;
+ebl_core_note_type_name;
+ebl_object_note_type_name;
+ebl_object_note;
+ebl_check_object_attribute;
+ebl_check_reloc_target_type;
+ebl_debugscn_p;
+ebl_copy_reloc_p;
+ebl_none_reloc_p;
+ebl_relative_reloc_p;
+ebl_section_strip_p;
+ebl_bss_plt_p;
+ebl_sysvhash_entrysize;
+ebl_return_value_location;
+ebl_register_info;
+ebl_syscall_abi;
+ebl_abi_cfi;
+ebl_core_note;
+ebl_auxv_info;
+ebl_set_initial_registers_tid;
+ebl_frame_nregs;
+ebl_ra_offset;
+ebl_func_addr_mask;
+ebl_dwarf_to_regno;
+ebl_normalize_pc;
+ebl_unwind;
+ebl_resolve_sym_value;
+} ELFUTILS_0.175;
diff --git a/src/ChangeLog b/src/ChangeLog
index a8b1e267..2ce39db0 100644
--- a/src/ChangeLog
+++ b/src/ChangeLog
@@ -1,9 +1,19 @@
 2019-07-05  Omar Sandoval  
 
* Makefile.am: Remove -ldl.
+   Remove libebl definition.
+   (readelf_LDADD): Remove $(libebl).
+   (nm_LDADD): Remove $(libebl).
+   (strip_LDADD): Remove $(libebl).
(elflint_LDADD): Add $(libdw).
+   Remove $(libebl).
(elfcmp_LDADD): Add $(libdw).
+   Remove $(libebl).
(objdump_LDADD): Add $(libdw).
+   Remove $(libebl).
+   (unstrip_LDADD): Remove $(libebl).
+   (stack_LDADD): Remove $(libebl).
+   (elfcompress_LDADD): Remove $(libebl).
 
 2019-06-25  Mark Wielaard  
 
diff --git a/src/Makefile.am b/src/Makefile.am
index 42970da3..d219e8fe 100644
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -47,7 +47,6 @@ libasm = ../libasm/libasm.so
 libdw = ../libdw/libdw.so
 libelf = ../libelf/libelf.so
 endif
-libebl = ../libebl/libebl.a ../backends/libebl_backends.a ../libcpu/libcpu.a
 libeu = ../lib/libeu.a
 
 if DEMANGLE
@@ -67,22 +66,22 @@ ranlib_no_Wstack_usage = yes
 ar_no_Wstack_usage = yes
 unstrip_no_Wstack_usage = yes
 
-readelf_LDADD = $(libdw) $(libebl) $(libelf) $(libeu) $(argp_LDADD)
-nm_LDADD = $(libdw) $(libebl) $(libelf) $(libeu) $(argp_LDADD) \
+readelf_LDADD = $(libdw) $(libelf) $(libeu) $(argp_LDADD)
+nm_LDADD = $(libdw) $(libelf) $(libeu) $(argp_LDADD) \
   $(demanglelib)
 size_LDADD = $(libelf) $(libeu) $(argp_LDADD)
-strip_LDADD = $(libebl) $(libelf) $(libdw) $(libeu) $(argp_LDADD)
-elflint_LDADD  = $(libebl) $(libdw) $(libelf) $(libeu) $(argp_LDADD)
+strip_LDADD = $(libelf) $(libdw) $(libeu) $(argp_LDADD)
+elflint_LDADD  = $(libdw) $(libelf) $(libeu) $(argp_LDADD)
 findtextrel_LDADD = $(libdw) $(libelf) $(libeu) $(argp_LDADD)
 addr2line_LDADD = $(libdw) $(libelf) $(libeu) $(argp_LDADD) $(demanglelib)
-elfcmp_LDADD = $(libebl) $(libdw) $(libelf) $(libeu) $(argp_LDADD)
-objdump_LDADD  = $(l

Re: [RFC PATCH 0/2] elfutils: don't use dlopen() for libebl modules

[Omar Sandoval]

On Mon, Jul 08, 2019 at 10:48:52PM +0200, Mark Wielaard wrote:
> On Wed, 2019-07-03 at 20:56 -0400, Frank Ch. Eigler wrote:
> > > Some of the binaries use libebl, and although libebl is linked into 
> > > libdw.so,
> > > the libebl symbols are not exported by libdw. So, libebl is linked in
> > > statically for the binaries.
> > > 
> > > This is why I suggested exporting those symbols from libdw.so. (I also
> > > considered adding a libebl.so, but that would have a circular dependency 
> > > with
> > > libdw.so, so they might as well be combined).
> > 
> > Ah, thanks for the (re-)explanation.  IMHO, if the API is useful
> > enough to be used by mainline elfutils binaries, it is useful enough
> > to be used by others.  So a +1 from me for solib-exporting them.

Hi, Mark,

> The API is kind of useful enough to be used by the elfutils binaries
> directly (or they would use it). But that doesn't make them good,
> supportable API that we should expose through libdw.so. That would mean
> we promise they are fixed and supported essentially forever. It means,
> they are quick hacks that might work for now for internal use only.

This makes sense. One thing I noted in the patch to export the libebl
symbols [1] is that exporting them wouldn't necessarily mean supporting
them as an official API. However, I can see why you'd be concerned with
developers using them anyways.

My patch submission last Friday [2] still works without the patch to
export the libebl symbols, the only drawback being the size increase, of
course.

> It might still be a good idea to expose/move some into libdw.so. But we
> should carefully review them one by one. This was done for example for
> the ELF/DWARF string table functions (Dwelf_Strtab and Dwelf_Strent).
> Or see my recent patch that replaces ebl->name with with a new libdw.so
> function dwelf_elf_e_machine_string (which works for any ELF e_machine
> value, not just for those which have a backend implementation):
> https://sourceware.org/ml/elfutils-devel/2019-q2/msg00130.html

I imagine that little by little, most of the libebl functionality could
be converted in this way, and that's how we could chip away at the size
increase of the elfutils binaries.

Do you have any objections to patches 1-4 of my submission [2]?

Thanks for the timely response!

1: https://sourceware.org/ml/elfutils-devel/2019-q3/msg00022.html
2: https://sourceware.org/ml/elfutils-devel/2019-q3/msg00020.html

> Cheers,
> 
> Mark

Re: [RFC PATCH 0/2] elfutils: don't use dlopen() for libebl modules

[Omar Sandoval]

On Tue, Jul 09, 2019 at 09:14:03PM +0200, Mark Wielaard wrote:
> Hi Omar,
> 
> On Mon, 2019-07-08 at 14:02 -0700, Omar Sandoval wrote:
> > I imagine that little by little, most of the libebl functionality could
> > be converted in this way, and that's how we could chip away at the size
> > increase of the elfutils binaries.
> > 
> > Do you have any objections to patches 1-4 of my submission [2]?
> > 
> > Thanks for the timely response!
> > 
> > 1: https://sourceware.org/ml/elfutils-devel/2019-q3/msg00022.html
> > 2: https://sourceware.org/ml/elfutils-devel/2019-q3/msg00020.html
> 
> I am still pondering this a little. But I do think (patches 1-4) is the
> (first) step forward. But I think I would like to do a 0.177 release
> with the last 4 months of fixes first. We normally do a release every
> 3-4 months, so it is about time, and it means your feature wouldn't
> have to wait that long. Although it would be a couple of months before
> it sees a release. But I think it would be good to have some time to
> make sure the idea works out in practice.
> 
> So I would propose a release end of this week/start of next week. But
> without the libebl build rewrite.
> 
> For the release there are then still 3 things pending:
> 
> - The eu-elfclassify tool
>   https://sourceware.org/ml/elfutils-devel/2019-q2/msg00018.html
>   I had hoped on some feedback from the rpm hackers, since they are
>   one if the intended users. And there are some (small) missing
>   features and tests. Lets see where we are end if the week to see
>   whether we can include that, or also postpone it to the next release.
> - The C-SKY backend:
>   https://sourceware.org/ml/elfutils-devel/2019-q2/msg7.html
>   This is really just blocked on me not making enough time for a
>   final review. It looks good, but I am confused about one aspect
>   of the DWARF register numbering issue.
> - The dwelf_elf_e_machine_string patch:
>   https://sourceware.org/ml/elfutils-devel/2019-q2/msg00130.html
>   I didn't see any objections, so I think this is good to go.
> 
> Then after the release, somewhere next week, we'll apply your patches
> first and can then deal with any fallout and followups. I am thinking
> of moving some of the functionality into libdw proper (as cleaned up,
> exported api) to reduce the size increase a little. And add a mechanism
> for only building some of the backends (or maybe just drop some old
> ones that nobody uses anyway).

This works for me, thanks! I'll keep an eye out for any followups, and
I'm happy to help clean things up on the ebl side.

[PATCH] libdwfl: Fix fd leak/closing wrong fd after dwfl_core_file_report()

[Omar Sandoval]

From: Omar Sandoval 

dwfl_segment_report_module() (used only by dwfl_core_file_report())
opens a file descriptor and/or an Elf handle, reports a module, and
assigns mod->main.elf. However, it doesn't assign mod->main.fd, so it is
left as 0. This causes two problems:

1. We leak the file descriptor for the module.
2. When we free the module, we close file descriptor 0 (stdin).

Fix it by assigning mod->main.fd.

Signed-off-by: Omar Sandoval 
---
 libdwfl/ChangeLog| 5 +
 libdwfl/dwfl_segment_report_module.c | 1 +
 2 files changed, 6 insertions(+)

diff --git a/libdwfl/ChangeLog b/libdwfl/ChangeLog
index 7c9a018b..8cbe90c9 100644
--- a/libdwfl/ChangeLog
+++ b/libdwfl/ChangeLog
@@ -1,3 +1,8 @@
+2019-08-05  Omar Sandoval  
+
+   * dwfl_segment_report_module.c (dwfl_segment_report_module): Assign
+   mod->main.fd.
+
 2019-04-28  Mark Wielaard  
 
* frame_unwind.c (expr_eval): Make sure we left shift a unsigned
diff --git a/libdwfl/dwfl_segment_report_module.c 
b/libdwfl/dwfl_segment_report_module.c
index 76ba1506..430e13d5 100644
--- a/libdwfl/dwfl_segment_report_module.c
+++ b/libdwfl/dwfl_segment_report_module.c
@@ -967,6 +967,7 @@ dwfl_segment_report_module (Dwfl *dwfl, int ndx, const char 
*name,
 {
   /* Install the file in the module.  */
   mod->main.elf = elf;
+  mod->main.fd = fd;
   elf = NULL;
   fd = -1;
   mod->main.vaddr = module_start - bias;
-- 
2.22.0

[PATCH v2 2/4] libcpu: merge libcpu_{i386,x86_64,bpf} into one library

[Omar Sandoval]

From: Omar Sandoval 

In preparation for combining the libebl backend modules, combine all of
the libcpu backends into libcpu.a.

Signed-off-by: Omar Sandoval 
---
 backends/ChangeLog   |  4 
 backends/Makefile.am | 12 +++-
 libcpu/ChangeLog |  4 
 libcpu/Makefile.am   | 12 +---
 4 files changed, 16 insertions(+), 16 deletions(-)

diff --git a/backends/ChangeLog b/backends/ChangeLog
index f1eaf14b..91790bb9 100644
--- a/backends/ChangeLog
+++ b/backends/ChangeLog
@@ -1,3 +1,7 @@
+2019-07-05  Omar Sandoval  
+
+   * Makefile.am: Replace libcpu_{i386,x86_64,bpf}.a with libcpu.a.
+
 2019-07-13 Mao Han 
 
* Makefile.am: Add C-SKY.
diff --git a/backends/Makefile.am b/backends/Makefile.am
index 175468f6..6470a313 100644
--- a/backends/Makefile.am
+++ b/backends/Makefile.am
@@ -44,6 +44,7 @@ noinst_LIBRARIES = $(libebl_pic)
 noinst_DATA = $(libebl_pic:_pic.a=.so)
 
 
+libcpu = ../libcpu/libcpu.a
 libelf = ../libelf/libelf.so
 libdw = ../libdw/libdw.so
 libeu = ../lib/libeu.a
@@ -51,7 +52,6 @@ libeu = ../lib/libeu.a
 i386_SRCS = i386_init.c i386_symbol.c i386_corenote.c i386_cfi.c \
i386_retval.c i386_regs.c i386_auxv.c i386_syscall.c \
i386_initreg.c i386_unwind.c
-cpu_i386 = ../libcpu/libcpu_i386.a
 libebl_i386_pic_a_SOURCES = $(i386_SRCS)
 am_libebl_i386_pic_a_OBJECTS = $(i386_SRCS:.c=.os)
 
@@ -62,7 +62,6 @@ am_libebl_sh_pic_a_OBJECTS = $(sh_SRCS:.c=.os)
 x86_64_SRCS = x86_64_init.c x86_64_symbol.c x86_64_corenote.c x86_64_cfi.c \
  x86_64_retval.c x86_64_regs.c i386_auxv.c x86_64_syscall.c \
  x86_64_initreg.c x86_64_unwind.c x32_corenote.c
-cpu_x86_64 = ../libcpu/libcpu_x86_64.a
 libebl_x86_64_pic_a_SOURCES = $(x86_64_SRCS)
 am_libebl_x86_64_pic_a_OBJECTS = $(x86_64_SRCS:.c=.os)
 
@@ -128,7 +127,6 @@ am_libebl_m68k_pic_a_OBJECTS = $(m68k_SRCS:.c=.os)
 m68k_corenote_no_Wpacked_not_aligned = yes
 
 bpf_SRCS = bpf_init.c bpf_regs.c bpf_symbol.c
-cpu_bpf = ../libcpu/libcpu_bpf.a
 libebl_bpf_pic_a_SOURCES = $(bpf_SRCS)
 am_libebl_bpf_pic_a_OBJECTS = $(bpf_SRCS:.c=.os)
 
@@ -142,20 +140,16 @@ csky_SRCS = csky_attrs.c csky_init.c csky_symbol.c 
csky_cfi.c \
 libebl_csky_pic_a_SOURCES = $(csky_SRCS)
 am_libebl_csky_pic_a_OBJECTS = $(csky_SRCS:.c=.os)
 
-libebl_%.so libebl_%.map: libebl_%_pic.a $(libelf) $(libdw) $(libeu)
+libebl_%.so libebl_%.map: libebl_%_pic.a $(libcpu) $(libelf) $(libdw) $(libeu)
@rm -f $(@:.so=.map)
$(AM_V_at)echo 'ELFUTILS_$(PACKAGE_VERSION) { global: $*_init; local: 
*; };' \
  > $(@:.so=.map)
$(AM_V_CCLD)$(LINK) $(dso_LDFLAGS) -o $(@:.map=.so) \
-   -Wl,--whole-archive $< $(cpu_$*) -Wl,--no-whole-archive \
+   -Wl,--whole-archive $< $(libcpu) -Wl,--no-whole-archive \
-Wl,--version-script,$(@:.so=.map),--no-undefined \
-Wl,--as-needed $(libelf) $(libdw) $(libeu)
@$(textrel_check)
 
-libebl_i386.so: $(cpu_i386)
-libebl_x86_64.so: $(cpu_x86_64)
-libebl_bpf.so: $(cpu_bpf)
-
 install: install-am install-ebl-modules
 install-ebl-modules:
$(mkinstalldirs) $(DESTDIR)$(libdir)/$(LIBEBL_SUBDIR)
diff --git a/libcpu/ChangeLog b/libcpu/ChangeLog
index adebbef8..c0ea72ec 100644
--- a/libcpu/ChangeLog
+++ b/libcpu/ChangeLog
@@ -1,3 +1,7 @@
+2019-07-05  Omar Sandoval  
+
+   * Makefile.am: Combine libcpu_{i386,x86_64,bpf}.a into libcpu.a.
+
 2018-11-04  Mark Wielaard  
 
* bpf_disasm.c (bpf_disasm): Recognize BPF_JLT, BPF_JLE, BPF_JSLT
diff --git a/libcpu/Makefile.am b/libcpu/Makefile.am
index 4c8778d1..a7d9f6fd 100644
--- a/libcpu/Makefile.am
+++ b/libcpu/Makefile.am
@@ -35,20 +35,16 @@ LEXCOMPILE = $(LEX) $(LFLAGS) $(AM_LFLAGS) -P$( $@T
$(AM_V_at)mv -f $@T $@
@@ -86,6 +82,8 @@ i386_gendis_LDADD = $(libeu) -lm
 
 i386_parse.h: i386_parse.c ;
 
+bpf_disasm_CFLAGS = -Wno-format-nonliteral
+
 EXTRA_DIST = defs/i386
 
 CLEANFILES += $(foreach P,i386 x86_64,$P_defs $P.mnemonics)
-- 
2.23.0

[PATCH v2 4/4] Don't use dlopen() for libebl modules

[Omar Sandoval]

From: Omar Sandoval 

Currently, architecture-specific code for libebl exists in separate
libebl_$ARCH.so libraries which libebl loads with dlopen() at runtime.
This makes it impossible to have standalone, statically-linked binaries
which use libdwfl if they depend on any architecture-specific
functionality. Additionally, when these libraries cannot be found, the
failure modes are non-obvious. So, let's get rid of libebl_$arch.so and
move it all into libdw.so/libdw.a, which simplifies things considerably.

Signed-off-by: Omar Sandoval 
---
 ChangeLog|   6 +
 Makefile.am  |   4 +-
 backends/ChangeLog   |   1 +
 backends/Makefile.am | 100 +++
 configure.ac |  12 +-
 libdw/ChangeLog  |   4 +
 libdw/Makefile.am|  21 ++--
 libebl/ChangeLog |   6 +
 libebl/Makefile.am   |   1 -
 libebl/eblclosebackend.c |   4 -
 libebl/eblopenbackend.c  | 266 +++
 libebl/libebl.h  |   8 +-
 libebl/libeblP.h |   3 -
 src/ChangeLog|   7 ++
 src/Makefile.am  |  20 +--
 tests/ChangeLog  |   5 +
 tests/Makefile.am|  36 +++---
 tests/test-subr.sh   |   4 +-
 18 files changed, 200 insertions(+), 308 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index bed3999f..911cf354 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,9 @@
+2019-07-05  Omar Sandoval  
+
+   * configure.ac: Get rid of --enable-libebl-subdir.
+   * Makefile.am (SUBDIRS): Reorder backends and libcpu before libebl to
+   satisfy build dependencies.
+
 2019-08-13  Mark Wielaard  
 
* configure.ac: Set version to 0.177.
diff --git a/Makefile.am b/Makefile.am
index 2ff444e7..51f54552 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -27,8 +27,8 @@ AM_MAKEFLAGS = --no-print-directory
 pkginclude_HEADERS = version.h
 
 # Add doc back when we have some real content.
-SUBDIRS = config m4 lib libelf libebl libdwelf libdwfl libdw libcpu libasm \
- backends src po tests
+SUBDIRS = config m4 lib libelf libcpu backends libebl libdwelf libdwfl libdw \
+ libasm src po tests
 
 EXTRA_DIST = elfutils.spec GPG-KEY NOTES CONTRIBUTING \
 COPYING COPYING-GPLV2 COPYING-LGPLV3
diff --git a/backends/ChangeLog b/backends/ChangeLog
index 3faee13e..3f5f9bb0 100644
--- a/backends/ChangeLog
+++ b/backends/ChangeLog
@@ -2,6 +2,7 @@
 
* Makefile.am: Replace libcpu_{i386,x86_64,bpf}.a with libcpu.a.
Replace libcpu.a with libcpu_pic.a.
+   Combine libebl_CPU.so modules into libebl_backends{,_pic}.a.
 
 2019-07-13 Mao Han 
 
diff --git a/backends/Makefile.am b/backends/Makefile.am
index be470528..f4052125 100644
--- a/backends/Makefile.am
+++ b/backends/Makefile.am
@@ -28,96 +28,60 @@
 ## the GNU Lesser General Public License along with this program.  If
 ## not, see <http://www.gnu.org/licenses/>.
 include $(top_srcdir)/config/eu.am
+if BUILD_STATIC
+AM_CFLAGS += $(fpic_CFLAGS)
+endif
 AM_CPPFLAGS += -I$(top_srcdir)/libebl -I$(top_srcdir)/libasm \
   -I$(top_srcdir)/libelf -I$(top_srcdir)/libdw
 
+noinst_LIBRARIES = libebl_backends.a libebl_backends_pic.a
 
 modules = i386 sh x86_64 ia64 alpha arm aarch64 sparc ppc ppc64 s390 \
  tilegx m68k bpf riscv csky
-libebl_pic = libebl_i386_pic.a libebl_sh_pic.a libebl_x86_64_pic.a\
-libebl_ia64_pic.a libebl_alpha_pic.a libebl_arm_pic.a\
-libebl_aarch64_pic.a libebl_sparc_pic.a libebl_ppc_pic.a \
-libebl_ppc64_pic.a libebl_s390_pic.a libebl_tilegx_pic.a \
-libebl_m68k_pic.a libebl_bpf_pic.a libebl_riscv_pic.a\
-libebl_csky_pic.a
-noinst_LIBRARIES = $(libebl_pic)
-noinst_DATA = $(libebl_pic:_pic.a=.so)
-
-
-libcpu = ../libcpu/libcpu_pic.a
-libelf = ../libelf/libelf.so
-libdw = ../libdw/libdw.so
-libeu = ../lib/libeu.a
 
 i386_SRCS = i386_init.c i386_symbol.c i386_corenote.c i386_cfi.c \
i386_retval.c i386_regs.c i386_auxv.c i386_syscall.c \
i386_initreg.c i386_unwind.c
-libebl_i386_pic_a_SOURCES = $(i386_SRCS)
-am_libebl_i386_pic_a_OBJECTS = $(i386_SRCS:.c=.os)
 
 sh_SRCS = sh_init.c sh_symbol.c sh_corenote.c sh_regs.c sh_retval.c
-libebl_sh_pic_a_SOURCES = $(sh_SRCS)
-am_libebl_sh_pic_a_OBJECTS = $(sh_SRCS:.c=.os)
 
 x86_64_SRCS = x86_64_init.c x86_64_symbol.c x86_64_corenote.c x86_64_cfi.c \
- x86_64_retval.c x86_64_regs.c i386_auxv.c x86_64_syscall.c \
- x86_64_initreg.c x86_64_unwind.c x32_corenote.c
-libebl_x86_64_pic_a_SOURCES = $(x86_64_SRCS)
-am_libebl_x86_64_pic_a_OBJECTS = $(x86_64_SRCS:.c=.os)
+ x86_64_retval.c x86_64_regs.c x86_64_syscall.c x86_64_initreg.c \
+ x86_64_unwind.c x32_corenote.c
+
 
 ia64_SRCS = ia64_init.c ia64_symbol.c ia64_regs.c ia64_retval.c
-libebl_ia64_pic_a_SOURCES = $(ia64_SRCS)
-am_libebl_ia64_pic_a_OBJECTS = $(ia64_SRCS:.c=.os)
 
 alpha_SRCS = alpha_init.c alpha_symbol.c alpha_retval.c a

[PATCH v2 1/4] libebl: remove unnecessary variable in Makefile.am

[Omar Sandoval]

From: Omar Sandoval 

gen_SOURCES is assigned to libebl_a_SOURCES and never used again. Get
rid of it.

Signed-off-by: Omar Sandoval 
---
 libebl/ChangeLog   |  4 
 libebl/Makefile.am | 39 +++
 2 files changed, 23 insertions(+), 20 deletions(-)

diff --git a/libebl/ChangeLog b/libebl/ChangeLog
index bb91c152..a772ffa3 100644
--- a/libebl/ChangeLog
+++ b/libebl/ChangeLog
@@ -1,3 +1,7 @@
+2019-07-05  Omar Sandoval  
+
+   * Makefile.am (gen_SOURCES): Remove.
+
 2019-04-29 Mao Han 
 
* eblopenbackend.c: Add C-SKY.
diff --git a/libebl/Makefile.am b/libebl/Makefile.am
index 737de6b0..ccc1a31a 100644
--- a/libebl/Makefile.am
+++ b/libebl/Makefile.am
@@ -37,26 +37,25 @@ lib_LIBRARIES = libebl.a
 
 pkginclude_HEADERS = libebl.h
 
-gen_SOURCES = eblopenbackend.c eblclosebackend.c \
- eblreloctypename.c eblsegmenttypename.c \
- eblsectiontypename.c eblmachineflagname.c \
- eblsymboltypename.c ebldynamictagname.c eblsectionname.c \
- eblsymbolbindingname.c eblbackendname.c eblosabiname.c \
- eblmachineflagcheck.c eblmachinesectionflagcheck.c \
- eblreloctypecheck.c eblrelocvaliduse.c eblrelocsimpletype.c \
- ebldynamictagcheck.c eblcorenotetypename.c eblobjnotetypename.c \
- eblcorenote.c eblobjnote.c ebldebugscnp.c \
- eblgotpcreloccheck.c eblcopyrelocp.c eblsectionstripp.c \
- eblelfclass.c eblelfdata.c eblelfmachine.c \
- ebl_check_special_symbol.c eblbsspltp.c eblretval.c \
- eblreginfo.c eblnonerelocp.c eblrelativerelocp.c \
- eblsysvhashentrysize.c eblauxvinfo.c eblcheckobjattr.c \
- ebl_check_special_section.c ebl_syscall_abi.c eblabicfi.c \
- eblstother.c eblinitreg.c ebldwarftoregno.c eblnormalizepc.c \
- eblunwind.c eblresolvesym.c eblcheckreloctargettype.c \
- ebl_data_marker_symbol.c
-
-libebl_a_SOURCES = $(gen_SOURCES)
+libebl_a_SOURCES = eblopenbackend.c eblclosebackend.c eblreloctypename.c \
+  eblsegmenttypename.c eblsectiontypename.c \
+  eblmachineflagname.c eblsymboltypename.c \
+  ebldynamictagname.c eblsectionname.c \
+  eblsymbolbindingname.c eblbackendname.c eblosabiname.c \
+  eblmachineflagcheck.c eblmachinesectionflagcheck.c \
+  eblreloctypecheck.c eblrelocvaliduse.c \
+  eblrelocsimpletype.c ebldynamictagcheck.c \
+  eblcorenotetypename.c eblobjnotetypename.c eblcorenote.c \
+  eblobjnote.c ebldebugscnp.c eblgotpcreloccheck.c \
+  eblcopyrelocp.c eblsectionstripp.c eblelfclass.c \
+  eblelfdata.c eblelfmachine.c ebl_check_special_symbol.c \
+  eblbsspltp.c eblretval.c eblreginfo.c eblnonerelocp.c \
+  eblrelativerelocp.c eblsysvhashentrysize.c eblauxvinfo.c \
+  eblcheckobjattr.c ebl_check_special_section.c \
+  ebl_syscall_abi.c eblabicfi.c eblstother.c eblinitreg.c \
+  ebldwarftoregno.c eblnormalizepc.c eblunwind.c \
+  eblresolvesym.c eblcheckreloctargettype.c \
+  ebl_data_marker_symbol.c
 
 noinst_HEADERS = libeblP.h ebl-hooks.h
 
-- 
2.23.0

[PATCH v2 3/4] Add PIC and non-PIC variants of libcpu and libebl

[Omar Sandoval]

From: Omar Sandoval 

Currently, libcpu and libebl are always compiled as PIC. An upcoming
change will add the objects from libcpu.a and libebl.a to libdw.a, which
should not be PIC unless configured that way. So, make libcpu.a and
libebl.a non-PIC and add libcpu_pic.a and libebl_pic.a.

Signed-off-by: Omar Sandoval 
---
 backends/ChangeLog   |  1 +
 backends/Makefile.am |  2 +-
 libcpu/ChangeLog |  2 ++
 libcpu/Makefile.am   | 18 +-
 libdw/ChangeLog  |  4 
 libdw/Makefile.am|  2 +-
 libebl/ChangeLog |  4 +++-
 libebl/Makefile.am   |  8 +++-
 8 files changed, 32 insertions(+), 9 deletions(-)

diff --git a/backends/ChangeLog b/backends/ChangeLog
index 91790bb9..3faee13e 100644
--- a/backends/ChangeLog
+++ b/backends/ChangeLog
@@ -1,6 +1,7 @@
 2019-07-05  Omar Sandoval  
 
* Makefile.am: Replace libcpu_{i386,x86_64,bpf}.a with libcpu.a.
+   Replace libcpu.a with libcpu_pic.a.
 
 2019-07-13 Mao Han 
 
diff --git a/backends/Makefile.am b/backends/Makefile.am
index 6470a313..be470528 100644
--- a/backends/Makefile.am
+++ b/backends/Makefile.am
@@ -44,7 +44,7 @@ noinst_LIBRARIES = $(libebl_pic)
 noinst_DATA = $(libebl_pic:_pic.a=.so)
 
 
-libcpu = ../libcpu/libcpu.a
+libcpu = ../libcpu/libcpu_pic.a
 libelf = ../libelf/libelf.so
 libdw = ../libdw/libdw.so
 libeu = ../lib/libeu.a
diff --git a/libcpu/ChangeLog b/libcpu/ChangeLog
index c0ea72ec..883896a2 100644
--- a/libcpu/ChangeLog
+++ b/libcpu/ChangeLog
@@ -1,6 +1,8 @@
 2019-07-05  Omar Sandoval  
 
* Makefile.am: Combine libcpu_{i386,x86_64,bpf}.a into libcpu.a.
+   Make libcpu.a non-PIC by default.
+   Add libcpu_pic.a.
 
 2018-11-04  Mark Wielaard  
 
diff --git a/libcpu/Makefile.am b/libcpu/Makefile.am
index a7d9f6fd..88717361 100644
--- a/libcpu/Makefile.am
+++ b/libcpu/Makefile.am
@@ -30,15 +30,22 @@
 include $(top_srcdir)/config/eu.am
 AM_CPPFLAGS += -I$(srcdir)/../libelf -I$(srcdir)/../libebl \
-I$(srcdir)/../libdw -I$(srcdir)/../libasm
-AM_CFLAGS += $(fpic_CFLAGS) -fdollars-in-identifiers
+if BUILD_STATIC
+AM_CFLAGS += $(fpic_CFLAGS)
+endif
+AM_CFLAGS += -fdollars-in-identifiers
 LEXCOMPILE = $(LEX) $(LFLAGS) $(AM_LFLAGS) -P$(
+
+   * Makefile.am (libdw_so_LIBS): Replace libebl.a with libebl_pic.a.
+
 2019-08-25  Jonathon Anderson  
 
* dwarf_getcfi.c (dwarf_getcfi): Set default_same_value to false.
diff --git a/libdw/Makefile.am b/libdw/Makefile.am
index 7a3d5322..7dc4cec0 100644
--- a/libdw/Makefile.am
+++ b/libdw/Makefile.am
@@ -106,7 +106,7 @@ libdw_pic_a_SOURCES =
 am_libdw_pic_a_OBJECTS = $(libdw_a_SOURCES:.c=.os)
 
 libdw_so_LIBS = libdw_pic.a ../libdwelf/libdwelf_pic.a \
- ../libdwfl/libdwfl_pic.a ../libebl/libebl.a
+ ../libdwfl/libdwfl_pic.a ../libebl/libebl_pic.a
 libdw_so_DEPS = ../lib/libeu.a ../libelf/libelf.so
 libdw_so_LDLIBS = $(libdw_so_DEPS) -ldl -lz $(argp_LDADD) $(zip_LIBS)
 libdw_so_SOURCES =
diff --git a/libebl/ChangeLog b/libebl/ChangeLog
index a772ffa3..3a836c89 100644
--- a/libebl/ChangeLog
+++ b/libebl/ChangeLog
@@ -1,6 +1,8 @@
 2019-07-05  Omar Sandoval  
 
-   * Makefile.am (gen_SOURCES): Remove.
+   * Makefile.am: Make libebl.a non-PIC by default.
+   Add libebl_pic.a.
+   (gen_SOURCES): Remove.
 
 2019-04-29 Mao Han 
 
diff --git a/libebl/Makefile.am b/libebl/Makefile.am
index ccc1a31a..b85ead01 100644
--- a/libebl/Makefile.am
+++ b/libebl/Makefile.am
@@ -28,12 +28,15 @@
 ## not, see <http://www.gnu.org/licenses/>.
 ##
 include $(top_srcdir)/config/eu.am
+if BUILD_STATIC
 AM_CFLAGS += $(fpic_CFLAGS)
+endif
 AM_CPPFLAGS += -I$(srcdir)/../libelf -I$(srcdir)/../libdw -I$(srcdir)/../libasm
 VERSION = 1
 LIBEBL_SUBDIR = @LIBEBL_SUBDIR@
 
 lib_LIBRARIES = libebl.a
+noinst_LIBRARIES = libebl_pic.a
 
 pkginclude_HEADERS = libebl.h
 
@@ -57,6 +60,9 @@ libebl_a_SOURCES = eblopenbackend.c eblclosebackend.c 
eblreloctypename.c \
   eblresolvesym.c eblcheckreloctargettype.c \
   ebl_data_marker_symbol.c
 
+libebl_pic_a_SOURCES =
+am_libebl_pic_a_OBJECTS = $(libebl_a_SOURCES:.c=.os)
+
 noinst_HEADERS = libeblP.h ebl-hooks.h
 
-CLEANFILES += $(am_libebl_pic_a_OBJECTS)
+MOSTLYCLEANFILES = $(am_libebl_pic_a_OBJECTS)
-- 
2.23.0

[PATCH v2 0/4] elfutils: don't use dlopen() for libebl modules

[Omar Sandoval]

From: Omar Sandoval 

This is a rebase of my previously posted series [1][2].

Besides the rebase, the only other change is dropping patch 5, which we
agreed is not necessary.

Thanks!

1: https://sourceware.org/ml/elfutils-devel/2019-q3/msg00020.html
2: https://sourceware.org/ml/elfutils-devel/2019-q3/msg00010.html

Omar Sandoval (4):
  libebl: remove unnecessary variable in Makefile.am
  libcpu: merge libcpu_{i386,x86_64,bpf} into one library
  Add PIC and non-PIC variants of libcpu and libebl
  Don't use dlopen() for libebl modules

 ChangeLog|   6 +
 Makefile.am  |   4 +-
 backends/ChangeLog   |   6 +
 backends/Makefile.am | 106 +++-
 configure.ac |  12 +-
 libcpu/ChangeLog |   6 +
 libcpu/Makefile.am   |  24 ++--
 libdw/ChangeLog  |   8 ++
 libdw/Makefile.am|  21 ++--
 libebl/ChangeLog |  12 ++
 libebl/Makefile.am   |  46 +++
 libebl/eblclosebackend.c |   4 -
 libebl/eblopenbackend.c  | 266 +++
 libebl/libebl.h  |   8 +-
 libebl/libeblP.h |   3 -
 src/ChangeLog|   7 ++
 src/Makefile.am  |  20 +--
 tests/ChangeLog  |   5 +
 tests/Makefile.am|  36 +++---
 tests/test-subr.sh   |   4 +-
 20 files changed, 261 insertions(+), 343 deletions(-)

-- 
2.23.0

[PATCH] Add elfclassify to .gitignore

[Omar Sandoval]

From: Omar Sandoval 

The tool was just added, but it's missing the ignore entry.

Signed-off-by: Omar Sandoval 
---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index e8201dcc..72f22855 100644
--- a/.gitignore
+++ b/.gitignore
@@ -56,6 +56,7 @@ Makefile.in
 /po/stamp-po
 /src/addr2line
 /src/ar
+/src/elfclassify
 /src/elfcmp
 /src/elfcompress
 /src/elflint
-- 
2.23.0