Re: [PATCH 6/6 v2] doc: Add elf_kind.3

[Aaron Merey]

On Wed, May 14, 2025 at 10:16 AM Mark Wielaard  wrote:
>
> Hi Aaron,
>
> On Mon, 2025-05-12 at 20:33 -0400, Aaron Merey wrote:
> > v2: Mention ar(1) and that elfutils libelf does not support COFF object
> > files.
> >
> > On Thu, May 1, 2025 at 1:17 PM Mark Wielaard  wrote:
> > >
> > > > +.SH SEE ALSO
> > > > +.BR libelf (3),
> > > > +.BR elf (5)
> > >
> > > elf.5 doesn't describe the ar format, should it? Or should we have an
> > > ar.5 man page?
> >
> > The ar format is part of the System V ABI so it would make sense for the
> > kernel man-pages project to include ar.5 along with elf.5.  For now I've
> > added ar(1) and ranlib(1) under SEE ALSO.
>
> OK, these refer to the binutils ar and ranlib versions. We do have eu-
> ar and eu-ranlib but those don't have man pages (yet?).

That's right, once we have eu-ar and eu-ranlib man pages we can
mention those here.

>
> Looks OK with one typo:
>
> > +.TP
> > +.B ELF_K_COFF
> > +COFF object file. COFF is supported by elfutils libelf.
>
> Missing a NOT.

Oops, thanks for catching this. Fixed.

>
> Cheers,
>
> Mark
>

Re: [RFC] tests/run-stack-live-test.sh: prototype 'live' eu-stack testing

[Serhei Makarov]

On Wed, May 14, 2025, at 9:54 AM, Mark Wielaard wrote:
> I like the testrun_compare_fuzzy name.
Nice, though I might have to end up with several versions of the function
(testrun_compare_fuzzy_stack, testrun_compare_fuzzy_stacktrace, ...).
It's an open question whether they need to be generalized to test_subr.sh,
to some other file, or kept in the local test case.

> Yes, just remove everything after @... (the symver).
> Maybe look at gcc optimization suffixes like .isra, .constprop, .clone,
> .ipa (don't know if there is some definite list).
> Try to stick to plain C, otherwise think about demangling (or not?).
Yes please :) I don't want to think about how consistent or not C++
mangling is across systems, although I think it's a standard procedure?

>> - Something better than sed for the scrubbing?
>
> It works, so...
Anything's good enough with enough comments!

> Most probably not generically. Especially not the container builders.
> But you might make some deal with a specific direct hardware worker
> admin?
Thankfully, it seems there's a fair chance I can test something without
needing root privileges, so I'm going to aim for that goal.

-- 
All the best,
Serhei
http://serhei.io

Re: [RFC] tests/run-stack-live-test.sh: prototype 'live' eu-stack testing

[Serhei Makarov]

On Wed, May 14, 2025, at 10:30 AM, Aaron Merey wrote:
> If it's not quite generic enough for test-subr.sh we can always
> include this in a stacktrace-subr.sh file, for instance.
Yeah, I'm thinking this is the more likely outcome.

>> +#
>> +# TODO(REVIEW): Better shell-isms for comparing file and regex?
>> +# \(\s\e\d\)\+\i\s\a\d\d\i\c\t\e\d\\t\o\b\a\ck\s\l\a\s\h\e\s
>
> I think the use of sed here is fine.  If you wanted to improve
> readability you could perform the scrubbing across multiple sed
> commands each with a comment explaining the step.
Anything's good enough with enough comments!

There's also the possibility of awk being powerful enough to make
me want to use it. Do you know if awk is considered sufficiently
standard for the testsuite to depend on it?

> FYI here is my results diff from running this test:
> -snip-
Thanks, that's another item for the dataset I got from the trybot runs.
Unfortunately, I won't know for sure until I start to test things like bsd
is there a list somewhere of unices we're supposed to support?

-- 
All the best,
Serhei
http://serhei.io

☠ Buildbot (Sourceware): elfutils - failed test (failure) (main)

[builder]

A new failure has been detected on builder elfutils-debian-ppc64 while building 
elfutils.

Full details are available at:
https://builder.sourceware.org/buildbot/#/builders/63/builds/509

Build state: failed test (failure)
Revision: 11c54284514498c29147972ddbf35eac4c95039c
Worker: debian-ppc64
Build Reason: (unknown)
Blamelist: Aaron Merey 

Steps:

- 0: worker_preparation ( success )

- 1: set package name ( success )

- 2: git checkout ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/509/steps/2/logs/stdio

- 3: autoreconf ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/509/steps/3/logs/stdio

- 4: configure ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/509/steps/4/logs/stdio
- config.log: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/509/steps/4/logs/config_log

- 5: get version ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/509/steps/5/logs/stdio
- property changes: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/509/steps/5/logs/property_changes

- 6: make ( warnings )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/509/steps/6/logs/stdio
- warnings (3): 
https://builder.sourceware.org/buildbot/#/builders/63/builds/509/steps/6/logs/warnings__3_

- 7: make check ( failure )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/509/steps/7/logs/stdio
- test-suite.log: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/509/steps/7/logs/test-suite_log

- 8: prep ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/509/steps/8/logs/stdio

- 9: build bunsen.cpio.gz ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/509/steps/9/logs/stdio

- 10: fetch bunsen.cpio.gz ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/509/steps/10/logs/stdio

- 11: unpack bunsen.cpio.gz ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/509/steps/11/logs/stdio

- 12: pass .bunsen.source.* ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/509/steps/12/logs/stdio

- 13: upload to bunsen ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/509/steps/13/logs/stdio

- 14: clean up ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/509/steps/14/logs/stdio

- 15: make distclean ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/509/steps/15/logs/stdio

☺ Buildbot (Sourceware): elfutils - build successful (main)

[builder]

A restored build has been detected on builder elfutils-debian-ppc64 while 
building elfutils.

Full details are available at:
https://builder.sourceware.org/buildbot/#/builders/63/builds/510

Build state: build successful
Revision: c18a9ef154bf5583183443a2bf7de8f4cea89d8f
Worker: debian-ppc64
Build Reason: (unknown)
Blamelist: Aaron Merey 

Steps:

- 0: worker_preparation ( success )

- 1: set package name ( success )

- 2: git checkout ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/510/steps/2/logs/stdio

- 3: autoreconf ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/510/steps/3/logs/stdio

- 4: configure ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/510/steps/4/logs/stdio
- config.log: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/510/steps/4/logs/config_log

- 5: get version ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/510/steps/5/logs/stdio
- property changes: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/510/steps/5/logs/property_changes

- 6: make ( warnings )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/510/steps/6/logs/stdio
- warnings (3): 
https://builder.sourceware.org/buildbot/#/builders/63/builds/510/steps/6/logs/warnings__3_

- 7: make check ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/510/steps/7/logs/stdio
- test-suite.log: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/510/steps/7/logs/test-suite_log

- 8: prep ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/510/steps/8/logs/stdio

- 9: build bunsen.cpio.gz ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/510/steps/9/logs/stdio

- 10: fetch bunsen.cpio.gz ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/510/steps/10/logs/stdio

- 11: unpack bunsen.cpio.gz ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/510/steps/11/logs/stdio

- 12: pass .bunsen.source.* ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/510/steps/12/logs/stdio

- 13: upload to bunsen ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/510/steps/13/logs/stdio

- 14: clean up ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/510/steps/14/logs/stdio

- 15: make distclean ( success )
Logs:
- stdio: 
https://builder.sourceware.org/buildbot/#/builders/63/builds/510/steps/15/logs/stdio

Re: [RFC] tests/run-stack-live-test.sh: prototype 'live' eu-stack testing

[Mark Wielaard]

Hi Serhei,

On Thu, 2025-05-08 at 18:28 -0400, Serhei Makarov wrote:
> Missing a few pieces, but worth sharing as an RFC. My idea is to
> ensure better test coverage for eu-stack and then
> eu-stacktrace+libdwfl_stacktrace by running against a live process
> with known content, stopped at a known location, and aggressively
> scrubbing output that's known to vary from testrun to testrun.
> 
> This is a very basic preview of how that might look. If the approach
> is sound, I hope to make it more sophisticated/reliable.

I like the testrun_compare_fuzzy name.

> Unanswered questions:
> - Scrub more data (e.g. libc symvers) from a more known program.
>   Scrub stack frame numbers to account for a case where extra frames
>   appear / are missing at the bottom of the stack?

Yes, just remove everything after @... (the symver).
Maybe look at gcc optimization suffixes like .isra, .constprop, .clone,
.ipa (don't know if there is some definite list).
Try to stick to plain C, otherwise think about demangling (or not?).
Scrub everything after you have seen main (or in multi-threaded
programs clone), so things like __libc_start_call_main, _start?

> - Something better than sed for the scrubbing?

It works, so...

> - An equivalent eu-stacktrace test will require privileged perf_events
>   access for profiling data and therefore likely to be skipped by
>   default. How feasible is it to be enabled on the buildbots, though?

Most probably not generically. Especially not the container builders.
But you might make some deal with a specific direct hardware worker
admin?

Cheers,

Mark

Re: [PATCH 1/6] doc/Makefile.am: Sort manpages in alphabetical order

[Mark Wielaard]

Hi Aaron,

Yes, please.

Thanks,

Mark

Re: [PATCH 2/6 v2] doc: Add elf_end.3

[Mark Wielaard]

Hi Aaron,

On Mon, 2025-05-12 at 20:33 -0400, Aaron Merey wrote:
> v2: Mention elf arg may be NULL.

Looks good.

> On Wed, Apr 30, 2025 at 1:18 PM Mark Wielaard  wrote:
> > 
> > So we do have a tiny elf_begin.3 man page, but it doesn't really
> > describe anything. The above seems a technically correct description of
> > ELF descriptor internal reference counting, but I am not sure it makes
> > sense without a full description of elf_begin.
> 
> I'll post a patch updating elf_begin.3.

Thanks,

Mark

Re: [PATCH 3/6 v2] doc: Add elf_fill.3

[Mark Wielaard]

Hi Aaron,

On Mon, 2025-05-12 at 20:33 -0400, Aaron Merey wrote:
> v2: Mention fill value only applies to new gaps and set Thread Safety
> atrribute to 'MT-Unsafe race'

Looks good.

Thanks,

Mark

Re: [PATCH 4/6] doc: Add elf_getbase.3

[Mark Wielaard]

Hi Aaron,

On Mon, 2025-05-12 at 20:33 -0400, Aaron Merey wrote:
> v2: mention that base offset returned is always 0 for non ELF_K_AR
> kinds. Also mention elf_getaroff and elf_rawelf.

Looks good.

Thanks,

Mark

Re: [PATCH 5/6 v2] doc: Add elf_hash.3

[Mark Wielaard]

Hi Aaron,

On Mon, 2025-05-12 at 20:33 -0400, Aaron Merey wrote:
> v2: Mention use with SHT_HASH and clarify that only the lower 32 bits of
> the return value are used.

Looks good.

Thanks,

Mark

Re: [PATCH 6/6 v2] doc: Add elf_kind.3

[Mark Wielaard]

Hi Aaron,

On Mon, 2025-05-12 at 20:33 -0400, Aaron Merey wrote:
> v2: Mention ar(1) and that elfutils libelf does not support COFF object
> files.
> 
> On Thu, May 1, 2025 at 1:17 PM Mark Wielaard  wrote:
> > 
> > > +.SH SEE ALSO
> > > +.BR libelf (3),
> > > +.BR elf (5)
> > 
> > elf.5 doesn't describe the ar format, should it? Or should we have an
> > ar.5 man page?
> 
> The ar format is part of the System V ABI so it would make sense for the
> kernel man-pages project to include ar.5 along with elf.5.  For now I've
> added ar(1) and ranlib(1) under SEE ALSO.

OK, these refer to the binutils ar and ranlib versions. We do have eu-
ar and eu-ranlib but those don't have man pages (yet?).

Looks OK with one typo:

> +.TP
> +.B ELF_K_COFF
> +COFF object file. COFF is supported by elfutils libelf.

Missing a NOT.

Cheers,

Mark

Re: [RFC] tests/run-stack-live-test.sh: prototype 'live' eu-stack testing

[Aaron Merey]

On Thu, May 8, 2025 at 6:28 PM Serhei Makarov  wrote:
>
> Missing a few pieces, but worth sharing as an RFC. My idea is to
> ensure better test coverage for eu-stack and then
> eu-stacktrace+libdwfl_stacktrace by running against a live process
> with known content, stopped at a known location, and aggressively
> scrubbing output that's known to vary from testrun to testrun.
>
> This is a very basic preview of how that might look. If the approach
> is sound, I hope to make it more sophisticated/reliable.
>
> Unanswered questions:
> - Scrub more data (e.g. libc symvers) from a more known program.
>   Scrub stack frame numbers to account for a case where extra frames
>   appear / are missing at the bottom of the stack?
> - Something better than sed for the scrubbing?

Sed is fine, we already use it (along with cut) in other test scripts
for scrubbing components of output that might differ between runs or
across environments. run-dwfl-core-noncontig.sh and run-sysroot.sh are
two examples.

> - An equivalent eu-stacktrace test will require privileged perf_events
>   access for profiling data and therefore likely to be skipped by
>   default. How feasible is it to be enabled on the buildbots, though?
>
> * tests/run-stack-live-test.sh: New test with wild and fuzzy
>   testrun_compare variant that scrubs inherently unpredictable parts of
>   the data. Needs to scrub even more.
> * tests/Makefile.am (TESTS): Add run-stack-live-test.sh.
> ---
>  tests/Makefile.am|  3 +-
>  tests/run-stack-live-test.sh | 64 
>  2 files changed, 66 insertions(+), 1 deletion(-)
>  create mode 100755 tests/run-stack-live-test.sh
>
> diff --git a/tests/Makefile.am b/tests/Makefile.am
> index 00ba754d..ecd514c7 100644
> --- a/tests/Makefile.am
> +++ b/tests/Makefile.am
> @@ -191,7 +191,8 @@ TESTS = run-arextract.sh run-arsymtest.sh run-ar.sh 
> newfile test-nlist \
> run-backtrace-core-s390x.sh run-backtrace-core-s390.sh \
> run-backtrace-core-aarch64.sh run-backtrace-core-sparc.sh \
> run-backtrace-demangle.sh run-stack-d-test.sh run-stack-i-test.sh \
> -   run-stack-demangled-test.sh run-readelf-zx.sh run-readelf-zp.sh \
> +   run-stack-demangled-test.sh run-stack-live-test.sh \
> +   run-readelf-zx.sh run-readelf-zp.sh \
> run-readelf-arm-flags.sh \
> run-readelf-addr.sh run-readelf-str.sh \
> run-readelf-multi-noline.sh \
> diff --git a/tests/run-stack-live-test.sh b/tests/run-stack-live-test.sh
> new file mode 100755
> index ..808421bb
> --- /dev/null
> +++ b/tests/run-stack-live-test.sh
> @@ -0,0 +1,64 @@
> +#! /bin/sh
> +# Copyright (C) 2025 Red Hat, Inc.
> +# This file is part of elfutils.
> +#
> +# This file is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# elfutils is distributed in the hope that it will be useful, but
> +# WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see .
> +
> +. $srcdir/test-subr.sh
> +
> +# Depending on whether we are running make check or make installcheck
> +# the actual binary name under test might be different. It is used in
> +# the error message, which we also try to match.
> +if test "$elfutils_testrun" = "installed"; then
> +STACKCMD=${bindir}/`program_transform stack`
> +else
> +STACKCMD=${abs_top_builddir}/src/stack
> +fi
> +
> +# TODO(REVIEW): Can we make the data-scrubbing generic enough
> +# (across multiple eu-stack/eu-stacktrace test cases) to move
> +# to test_subr.sh?

If it's not quite generic enough for test-subr.sh we can always
include this in a stacktrace-subr.sh file, for instance.

> +#
> +# TODO(REVIEW): Better shell-isms for comparing file and regex?
> +# \(\s\e\d\)\+\i\s\a\d\d\i\c\t\e\d\\t\o\b\a\ck\s\l\a\s\h\e\s

I think the use of sed here is fine.  If you wanted to improve
readability you could perform the scrubbing across multiple sed
commands each with a comment explaining the step.

> +testrun_compare_fuzzy()
> +{
> +outfile="${1##*/}.out"
> +testrun_out $outfile "$@"
> +sed -i 's/\(PID\|TID\|#[0-9]\+\)\( \+\)\(\(0x\)\?[0-9a-f]\+\)/\1\2nn/g' 
> $outfile
> +diff -u $outfile -
> +}
> +
> +# TODO: Need to scrub more data (e.g. GLIBC_ bits),
> +# and use a program whose inner content we control:
> +sleep 10 &
> +PID=$!
> +testrun_compare_fuzzy ${abs_top_builddir}/src/stack -p $PID < +PID nn - process
> +TID nn:
> +#0  nn clock_nanosleep@GLIBC_2.2.5
> +#1  nn __nanosleep
> +#2  nn main
> +#3  nn __libc_start_call_main
> +#4  nn __libc_start_main@@GLIBC_2.34
> +#5  nn _start
> +EOF