Re: PATCH: debuginfod thread naming
On Tue, Mar 30, 2021 at 01:57:08PM -0400, Frank Ch. Eigler via Elfutils-devel wrote: > Planning to merge this as obvious. > > > Author: Frank Ch. Eigler > Date: Tue Mar 30 13:22:43 2021 -0400 > > debuginfod: Set child thread names via pthread_setname_np() > > In order to assist problem diagnosis / monitoring, use this > gnu-flavoured pthread function to set purpose names to the various > child threads debuginfod starts. libmicrohttpd already sets this for > its threads. Yes please. This looks useful. Thanks, Mark
[Bug debuginfod/27669] extend webapi for partial match source queries
https://sourceware.org/bugzilla/show_bug.cgi?id=27669 Mark Wielaard changed: What|Removed |Added CC||mark at klomp dot org --- Comment #1 from Mark Wielaard --- (In reply to Frank Ch. Eigler from comment #0) > It may be desirable to fetch source files by buildid and __FILE__ name only, > as these can be introspectively extracted sometimes from a running program > image, without DWARF comp_dir etc. work. > > It could look like this: > >/buildid/HEXCODE/source*/PARTIAL_PATH > > and implement the search by using a sql LIKE operator > >artifactsrc like '%PARTIAL_PATH' > > in the mhd-query-s block of handle_buildid() This might indeed be useful. But how exactly does one get just the buildid and a source file (base) name? If this comes through DWARF then why not get the comp_dir at the same time? -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/27669] extend webapi for partial match source queries
https://sourceware.org/bugzilla/show_bug.cgi?id=27669 --- Comment #2 from Frank Ch. Eigler --- > But how exactly does one get just the buildid and a source file (base) name? You do not want to know. :-) To find self-buildid: https://github.com/systemd/systemd/pull/19160 To find self source name: https://github.com/systemd/systemd/blob/fee6441601c979165ebcbb35472036439f8dad5f/src/systemd/sd-journal.h (Use __FILE__ in sd_journal_() macro apis to find the CU name, sort of) -- You are receiving this mail because: You are on the CC list for the bug.
Buildbot failure in Wildebeest Builder on whole buildset
The Buildbot has detected a failed build on builder whole buildset while building elfutils. Full details are available at: https://builder.wildebeest.org/buildbot/#builders/10/builds/714 Buildbot URL: https://builder.wildebeest.org/buildbot/ Worker for this Build: fedora-s390x Build Reason: Blamelist: Frank Ch. Eigler BUILD FAILED: failed test (failure) Sincerely, -The Buildbot
Re: Buildbot failure in Wildebeest Builder on whole buildset
On Wed, Mar 31, 2021 at 11:29:10AM +, build...@builder.wildebeest.org wrote: > The Buildbot has detected a failed build on builder whole buildset while > building elfutils. > Full details are available at: > https://builder.wildebeest.org/buildbot/#builders/10/builds/714 > > Buildbot URL: https://builder.wildebeest.org/buildbot/ > > Worker for this Build: fedora-s390x > > Build Reason: > Blamelist: Frank Ch. Eigler This cannot be blamed on Frank. Sorry. It looks like the build (compile) timed out. No idea if that was because the machine is overloaded or something else. I'll try to rebuild it. Cheers, Mark
[Bug debuginfod/27673] New: [debuginfod] Handle source requests for same buildid more efficiently
https://sourceware.org/bugzilla/show_bug.cgi?id=27673 Bug ID: 27673 Summary: [debuginfod] Handle source requests for same buildid more efficiently Product: elfutils Version: unspecified Status: NEW Severity: enhancement Priority: P2 Component: debuginfod Assignee: unassigned at sourceware dot org Reporter: vries at gcc dot gnu.org CC: elfutils-devel at sourceware dot org Target Milestone: --- Consider the following debuginfod usage session from gdb (reported in gdb PR27667): ... $ time gdb -batch -q ./mi-sym-info -ex start -ex "interpreter-exec mi 111-symbol-info-functions" > OUTPUT TRYING DEBUGINFO FOR: /data/gdb_versions/devel/system-supplied DSO at 0x77fca000 BUILDID: e77a560007595d823e2c5a34697c0b0ae1ffc767 TOOK: 0.84 seconds RESULT fd: 16 TRYING SOURCE: /home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/config/i386/sfp-exceptions.c BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83 TOOK: 0.601374 seconds RESULT fd: -2 TRYING SOURCE: /home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/libgcc2.c BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83 TOOK: 0.486450 seconds RESULT fd: -2 TRYING SOURCE: /home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/libgcc2.c BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83 TOOK: 0.493712 seconds RESULT fd: -2 TRYING SOURCE: /home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/libgcc2.c BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83 TOOK: 0.487850 seconds RESULT fd: -2 TRYING SOURCE: /home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/libgcc2.c BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83 TOOK: 0.488300 seconds RESULT fd: -2 TRYING SOURCE: /home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/libgcc2.c BUILDID: 7076274fc469c32fa8f4201f467447da3ad2b6be TOOK: 0.493761 seconds RESULT fd: -2 TRYING SOURCE: /home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/addtf3.c BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83 TOOK: 0.486202 seconds RESULT fd: -2 TRYING SOURCE: /home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/divtf3.c BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83 TOOK: 0.485941 seconds RESULT fd: -2 TRYING SOURCE: /home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/eqtf2.c BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83 TOOK: 0.477989 seconds RESULT fd: -2 TRYING SOURCE: /home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/extenddftf2.c BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83 TOOK: 0.487699 seconds RESULT fd: -2 TRYING SOURCE: /home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/fixtfdi.c BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83 TOOK: 0.488412 seconds RESULT fd: -2 TRYING SOURCE: /home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/fixtfsi.c BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83 TOOK: 0.496897 seconds RESULT fd: -2 TRYING SOURCE: /home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/floatditf.c BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83 TOOK: 0.492576 seconds RESULT fd: -2 TRYING SOURCE: /home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/floatsitf.c BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83 TOOK: 0.477747 seconds RESULT fd: -2 TRYING SOURCE: /home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/floatunsitf.c BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83 TOOK: 0.480452 seconds RESULT fd: -2 TRYING SOURCE: /home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/getf2.c BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83 TOOK: 0.487295 seconds RESULT fd: -2 TRYING SOURCE: /home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/letf2.c BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83 TOOK: 0.479365 seconds RESULT fd: -2 TRYING SOURCE: /home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/multf3.c BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83 TOOK: 0.486071
[Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently
https://sourceware.org/bugzilla/show_bug.cgi?id=27673 Frank Ch. Eigler changed: What|Removed |Added CC||fche at redhat dot com --- Comment #1 from Frank Ch. Eigler --- Tom, can you break down for me what debuginfod or client side change you think we need, from your gdb trace? -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently
https://sourceware.org/bugzilla/show_bug.cgi?id=27673 Frank Ch. Eigler changed: What|Removed |Added Status|NEW |WAITING --- Comment #2 from Frank Ch. Eigler --- The subject line "handle sources ... from same buildid ... more efficiently" suggests a caching opportunity. The debuginfod --fdcache-prefetch parameter is probably the gadget of interest here. The opensuse operators could experiment with a much larger than default figure. (You can test it locally by running debuinfod for yourself, against a local set of relevant -debug* RPMs.) -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently
https://sourceware.org/bugzilla/show_bug.cgi?id=27673 Tom de Vries changed: What|Removed |Added Status|WAITING |UNCONFIRMED Ever confirmed|1 |0 --- Comment #3 from Tom de Vries --- (In reply to Frank Ch. Eigler from comment #2) > The subject line "handle sources ... from same buildid ... more efficiently" > suggests a caching opportunity. The debuginfod --fdcache-prefetch parameter > is probably the gadget of interest here. The opensuse operators could > experiment with a much larger than default figure. (You can test it locally > by running debuinfod for yourself, against a local set of relevant -debug* > RPMs.) The time it takes for the client to see the response of the server to the request consist of: - time for request to travel to server (latency) - time for server to react to request - time for answer to travel back to client (latency again) I've looked at the documentation of the option fdcache-prefetch, and AFAIU this improves "time for server to react to request". Instead, the focus of this PR is to hide latency (since that's where the majority of the time is spent) by using more bandwidth. Proposal a: debuginfod: - when receiving a source request and ENOENT, send as reply the list of available files for the buildid client: - when receiving a list of available files for a buildid, store it and use it to reply to source requests related to the buildid. That is, if the file is not in the list, reply with -2. Otherwise, send a request to debuginfod, and expect it to succeed. Proposal b: debuginfod: - when receiving a source request, send a package with the sources for that buildid to the client. client: - when receiving a package with the sources for a buildid, store them and use them to reply to source requests related to the buildid. -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently
https://sourceware.org/bugzilla/show_bug.cgi?id=27673 --- Comment #4 from Frank Ch. Eigler --- > The time it takes for the client to see the response of the server to the > request consist of: > - time for request to travel to server (latency) > - time for server to react to request > - time for answer to travel back to client (latency again) > > I've looked at the documentation of the option fdcache-prefetch, and AFAIU > this improves "time for server to react to request". That's correct. > debuginfod: > - when receiving a source request and ENOENT, send as reply the list of > available files for the buildid > > client: > - when receiving a list of available files for a buildid, store it and > use it to reply to source requests related to the buildid. That is, > if the file is not in the list, reply with -2. Otherwise, send a > request to debuginfod, and expect it to succeed. Interesting. A more first-class solution could be a new webapi to enumerate source files: a "/buildid/HEXCODE/sourcelist" query that returns a structured piece of data. This can be computed by debuginfod fairly rapidly. The client could cache that and use it to drive a negative-cache hit on a subsequent source query. > Proposal b: > > debuginfod: > - when receiving a source request, send a package with the sources > for that buildid to the client. > > client: > - when receiving a package with the sources for a buildid, store them > and use them to reply to source requests related to the buildid. So this could be a "/buildid/HEXCODE/sources" query that returns a tarball of all sources related to a given buildid. This is challenging in principle because sources may not live in a single upstream package we can just relay verbatim. debuginfod may have to assemble a new one on the fly, kind of like gitweb's 'archive' buttons ... which are disabled by default for performance reasons. Worth a consideration I guess, but risky to deploy. By the way, a client also has another option: querying in parallel. If it knows all interesting file names, it can fork N threads and make N concurrent requests to debuginfod. The poor server may get larger bursts of load but total elapsed time should be better. And another option: if connection establishment / teardown are a bit part of the problem - and they can be with TLS - we could teach the client code to activate as much curl level http-keepalive as possible. So as long as a single debuginfod_client object were reused, it could avoid the TCP/TLS handshakes. (It MIGHT already be doing that.) -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently
https://sourceware.org/bugzilla/show_bug.cgi?id=27673 --- Comment #5 from Tom de Vries --- (In reply to Frank Ch. Eigler from comment #4) > > debuginfod: > > - when receiving a source request and ENOENT, send as reply the list of > > available files for the buildid > > > > client: > > - when receiving a list of available files for a buildid, store it and > > use it to reply to source requests related to the buildid. That is, > > if the file is not in the list, reply with -2. Otherwise, send a > > request to debuginfod, and expect it to succeed. > > Interesting. A more first-class solution could be a new webapi to > enumerate source files: a "/buildid/HEXCODE/sourcelist" query that > returns a structured piece of data. This can be computed by debuginfod > fairly rapidly. The client could cache that and use it to drive a > negative-cache hit on a subsequent source query. Agreed, it would be good to have this kind of functionality available more generically. Then the client (preferably the elfutils client itself using some setting, otherwise the part using the elfutils client) could upon the first source request send a sourcelist request in parallel, to have the latencies of the two requests overlap. -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently
https://sourceware.org/bugzilla/show_bug.cgi?id=27673 Tom de Vries changed: What|Removed |Added Blocks||27678 Referenced Bugs: https://sourceware.org/bugzilla/show_bug.cgi?id=27678 [Bug 27678] [gdb/symtab] Use debuginfod sourcelist api to test for file existence -- You are receiving this mail because: You are on the CC list for the bug.
