Re: ☠ Buildbot (GNU Toolchain): elfutils - failed test (failure) (master)
On Fri, May 27, 2022 at 10:34:19PM -0400, Frank Ch. Eigler wrote:
> Hi -
>
> n-metrics.sh: line 198: 27160 Aborted (core dumped) env
> LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d
> ${DB} -F -U -t0 -g0 -p $PORT1 L D F > vlog$PORT1 2>&1
> > [...]
> > Process 27160 (debuginfod) of user 994 killed by SIGABRT - ignoring (low
> > free space)
> > Lets see what happens on a rebuild.
>
> Did the machine run out of memory / storage?
Maybe, hard to tell from the logs. At the moment it looks like:
$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs3.7G 0 3.7G 0% /dev
tmpfs 3.7G 0 3.7G 0% /dev/shm
tmpfs 3.7G 17M 3.7G 1% /run
tmpfs 3.7G 0 3.7G 0% /sys/fs/cgroup
/dev/vda212G 11G 881M 93% /
/dev/vdb140G 8.3G 32G 21% /srv
tmpfs 748M 0 748M 0% /run/user/994
tmpfs 748M 0 748M 0% /run/user/1000
$ free -h
totalusedfree shared buff/cache available
Mem: 7.3G418M2.4G 16M4.5G6.6G
Swap: 511M 0B511M
So maybe / was slightly full. The buildbot itself runs under /srv/
I freed 2.5G from / and uninstalled abrt so it doesn't intercept
generating the core files.
Cheers,
Mark
☠ Buildbot (GNU Toolchain): elfutils - failed test (failure) (master)
A new failure has been detected on builder elfutils-centos-x86_64 while building elfutils. Full details are available at: https://builder.sourceware.org/buildbot/#builders/39/builds/33 Build state: failed test (failure) Revision: b8713b3fd0617415c76df8c9da70f8e2f26d3134 Worker: centos-x86_64 Build Reason: (unknown) Blamelist: Mark Wielaard Steps: - 0: worker_preparation ( success ) - 1: set package name ( success ) - 2: git checkout ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/2/logs/stdio - 3: autoreconf ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/3/logs/stdio - 4: configure ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/4/logs/stdio - 5: get version ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/5/logs/stdio - property changes: https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/5/logs/property_changes - 6: make ( warnings ) Logs: - stdio: https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/6/logs/stdio - warnings (2): https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/6/logs/warnings__2_ - 7: make check ( failure ) Logs: - stdio: https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/7/logs/stdio - test-suite.log: https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/7/logs/test-suite_log - 8: prep ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/8/logs/stdio - 9: fetch ['tests/*.trs', 'tests/*.log'] ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/9/logs/stdio - 10: fetch ['elfutils-*/_build/sub/tests/*.trs', 'elfut ( skipped ) Logs: - stdio: https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/10/logs/stdio - 11: fetch ['config.log'] ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/11/logs/stdio - 12: fetch ['elfutils-*/_build/sub/config.log'] ( skipped ) Logs: - stdio: https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/12/logs/stdio - 13: pass .bunsen.source.gitname ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/13/logs/stdio - 14: pass .bunsen.source.gitrepo ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/14/logs/stdio - 15: upload to bunsen ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/15/logs/stdio - 16: clean up ( success ) Logs: - stdio: https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/16/logs/stdio
Re: ☠ Buildbot (GNU Toolchain): elfutils - failed test (failure) (master)
Hi,
On Sat, May 28, 2022 at 09:15:30AM +, builder--- via Elfutils-devel wrote:
> A new failure has been detected on builder elfutils-centos-x86_64 while
> building elfutils.
>
> Full details are available at:
> https://builder.sourceware.org/buildbot/#builders/39/builds/33
>
> Build state: failed test (failure)
> Revision: b8713b3fd0617415c76df8c9da70f8e2f26d3134
> Worker: centos-x86_64
> Build Reason: (unknown)
> Blamelist: Mark Wielaard
>
> - 7: make check ( failure )
> Logs:
> - stdio:
> https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/7/logs/stdio
> - test-suite.log:
> https://builder.sourceware.org/buildbot/#builders/39/builds/33/steps/7/logs/test-suite_log
Well that is interesting, fails at the same point:
/srv/buildbot/worker/elfutils-centos-x86_64/build/tests/run-debuginfod-federation-metrics.sh:
line 198: 23793 Aborted env LD_LIBRARY_PATH=$ldpath
${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB} -F -U -t0 -g0 -p
$PORT1 L D F > vlog$PORT1 2>&1
Still no core :{
Retrying with ulimit -c unlimited...
O! It failed again, now with core
(gdb) where
#0 0x7f4d528be387 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1 0x7f4d528bfa78 in __GI_abort () at abort.c:90
#2 0x7f4d53dfae1f in mhd_panic_std (cls=, file=,
line=, reason=) at daemon.c:117
#3 0x7f4d53dfc69e in MHD_cleanup_connections
(daemon=daemon@entry=0xdf38e0) at daemon.c:1826
#4 0x7f4d53dfdf60 in MHD_run_from_select (daemon=daemon@entry=0xdf38e0,
read_fd_set=read_fd_set@entry=0x7f4d4c153c10,
write_fd_set=write_fd_set@entry=0x7f4d4c153c90,
except_fd_set=except_fd_set@entry=0x7f4d4c153d10) at daemon.c:2014
#5 0x7f4d53dfe1e9 in MHD_select (daemon=daemon@entry=0xdf38e0,
may_block=may_block@entry=1)
at daemon.c:2109
#6 0x7f4d53dfe3b2 in MHD_select_thread (cls=0xdf38e0) at daemon.c:2632
#7 0x7f4d53681ea5 in start_thread (arg=0x7f4d4c154700) at
pthread_create.c:307
#8 0x7f4d52986b0d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)
#3 0x7f4d53dfc69e in MHD_cleanup_connections
(daemon=daemon@entry=0xdf38e0) at daemon.c:1826
1826MHD_PANIC ("close failed\n");
(gdb) list
1821{
1822#ifdef WINDOWS
1823 SHUTDOWN (pos->socket_fd, SHUT_WR);
1824#endif
1825 if (0 != CLOSE (pos->socket_fd))
1826MHD_PANIC ("close failed\n");
1827}
1828 if (NULL != pos->addr)
1829free (pos->addr);
1830 free (pos);
Hohum. That does look like a bug in libmicrohttpd. Or do we manipulate
the socket_fd in any way?
Cheers,
Mark
Re: ☠ Buildbot (GNU Toolchain): elfutils - failed test (failure) (master)
On Sat, May 28, 2022 at 11:35:19AM +0200, Mark Wielaard wrote:
> Retrying with ulimit -c unlimited...
>
> O! It failed again, now with core
>
> (gdb) where
> #0 0x7f4d528be387 in __GI_raise (sig=sig@entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:55
> #1 0x7f4d528bfa78 in __GI_abort () at abort.c:90
> #2 0x7f4d53dfae1f in mhd_panic_std (cls=, file= out>,
> line=, reason=) at daemon.c:117
> #3 0x7f4d53dfc69e in MHD_cleanup_connections
> (daemon=daemon@entry=0xdf38e0) at daemon.c:1826
> #4 0x7f4d53dfdf60 in MHD_run_from_select (daemon=daemon@entry=0xdf38e0,
> read_fd_set=read_fd_set@entry=0x7f4d4c153c10,
> write_fd_set=write_fd_set@entry=0x7f4d4c153c90,
> except_fd_set=except_fd_set@entry=0x7f4d4c153d10) at daemon.c:2014
> #5 0x7f4d53dfe1e9 in MHD_select (daemon=daemon@entry=0xdf38e0,
> may_block=may_block@entry=1)
> at daemon.c:2109
> #6 0x7f4d53dfe3b2 in MHD_select_thread (cls=0xdf38e0) at daemon.c:2632
> #7 0x7f4d53681ea5 in start_thread (arg=0x7f4d4c154700) at
> pthread_create.c:307
> #8 0x7f4d52986b0d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
>
> (gdb)
> #3 0x7f4d53dfc69e in MHD_cleanup_connections
> (daemon=daemon@entry=0xdf38e0) at daemon.c:1826
> 1826 MHD_PANIC ("close failed\n");
> (gdb) list
> 1821 {
> 1822 #ifdef WINDOWS
> 1823SHUTDOWN (pos->socket_fd, SHUT_WR);
> 1824 #endif
> 1825if (0 != CLOSE (pos->socket_fd))
> 1826 MHD_PANIC ("close failed\n");
> 1827 }
> 1828if (NULL != pos->addr)
> 1829 free (pos->addr);
> 1830free (pos);
>
> Hohum. That does look like a bug in libmicrohttpd. Or do we manipulate
> the socket_fd in any way?
And in case it is useful for analysis:
(gdb) print *pos
$2 = {nextE = 0x0, prevE = 0x0, next = 0x0, prev = 0x0, nextX = 0x0, prevX =
0x0, daemon = 0xdf38e0,
headers_received = 0x0, headers_received_tail = 0x0, response = 0x0, pool =
0x7f4d242f2450,
client_context = 0x0, method = 0x0, url = 0x0, version = 0x0,
read_buffer = 0x7f4d242f2480 "\240\355\v$M\177", write_buffer = 0x0, last =
0x0, colon = 0x0,
addr = 0x7f4d242fa490, pid = 139968370415360, read_buffer_size = 16384,
read_buffer_offset = 0,
write_buffer_size = 0, write_buffer_send_offset = 0,
write_buffer_append_offset = 0,
remaining_upload_size = 0, response_write_position = 0,
continue_message_write_offset = 0,
addr_len = 28, last_activity = 599293, connection_timeout = 0, client_aware =
0, socket_fd = 96,
read_closed = 1, thread_joined = 0, in_idle = 0, epoll_state =
MHD_EPOLL_STATE_UNREADY,
state = MHD_CONNECTION_CLOSED, event_loop_info = MHD_EVENT_LOOP_INFO_CLEANUP,
responseCode = 0,
response_unready = 0, have_chunked_upload = 0, current_chunk_size = 0,
current_chunk_offset = 0,
read_handler = 0x7f4d53df7680 ,
write_handler = 0x7f4d53df78d0 ,
idle_handler = 0x7f4d53df83a0 ,
recv_cls = 0x7f4d53dfa030 , send_cls = 0x7f4d53df9ec0
,
tls_session = 0x0, protocol = 0, cipher = 0, tls_read_ready = 0, suspended =
0, resuming = 0}
$ rpm -q libmicrohttpd
libmicrohttpd-0.9.33-2.el7.x86_64
Cheers,
Mark
