[Bug debuginfod/25366] debuginfod scan: more progress/status

2020-11-11 Thread fche at redhat dot com via Elfutils-devel
https://sourceware.org/bugzilla/show_bug.cgi?id=25366

Frank Ch. Eigler  changed:

   What|Removed |Added

 Status|NEW |RESOLVED
 Resolution|--- |FIXED

--- Comment #1 from Frank Ch. Eigler  ---
elfutils 0.182 came out with several new scanning / grooming progress metrics

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[PATCH] debuginfod-find: Be a bit less verbose with -v

2020-11-11 Thread Mark Wielaard
debuginfod-find -v enables a progressfn that prints the Progress every
time the callback is called. For slow transfers or big downloads this
can be really verbose (hundreds a times a second). Slow it down a bit,
so it only prints the progress at most 5 times a second.

Signed-off-by: Mark Wielaard 
---
 debuginfod/ChangeLog |  5 +
 debuginfod/debuginfod-find.c | 24 +++-
 2 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog
index a02643e1..d4face2d 100644
--- a/debuginfod/ChangeLog
+++ b/debuginfod/ChangeLog
@@ -1,3 +1,8 @@
+2020-11-11  Mark Wielaard  
+
+   * debuginfod-find.c (progressfn): Use clock_gettime to print Progress
+   at most 5 times a second.
+
 2020-11-01  Érico N. Rolim  
 
* debuginfod-client.c (debuginfod_init_cache): Use ACCESSPERMS for
diff --git a/debuginfod/debuginfod-find.c b/debuginfod/debuginfod-find.c
index 88a460f8..5f9bccc3 100644
--- a/debuginfod/debuginfod-find.c
+++ b/debuginfod/debuginfod-find.c
@@ -24,6 +24,7 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
@@ -64,7 +65,28 @@ static int verbose;
 int progressfn(debuginfod_client *c __attribute__((__unused__)),
   long a, long b)
 {
-  fprintf (stderr, "Progress %ld / %ld\n", a, b);
+  static bool first = true;
+  static struct timespec last;
+  struct timespec now;
+  uint64_t delta;
+  if (!first)
+{
+  clock_gettime (CLOCK_MONOTONIC, &now);
+  delta = ((now.tv_sec - last.tv_sec) * 100
+  + (now.tv_nsec - last.tv_nsec) / 1000);
+}
+  else
+{
+  first = false;
+  delta = 25;
+}
+
+  /* Show progress the first time and then at most 5 times a second. */
+  if (delta > 20)
+{
+  fprintf (stderr, "Progress %ld / %ld\n", a, b);
+  clock_gettime (CLOCK_MONOTONIC, &last);
+}
   return 0;
 }
 
-- 
2.18.4



Re: [PATCH] debuginfod-find: Be a bit less verbose with -v

2020-11-11 Thread Frank Ch. Eigler via Elfutils-devel
Hi -

On Wed, Nov 11, 2020 at 09:31:38PM +0100, Mark Wielaard wrote:
> debuginfod-find -v enables a progressfn that prints the Progress every
> time the callback is called. [...]
> [...]
> -  fprintf (stderr, "Progress %ld / %ld\n", a, b);
> [...]

Another option is to use something close what the builtin env
DEBUGINFOD_PROGRESS=1 code does: print self-overwriting messages with
\r rather than \n.  That way many messages can come, but they don't
overpower the screen.  Really, the main reason I put in this
progressfn into debuginfod-find was to help test that API within the
testsuite.  Maybe now, we don't need that option to do anything but
set the env var, therby using the common code.

- FChE



[PATCH] debuginfod-client: Add debuginfod_set_verbose_fd and DEBUGINFOD_VERBOSE

2020-11-11 Thread Mark Wielaard
Introduce a new function debuginfod_set_verbose_fd which will produce
verbose output on a given file descriptor (STDERR_FILENO if the
environment variable DEBUGINFOD_VERBOSE is set) showing how the search
for a particular client query is going.

Example output:

debuginfod_find_debuginfo 1234567890
server urls "https://debuginfod.elfutils.org/ 
http://debuginfod.usersys.redhat.com:3632/";
checking build-id
checking cache dir /root/.cache/debuginfod_client
using timeout 90
init server 0 https://debuginfod.elfutils.org/
url 0 https://debuginfod.elfutils.org/buildid/1234567890/debuginfo
init server 1 http://debuginfod.usersys.redhat.com:3632/
url 1 http://debuginfod.usersys.redhat.com:3632/buildid/1234567890/debuginfo
query 2 urls in parallel
server response HTTP response code said error
url 0 The requested URL returned error: 404 Not Found
server response HTTP response code said error
url 1 The requested URL returned error: 404 Not Found
not found No such file or directory (err=-2)

Signed-off-by: Mark Wielaard 
---
 debuginfod/ChangeLog|  16 
 debuginfod/debuginfod-client.c  | 134 +---
 debuginfod/debuginfod-find.c|   4 +-
 debuginfod/debuginfod.h |   3 +
 debuginfod/libdebuginfod.map|   5 +-
 doc/ChangeLog   |   7 ++
 doc/debuginfod_find_debuginfo.3 |  17 
 7 files changed, 174 insertions(+), 12 deletions(-)

diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog
index d4face2d..5e507f0f 100644
--- a/debuginfod/ChangeLog
+++ b/debuginfod/ChangeLog
@@ -1,3 +1,19 @@
+2020-11-11  Mark Wielaard  
+
+   * debuginfod-client.c (debuginfod_set_verbose_fd): New function.
+   (struct debuginfod_client): Add verbose_fd.
+   (struct handle_data): Add errbuf.
+   (debuginfod_query_server): Produce verbose output when
+   debuginfod_client verbose_fd is set. Only clear old data and set
+   default_headers when any work is done. Always goto out when setting
+   rc to an error value. Use CURLOPT_ERRORBUFFER to get more error
+   output when verbose output is requested.
+   * debuginfod.h (DEBUGINFOD_VERBOSE_ENV_VAR): New.
+   (debuginfod_set_verbose_fd): Added.
+   * debuginfod-find.c (parse_opt): Set debuginfod_set_verbose_fd on -v.
+   * bdebuginfod.map (ELFUTILS_0.183): New section, add
+   debuginfod_set_verbose_fd.
+
 2020-11-11  Mark Wielaard  
 
* debuginfod-find.c (progressfn): Use clock_gettime to print Progress
diff --git a/debuginfod/debuginfod-client.c b/debuginfod/debuginfod-client.c
index ce1d819b..2bf1543a 100644
--- a/debuginfod/debuginfod-client.c
+++ b/debuginfod/debuginfod-client.c
@@ -56,6 +56,7 @@ int debuginfod_find_source (debuginfod_client *c, const 
unsigned char *b,
 int s, const char *f, char **p)  { return -ENOSYS; 
}
 void debuginfod_set_progressfn(debuginfod_client *c,
   debuginfod_progressfn_t fn) { }
+void debuginfod_set_verbose_fd(debuginfod_client *c, int fd) { }
 void debuginfod_set_user_data (debuginfod_client *c, void *d) { }
 void* debuginfod_get_user_data (debuginfod_client *c) { return NULL; }
 const char* debuginfod_get_url (debuginfod_client *c) { return NULL; }
@@ -115,6 +116,9 @@ struct debuginfod_client
  debuginfod_end needs to terminate. */
   int default_progressfn_printed_p;
 
+  /* File descriptor to output any verbose messages if > 0.  */
+  int verbose_fd;
+
   /* Can contain all other context, like cache_path, server_urls,
  timeout or other info gotten from environment variables, the
  handle data, etc. So those don't have to be reparsed and
@@ -158,6 +162,9 @@ struct handle_data
   /* URL queried by this handle.  */
   char url[PATH_MAX];
 
+  /* error buffer for this handle.  */
+  char errbuf[CURL_ERROR_SIZE];
+
   /* This handle.  */
   CURL *handle;
 
@@ -498,28 +505,51 @@ debuginfod_query_server (debuginfod_client *c,
   char *target_cache_tmppath = NULL;
   char suffix[PATH_MAX + 1]; /* +1 for zero terminator.  */
   char build_id_bytes[MAX_BUILD_ID_BYTES * 2 + 1];
+  int vfd = c->verbose_fd;
   int rc;
 
-  /* Clear the obsolete URL from a previous _find operation. */
-  free (c->url);
-  c->url = NULL;
-
-  add_default_headers(c);
+  if (vfd >= 0)
+{
+  dprintf (vfd, "debuginfod_find_%s ", type);
+  if (build_id_len == 0) /* expect clean hexadecimal */
+   dprintf (vfd, "%s", (const char *) build_id);
+  else
+   for (int i = 0; i < build_id_len; i++)
+ dprintf (vfd, "%02x", build_id[i]);
+  if (filename != NULL)
+   dprintf (vfd, " %s\n", filename);
+  dprintf (vfd, "\n");
+}
 
   /* Is there any server we can query?  If not, don't do any work,
  just return with ENOSYS.  Don't even access the cache.  */
   urls_envvar = getenv(server_urls_envvar);
+  if (vfd >= 0)
+dprintf (vfd, "server urls \"%s\"\n",
+urls_envvar != NULL ? urls_envvar : "");
   if (urls_envvar == NULL || urls_envvar

Re: [PATCH] debuginfod-find: Be a bit less verbose with -v

2020-11-11 Thread Mark Wielaard
Hi Frank,

On Wed, 2020-11-11 at 15:57 -0500, Frank Ch. Eigler wrote:
> On Wed, Nov 11, 2020 at 09:31:38PM +0100, Mark Wielaard wrote:
> > debuginfod-find -v enables a progressfn that prints the Progress
> > every
> > time the callback is called. [...]
> > [...]
> > -  fprintf (stderr, "Progress %ld / %ld\n", a, b);
> > [...]
> 
> Another option is to use something close what the builtin env
> DEBUGINFOD_PROGRESS=1 code does: print self-overwriting messages with
> \r rather than \n.  That way many messages can come, but they don't
> overpower the screen.  Really, the main reason I put in this
> progressfn into debuginfod-find was to help test that API within the
> testsuite.  Maybe now, we don't need that option to do anything but
> set the env var, therby using the common code.

It was indeed the specific testcase that made me keep the messages as
is. And it seems a good idea to have a code path that explicitly uses
the api calls instead of relying on the environment variable. Also I
found it a bit more difficult to combine the self-overwriting messages
with other verbose output. See my followup patch for producing verbose
output. I think the "per line" verbose/progress for debuginfod-find -v
works out well.

Cheers,

Mark