Re: test-symlink fails on "lustre" file system

[Bruno Haible]

Assaf Gordon wrote:
> based on the code in test-symlink.h, perhaps lustre returns an 
> unexpected errno ?
> 
> line 69: ASSERT (func ("nowhere", BASE "dir/") == -1); 
> 
> line 70: ASSERT (errno == EEXIST || errno == EINVAL);

Yes, this is most likely the cause. Can you please insert
this line between line 69 and line 70

fprintf(stderr,"INSERTED errno=%d\n",errno); fflush(stderr);

then do
   make clean
   make
   make check

and report the output and contents of test-symlink.log ?

Lustre contains a number of kernel patches [1]; this can surely
modify the behaviour of system calls.

Bruno

[1] http://lustre.org/download/

error: GL_GENERATE_ERRNO_H does not appear in AM_CONDITIONAL

[Bruce Korb]

Hi, what is the correct fix? I've not rebuilt for a couple of years
and this is a new surprise. Thanks.

Bootstrap messages:

configure.ac:30: installing 'config/missing'
Makefile.am: installing './COPYING' using GNU General Public License v3 file
Makefile.am: Consider adding the COPYING file to the version control system
Makefile.am: for your code, to avoid questions about which license
your proje
ct uses
agen5/Makefile.am: installing 'config/depcomp'
parallel-tests: installing 'config/test-driver'
autoopts/Makefile.am:97: error: GL_GENERATE_ERRNO_H does not appear in
AM_CONDITI
ONAL
autoopts/Makefile.am:152: error: GL_GENERATE_STDDEF_H does not appear
in AM_CONDI
TIONAL
doc/Makefile.am:38: installing 'config/texinfo.tex'
bootstrap failure:  FAILURE 1:  /usr/local/bin/automake --gnu --add-missing
cp: '/u/bkorb/tools/gnulib/build-aux/config.rpath' and
'config/./config.rpath' ar
e the same file
cp: '/u/bkorb/tools/gnulib/m4/lib-link.m4' and 'config/./lib-link.m4'
are the sam
e file
fatal: Not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
gitlog-to-changelog: error closing pipe from git log --log-size
'--pretty=format:
%H:%ct  %an  <%ae>%n%n%s%n%b%n'

RE: test-symlink fails on "lustre" file system

[Oppe, Thomas C ERDC-RDE-ITL-MS Contractor]

Bruno,

The contents of the file

gettext-0.20.1/gettext-tools/gnulib-tests/test-symlink.log

are

INSERTED errno=2
test-symlink.h:71: assertion 'errno == EEXIST || errno == EINVAL' failed
FAIL test-symlink (exit status: 134)

Tom Oppe


-Original Message-
From: Bruno Haible [mailto:br...@clisp.org] 
Sent: Friday, September 6, 2019 5:43 AM
To: bug-gnulib@gnu.org
Cc: Assaf Gordon ; 37...@debbugs.gnu.org; Oppe,
Thomas C ERDC-RDE-ITL-MS Contractor 
Subject: Re: test-symlink fails on "lustre" file system

Assaf Gordon wrote:
> based on the code in test-symlink.h, perhaps lustre returns an 
> unexpected errno ?
> 
> line 69: ASSERT (func ("nowhere", BASE "dir/") == -1);
> 
> line 70: ASSERT (errno == EEXIST || errno == EINVAL);

Yes, this is most likely the cause. Can you please insert this line between
line 69 and line 70

fprintf(stderr,"INSERTED errno=%d\n",errno); fflush(stderr);

then do
   make clean
   make
   make check

and report the output and contents of test-symlink.log ?

Lustre contains a number of kernel patches [1]; this can surely modify the
behaviour of system calls.

Bruno

[1] Blockedhttp://lustre.org/download/Blocked




smime.p7s
Description: S/MIME cryptographic signature

Re: test-symlink fails on "lustre" file system

[Bruno Haible]

Thomas Oppe wrote:
> INSERTED errno=2
> test-symlink.h:71: assertion 'errno == EEXIST || errno == EINVAL' failed
> FAIL test-symlink (exit status: 134)

Thanks. I'm pushing this patch; it will fix the failure in this place.


2019-09-06  Bruno Haible  

symlink tests: Avoid test failure on Linux with Lustre file system.
Reported by Thomas C Oppe 
at .
* tests/test-symlink.h (test_symlink): Accept errno value ENOENT.

diff --git a/tests/test-symlink.h b/tests/test-symlink.h
index c556347..8fd55c0 100644
--- a/tests/test-symlink.h
+++ b/tests/test-symlink.h
@@ -67,7 +67,8 @@ test_symlink (int (*func) (char const *, char const *), bool 
print)
   ASSERT (errno == EEXIST);
   errno = 0;
   ASSERT (func ("nowhere", BASE "dir/") == -1);
-  ASSERT (errno == EEXIST || errno == EINVAL);
+  ASSERT (errno == EEXIST || errno == EINVAL
+  || errno == ENOENT /* Lustre FS on Linux */);
   ASSERT (close (creat (BASE "file", 0600)) == 0);
   errno = 0;
   ASSERT (func ("nowhere", BASE "file") == -1);

Re: error: GL_GENERATE_ERRNO_H does not appear in AM_CONDITIONAL

[Bruno Haible]

Hi Bruce,

> Hi, what is the correct fix? I've not rebuilt for a couple of years
> and this is a new surprise. Thanks.
> 
> Bootstrap messages:
> 
> autoopts/Makefile.am:97: error: GL_GENERATE_ERRNO_H does not appear in
> AM_CONDITIONAL
> autoopts/Makefile.am:152: error: GL_GENERATE_STDDEF_H does not appear
> in AM_CONDITIONAL

These two message indicate that the 'configure' script was not correctly
generated.

There are many possible causes. Before you revisit your build system,
I would
  - try the same thing from a pristine checkout (no leftover files from
a couple of years ago),
  - verify that the -I options passed to aclocal contain the directory
where errno_h.m4 and stddef_h.m4 have been copied to.

Bruno

[PATCH] findprog: Support searching in a specified path string

[Paul Smith]

find_prog_in_path() always uses the PATH value in the current
environment.  It can be very useful to search for programs on
a path without having to modify the environment first.

Provide find_in_path_str() which takes a path string to search.
If the path passed in is NULL, fall back to searching in the
environment's PATH value.

* lib/findprog.h (find_prog_in_path_str): Added.
(find_prog_in_path): Turn into a macro invoking the new function.
* lib/findprog.c (find_prog_in_path): Rename, add the extra
argument "in_path", and only look up PATH if it's NULL.
---
 lib/findprog.c | 12 +++-
 lib/findprog.h | 10 +-
 2 files changed, 16 insertions(+), 6 deletions(-)

diff --git a/lib/findprog.c b/lib/findprog.c
index 89fbe5fdf..94c938306 100644
--- a/lib/findprog.c
+++ b/lib/findprog.c
@@ -36,7 +36,7 @@
 
 
 const char *
-find_in_path (const char *progname)
+find_in_path_str (const char *progname, const char *in_path)
 {
 #if defined _WIN32 || defined __CYGWIN__ || defined __EMX__ || defined 
__DJGPP__
   /* Native Windows, Cygwin, OS/2, DOS */
@@ -54,17 +54,19 @@ find_in_path (const char *progname)
the current directory.  PATH is not used.  */
 return progname;
 
-  path = getenv ("PATH");
-  if (path == NULL || *path == '\0')
+  if (in_path == NULL)
+in_path = getenv ("PATH");
+
+  if (in_path == NULL || *in_path == '\0')
 /* If PATH is not set, the default search path is implementation
dependent.  */
 return progname;
 
   /* Make a copy, to prepare for destructive modifications.  */
 # if !IN_FINDPROG_LGPL
-  path = xstrdup (path);
+  path = xstrdup (in_path);
 # else
-  path = strdup (path);
+  path = strdup (in_path);
   if (path == NULL)
 /* Out of memory.  */
 return progname;
diff --git a/lib/findprog.h b/lib/findprog.h
index a354f67f7..5af3f1a90 100644
--- a/lib/findprog.h
+++ b/lib/findprog.h
@@ -29,7 +29,15 @@ extern "C" {
Because of the latter case, callers should use execlp/execvp, not
execl/execv on the returned pathname.
The returned string is freshly malloc()ed if it is != PROGNAME.  */
-extern const char *find_in_path (const char *progname);
+#define find_in_path(_p) find_in_path_str(_p, 0)
+
+
+/* Look up a program in the provided path.
+   Behaves the same as find_in_path() but looks in the provided path string
+   rather than in the PATH environment variable.  If path is null then use
+   the PATH environment variable.
+   The returned string is freshly malloc()ed if it is != PROGNAME.  */
+extern const char *find_in_path_str (const char *progname, const char *path);
 
 
 #ifdef __cplusplus
-- 
2.18.0

Re: error: GL_GENERATE_ERRNO_H does not appear in AM_CONDITIONAL

[Bruce Korb]

Hi Bruno,

On Fri, Sep 6, 2019 at 3:25 PM Bruno Haible  wrote:

> Hi Bruce,
> > Bootstrap messages:
> >
> > autoopts/Makefile.am:97: error: GL_GENERATE_ERRNO_H does not appear in
> > AM_CONDITIONAL
> > autoopts/Makefile.am:152: error: GL_GENERATE_STDDEF_H does not appear
> > in AM_CONDITIONAL
>
> These two message indicate that the 'configure' script was not correctly
> generated.
>

I had guessed that.

There are many possible causes. Before you revisit your build system,
> I would
>   - try the same thing from a pristine checkout (no leftover files from
> a couple of years ago),
>

Countless years ago, my build process became blowing away the build tree,
cloning the repo tree, populating it only with managed files, and *then*
go in and bootstrap/autoreconf/configure/make. The configure stuff
has bit-rotted.


>   - verify that the -I options passed to aclocal contain the directory
> where errno_h.m4 and stddef_h.m4 have been copied to.
>

It looks right:

libtoolize: linking file 'config/lt~obsolete.m4'
+shlib=run_autotools-177> doit /usr/local/bin/aclocal -I config
((note: I'm using a fancy PS4 here.
viz. PS4="+${prog}=\${FUNCNAME:-=}-\$LINENO> "))
+shlib=doit-34> test x/usr/local/bin/aclocal '!=' x:
+shlib=doit-36> echo 'RUN:  /usr/local/bin/aclocal -I config'
RUN:  /usr/local/bin/aclocal -I config
+shlib=doit-37> eval /usr/local/bin/aclocal -I config
++shlib=doit-37> /usr/local/bin/aclocal -I config
configure.ac:121: warning: AC_LANG_CONFTEST: no AC_LANG_SOURCE call
detected in body
../../lib/autoconf/lang.m4:193: AC_LANG_CONFTEST is expanded from...
../../lib/autoconf/general.m4:2661: _AC_LINK_IFELSE is expanded from...
../../lib/autoconf/general.m4:2678: AC_LINK_IFELSE is expanded from...
../../lib/m4sugar/m4sh.m4:639: AS_IF is expanded from...
../../lib/autoconf/general.m4:2031: AC_CACHE_VAL is expanded from...
config/ag_macros.m4:215: AG_LINK_SETJMP is expanded from...
config/ag_macros.m4:568: INVOKE_AG_MACROS is expanded from...
configure.ac:121: the top level
configure.ac:121: warning: AC_LANG_CONFTEST: no AC_LANG_SOURCE call
detected in body
.
Makefile.am: installing './COPYING' using GNU General Public License v3 file
Makefile.am: Consider adding the COPYING file to the version control
system
Makefile.am: for your code, to avoid questions about which license your
project uses
agen5/Makefile.am: installing 'config/depcomp'
parallel-tests: installing 'config/test-driver'
autoopts/Makefile.am:97: error: GL_GENERATE_ERRNO_H does not appear in
AM_CONDITIONAL
autoopts/Makefile.am:152: error: GL_GENERATE_STDDEF_H does not appear in
AM_CONDITIONAL
doc/Makefile.am:38: installing 'config/texinfo.tex'
+shlib=doit-37> die 'FAILURE 1:  /usr/local/bin/automake --gnu
--add-missing'
+shlib=die-27> echo 'bootstrap failure:  FAILURE 1:
/usr/local/bin/automake --gnu --add-missing'

$ ls config
00gnulib.m4 extern-inline.m4   lib-ld.m4   pkg.m4
absolute-header.m4  gendocs.sh lib-link.m4 snippet
ag_macros.m4gen-files.tar.gz   liboptschk.m4   snprintfv.m4
bootstrap   gen-list.txt   libopts.m4  stat-time.m4
bootstrap.local gnulib-cache.m4lib-prefix.m4   stddef_h.m4
bootstrap.shlib gnulib-common.m4   libtool.m4  stdnoreturn.m4
compile gnulib-comp.m4 ltmain.sh   test-driver
config.guessgnulib-tool.m4 lt~obsolete.m4  texinfo.tex
config.rpathguile.m4   ltoptions.m4time_h.m4
config.sub  host-cpu-c-abi.m4  ltsugar.m4  unlocked-io.m4
depcomp include_next.m4ltversion.m4warn-on-use.m4
errno_h.m4  install-defs.shmissing wchar_t.m4
extensions.m4   install-sh mk-shdefs.in

Re: [PATCH] findprog: Support searching in a specified path string

[Paul Smith]

I don't know if people would prefer to actually define two real
functions one of which just invokes the other, rather than use a
preprocessor macro to forward the old one to the new one.

I'm fine with it either way.


On Fri, 2019-09-06 at 19:10 -0400, Paul Smith wrote:
> find_prog_in_path() always uses the PATH value in the current
> environment.  It can be very useful to search for programs on
> a path without having to modify the environment first.
> 
> Provide find_in_path_str() which takes a path string to search.
> If the path passed in is NULL, fall back to searching in the
> environment's PATH value.
> 
> * lib/findprog.h (find_prog_in_path_str): Added.
> (find_prog_in_path): Turn into a macro invoking the new function.
> * lib/findprog.c (find_prog_in_path): Rename, add the extra
> argument "in_path", and only look up PATH if it's NULL.
> ---
>  lib/findprog.c | 12 +++-
>  lib/findprog.h | 10 +-
>  2 files changed, 16 insertions(+), 6 deletions(-)
> 
> diff --git a/lib/findprog.c b/lib/findprog.c
> index 89fbe5fdf..94c938306 100644
> --- a/lib/findprog.c
> +++ b/lib/findprog.c
> @@ -36,7 +36,7 @@
>  
>  
>  const char *
> -find_in_path (const char *progname)
> +find_in_path_str (const char *progname, const char *in_path)
>  {
>  #if defined _WIN32 || defined __CYGWIN__ || defined __EMX__ || defined 
> __DJGPP__
>/* Native Windows, Cygwin, OS/2, DOS */
> @@ -54,17 +54,19 @@ find_in_path (const char *progname)
> the current directory.  PATH is not used.  */
>  return progname;
>  
> -  path = getenv ("PATH");
> -  if (path == NULL || *path == '\0')
> +  if (in_path == NULL)
> +in_path = getenv ("PATH");
> +
> +  if (in_path == NULL || *in_path == '\0')
>  /* If PATH is not set, the default search path is implementation
> dependent.  */
>  return progname;
>  
>/* Make a copy, to prepare for destructive modifications.  */
>  # if !IN_FINDPROG_LGPL
> -  path = xstrdup (path);
> +  path = xstrdup (in_path);
>  # else
> -  path = strdup (path);
> +  path = strdup (in_path);
>if (path == NULL)
>  /* Out of memory.  */
>  return progname;
> diff --git a/lib/findprog.h b/lib/findprog.h
> index a354f67f7..5af3f1a90 100644
> --- a/lib/findprog.h
> +++ b/lib/findprog.h
> @@ -29,7 +29,15 @@ extern "C" {
> Because of the latter case, callers should use execlp/execvp, not
> execl/execv on the returned pathname.
> The returned string is freshly malloc()ed if it is != PROGNAME.  */
> -extern const char *find_in_path (const char *progname);
> +#define find_in_path(_p) find_in_path_str(_p, 0)
> +
> +
> +/* Look up a program in the provided path.
> +   Behaves the same as find_in_path() but looks in the provided path string
> +   rather than in the PATH environment variable.  If path is null then use
> +   the PATH environment variable.
> +   The returned string is freshly malloc()ed if it is != PROGNAME.  */
> +extern const char *find_in_path_str (const char *progname, const char *path);
>  
>  
>  #ifdef __cplusplus

Re: error: GL_GENERATE_ERRNO_H does not appear in AM_CONDITIONAL

[Bruno Haible]

Hi Bruce,

> It looks right:

Two other things to try:

* Try using the *newest* *releases* of Autoconf and Automake. This is the
  combination that is most reliable. It is not intended, but it may
  happen that older releases don't work.

* After running bootstrap, run the commands in order and check the results
  manually:
0. rm -rf autom4te.cache
1. aclocal -I ...
2. check the aclocal.m4
3. autoconf
4. check the configure
5. autoheader && touch config.h.in
6. check the config.h.in
7. automake --add-missing --copy
8. check the various Makefile.in_s

Bruno

Re: bitset: check memory allocation

[Akim Demaille]

Hi Paul,

Thanks a lot for the detailed answer!

> Le 5 sept. 2019 à 22:54, Paul Eggert  a écrit :
> 
> On 9/5/19 12:59 PM, Akim Demaille wrote:
> 
>>EBITSET_ELTS (src)
>> -= realloc (EBITSET_ELTS (src), newsize * sizeof (tbitset_elt 
>> *));
>> += xrealloc (EBITSET_ELTS (src), newsize * sizeof (tbitset_elt 
>> *));
> 
> This code is trying to shrink the bitset, and in the unlikely event that such 
> an attempt fails there's no need call xrealloc which will exit the whole 
> program; the code can continue to call realloc and if that fails, the code 
> can forge ahead with the unshrunk bitset.
> 
> There's a similar issue in the vector.c patch.

I never noticed realloc would return 0 yet preserve the allocated region, 
thanks!  This patch should address your concerns:

commit c9c23ad1d50cb5746dc2dd0265e0ae91c746b0b9
Author: Akim Demaille 
Date:   Thu Sep 5 11:36:39 2019 +0200

bitset: check memory allocation

Reported by 江 祖铭 (Zu-Ming Jiang).
With help from Paul Eggert.
https://lists.gnu.org/archive/html/bug-bison/2019-08/msg00016.html

* lib/bitset/table.c (tbitset_resize): When growing, use xrealloc
instead of realloc.
When shrinking, accept failures.
* lib/bitset/vector.c (vbitset_resize): Likewise.

diff --git a/ChangeLog b/ChangeLog
index fbf95966d..08b2313cb 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,14 @@
+2019-09-06  Akim Demaille  
+
+   bitset: check memory allocation
+   Reported by 江 祖铭 (Zu-Ming Jiang).
+   With help from Paul Eggert.
+   https://lists.gnu.org/archive/html/bug-bison/2019-08/msg00016.html
+   * lib/bitset/table.c (tbitset_resize): When growing, use xrealloc
+   instead of realloc.
+   When shrinking, accept failures.
+   * lib/bitset/vector.c (vbitset_resize): Likewise.
+
 2019-09-01  Bruno Haible  
 
gitsub.sh: Add support for shallow-cloning of subdirectories.
diff --git a/lib/bitset/table.c b/lib/bitset/table.c
index 07184d657..7691d9805 100644
--- a/lib/bitset/table.c
+++ b/lib/bitset/table.c
@@ -25,6 +25,7 @@
 #include 
 
 #include "obstack.h"
+#include "xalloc.h"
 
 /* This file implements expandable bitsets.  These bitsets can be of
arbitrary length and are more efficient than arrays of bits for
@@ -142,7 +143,7 @@ tbitset_resize (bitset src, bitset_bindex n_bits)
 
   bitset_windex size = oldsize == 0 ? newsize : newsize + newsize / 4;
   EBITSET_ELTS (src)
-= realloc (EBITSET_ELTS (src), size * sizeof (tbitset_elt *));
+= xrealloc (EBITSET_ELTS (src), size * sizeof (tbitset_elt *));
   EBITSET_ASIZE (src) = size;
 }
 
@@ -155,9 +156,13 @@ tbitset_resize (bitset src, bitset_bindex n_bits)
  the memory unless it is shrinking by a reasonable amount.  */
   if ((oldsize - newsize) >= oldsize / 2)
 {
-  EBITSET_ELTS (src)
+  void *p
 = realloc (EBITSET_ELTS (src), newsize * sizeof (tbitset_elt *));
-  EBITSET_ASIZE (src) = newsize;
+  if (p)
+{
+  EBITSET_ELTS (src) = p;
+  EBITSET_ASIZE (src) = newsize;
+}
 }
 
   /* Need to prune any excess bits.  FIXME.  */
diff --git a/lib/bitset/vector.c b/lib/bitset/vector.c
index 54f148d56..5e543283a 100644
--- a/lib/bitset/vector.c
+++ b/lib/bitset/vector.c
@@ -24,6 +24,8 @@
 #include 
 #include 
 
+#include "xalloc.h"
+
 /* This file implements variable size bitsets stored as a variable
length array of words.  Any unused bits in the last word must be
zero.
@@ -74,7 +76,7 @@ vbitset_resize (bitset src, bitset_bindex n_bits)
 
   bitset_windex size = oldsize == 0 ? newsize : newsize + newsize / 4;
   VBITSET_WORDS (src)
-= realloc (VBITSET_WORDS (src), size * sizeof (bitset_word));
+= xrealloc (VBITSET_WORDS (src), size * sizeof (bitset_word));
   VBITSET_ASIZE (src) = size;
 }
 
@@ -88,9 +90,13 @@ vbitset_resize (bitset src, bitset_bindex n_bits)
  the memory unless it is shrinking by a reasonable amount.  */
   if ((oldsize - newsize) >= oldsize / 2)
 {
-  VBITSET_WORDS (src)
+  void *p
 = realloc (VBITSET_WORDS (src), newsize * sizeof (bitset_word));
-  VBITSET_ASIZE (src) = newsize;
+  if (p)
+{
+  VBITSET_WORDS (src) = p;
+  VBITSET_ASIZE (src) = newsize;
+}
 }
 
   /* Need to prune any excess bits.  FIXME.  */


> Also, a nit in nearby code: xcalloc (1, N) is better written as xzalloc (N).

How about this stylistic patch?

commit 44e2124e57689417228fc04ded0026dc53cbee57
Author: Akim Demaille 
Date:   Fri Sep 6 11:38:48 2019 +0200

bitset: style changes

* lib/bitset/vector.c (vbitset_resize): Factor computation.
* lib/bitset.c, lib/bitset/stats.c, lib/bitsetv.c: Prefer
xzalloc to xcalloc.
Suggested by Paul Eggert.

diff