Re: m4 1.4.14 esyscmd problems on FreeBSD

[Johan van Selst]

Johan van Selst wrote:
> Eric Blake wrote:
> > I'll try and get on a FreeBSD machine and see if I can reproduce this.
> > But my initial pre-release testing on BSD did not fail, so I will have to
> > figure out what is different between my environment and yours.
> I shall check it on different systems later (tomorrow probably; not
> much time now).

Oops, the machine that I previously used for FreeBSD 7.x testing has
been upgraded and is now running 8.x as well. I correct my previous
claim: The code actually works fine on FreeBSD 7.2 machines. It's also
fine on OpenBSD (4.7) and MacOS (Darwin 8.11). So it seems something was
changed / broken in FreeBSD 8.0.

For completeness, I also tested on a FreeBSD 9-CURRENT system and this
shows the same broken behaviour as the FreeBSD 8.0 release.


Johan


pgpaSceJZCqKC.pgp
Description: PGP signature

Re: gnulib module install

[Simon Josefsson]

k...@freefriends.org (Karl Berry) writes:

> What I mean - I wouldn't mind commiting gnulib stuff to VCS, but only
> if I can omit files generated by autoreconf.
>
> FWIW, that is exactly what I do for Texinfo.  It helps my contributors
> to not spend time dealing with gnulib, and it helps me because it avoids
> gnulib-skew differences between me and my contributors.
>
> Of course, you have to be careful not to commit (and to .vcignore) the
> files generated by make, such as lib/foo.h when lib/foo.in.h exists.
>
> Feel free to peruse the Texinfo source repo
> (https://savannah.gnu.org/cvs/?group=texinfo) if you're interested.
>
> >From reading the manual, I got the impression that it is strictly the
> either-or approach, 
>
> I think the manual (node "VCS Issues") could mention this "middle road"
> as a possibility.  As far as I can see, you are correct that it only
> describes the all-or-nothing approach now.
>
> If the idea acceptable to the other gnulib folks, I could try to write a
> patch for it.

Please do, I'm using the same approach as you do and I think it should
be supported.  One of my reasons is that bootstrap cannot handle
multiple gnulib instances in a project.  This middle road approach
supports that.

/Simon

Re: gnulib-tool feature request

[Simon Josefsson]

Sam Steingold  writes:

> On 3/1/10, Simon Josefsson  wrote:
>>
>> It would be nice it gnulib-tool could do it, but I have a hard time
>>  thinking how that would actually be implemented.  There are so many
>>  different ways you may want to organize your gnulib directories that
>>  having gnulib-tool support them all is probably going to overload the
>>  poor shell script.  Concrete ideas are always welcome, though...
>
> gnulib-tool accepts --source-base and --m4-base as destination for files.
> it should also accept --source-common and --m4-common which specify
> the location of modules which are in the "core" of the project, i.e., which
> can be assumed to be present.
> So I will write something like:
> gnulib-tool --source-base=src/gllib --m4-base=src/glm4 foo
> gnulib-tool --source-common=src/gllib --m4-common=src/glm4
>   --source-base=modules/syscalls/gllib --m4-base=modules/syscalls/glm4 bar
> and if bar depends on foo, then foo files will not be added to
> modules/syscalls/gllib
> and modules/syscalls/glm4, instead they will be found in src/gllib & src/glm4
> by the second invocation of gnulib-tool.

I don't think that approach works if modules/syscalls/ depends on a
gnulib module (e.g., chown or malloc) that needs to modify a system
header (i.e., unistd.h or stdlib.h) to work, and the system header is
already in src/.  Unfortunately, that is a quite common pattern in
gnulib.

Generally, the approach only seems to work if the gnulib modules used in
src/ are self-contained, i.e., enabling the module doesn't cause other
gnulib modules to behave differently.

Right now gnulib-tool doesn't know whether a module is "standalone" or
not -- but it could be encoded in the modules/ file.

So if you want to implement this approach, I think you need to add a new
field in the modules/ files 'Standalone: yes/no', and make gnulib-tool
only put modules with 'Standalone: yes' in the src/ directory.  Any
module that has 'Standalone: no', or any module with 'Standalone: yes'
but depends on any module with 'Standalone: no', must go into
modules/syscalls/ etc.  (Maintaining this information will be a pain
though.)

However, I'm not sure this will work out well.  I suspect the number of
gnulib modules that can be put into src/ and still work will be very
small.  A gnulib module that appear to be standalone may use some system
call that does not work well on some platform, and thus ends up
depending on that.  For example: malloc.  Then the otherwise standalone
module cannot be put in the core directory.

More ideas still welcome, or clarification if I'm missing something.  I
am experiencing problems in my projects over this gnulib design too (in
particular, slow ./configure.ac time because of duplciation, and code
duplication because all dependencies needs to be available in several
places) but haven't been able to come up with a solution that is
implementable.  Right now, I just let disk/CPU pay the price for this.
The runtime costs are quite small, if any, on any modern system.

/Simon

Re: m4 1.4.14 esyscmd problems on FreeBSD

[Bruno Haible]

Hi Johan,

Thanks for the added details.

> The code actually works fine on FreeBSD 7.2 machines. It's also
> fine on OpenBSD (4.7) and MacOS (Darwin 8.11). So it seems something was
> changed / broken in FreeBSD 8.0.

FreeBSD 8.0 has its own, new implementation of the posix_spawn* functions,
whereas on the other platforms the gnulib replacement is used.

> > This quickly identifies a problem:
> > 
> >   2143 gm4  CALL  sigprocmask(SIG_BLOCK,0x545a20,0)
> >   2143 gm4  RET   sigprocmask 0
> >   2143 gm4  CALL  vfork
> >   2144 gm4  RET   fork 0
> >   2144 gm4  CALL  sched_setscheduler(0,,0x800a0e918)
> >   2144 gm4  RET   sched_setscheduler -1 errno 1 Operation not permitted
> >   2144 gm4  CALL  exit(0x7f)
> >   2143 gm4  RET   vfork 2144/0x860
> >   2143 gm4  CALL  sigprocmask(SIG_UNBLOCK,0x545a20,0)
> >   2143 gm4  RET   sigprocmask 0

How come that sched_setscheduler is called? The only use of posix_spawn and
posix_spawnp that is made by m4 1.4.14 (see lib/execute.c and lib/pipe.c) is
with a flags value of POSIX_SPAWN_SETSIGMASK. Whereas in the native FreeBSD
posix_spawn[p] (as well as in the gnulib replacement),  sched_setscheduler
is only called if the flags word contains the POSIX_SPAWN_SETSCHEDULER bit.

FreeBSD's POSIX_SPAWN_SETSIGMASK is 0x20, FreeBSD's POSIX_SPAWN_SETSCHEDULER
is 0x08. Whereas the gnulib replacement value for POSIX_SPAWN_SETSIGMASK is
0x08. It seems gnulib's replacement value is in use when it should not. I'm
applying this fix.


2010-03-02  Bruno Haible  

spawn: Don't override the system defined values on FreeBSD 8.
* lib/spawn.in.h (POSIX_SPAWN_RESETIDS, POSIX_SPAWN_SETPGROUP,
POSIX_SPAWN_SETSIGDEF, POSIX_SPAWN_SETSIGMASK,
POSIX_SPAWN_SETSCHEDPARAM, POSIX_SPAWN_SETSCHEDULER): Don't redefine
if HAVE_POSIX_SPAWN is 1.
Reported by Johan van Selst  via Eric Blake.

*** lib/spawn.in.h.orig Tue Mar  2 11:53:02 2010
--- lib/spawn.in.h  Tue Mar  2 11:48:54 2010
***
*** 110,126 
  
  
  /* Flags to be set in the `posix_spawnattr_t'.  */
! #if @REPLACE_POSIX_SPAWN@
  /* Use the values from the system, for better compatibility.  */
  /* But this implementation does not support AIX extensions.  */
! # undef POSIX_SPAWN_FORK_HANDLERS
! #else
! # define POSIX_SPAWN_RESETIDS   0x01
! # define POSIX_SPAWN_SETPGROUP  0x02
! # define POSIX_SPAWN_SETSIGDEF  0x04
! # define POSIX_SPAWN_SETSIGMASK 0x08
! # define POSIX_SPAWN_SETSCHEDPARAM  0x10
! # define POSIX_SPAWN_SETSCHEDULER   0x20
  #endif
  /* A GNU extension.  Use the next free bit position.  */
  #define POSIX_SPAWN_USEVFORK \
--- 110,128 
  
  
  /* Flags to be set in the `posix_spawnattr_t'.  */
! #if !...@have_posix_spawn@
! # if @REPLACE_POSIX_SPAWN@
  /* Use the values from the system, for better compatibility.  */
  /* But this implementation does not support AIX extensions.  */
! #  undef POSIX_SPAWN_FORK_HANDLERS
! # else
! #  define POSIX_SPAWN_RESETIDS   0x01
! #  define POSIX_SPAWN_SETPGROUP  0x02
! #  define POSIX_SPAWN_SETSIGDEF  0x04
! #  define POSIX_SPAWN_SETSIGMASK 0x08
! #  define POSIX_SPAWN_SETSCHEDPARAM  0x10
! #  define POSIX_SPAWN_SETSCHEDULER   0x20
! # endif
  #endif
  /* A GNU extension.  Use the next free bit position.  */
  #define POSIX_SPAWN_USEVFORK \

Re: m4 1.4.14 esyscmd problems on FreeBSD

[Johan van Selst]

Hi Bruno,

Bruno Haible wrote:
> FreeBSD's POSIX_SPAWN_SETSIGMASK is 0x20, FreeBSD's POSIX_SPAWN_SETSCHEDULER
> is 0x08. Whereas the gnulib replacement value for POSIX_SPAWN_SETSIGMASK is
> 0x08. It seems gnulib's replacement value is in use when it should not. I'm
> applying this fix.

Thanks for your quick analysis and response. I can confirm that your
patch does the trick for me on all FreeBSD systems where I can my tests.

My thanks to Eric as well. I now have a fully functional m4 1.4.14:
 $ make check
 
 
 All 103 tests passed
 


Johan


pgpr9lXhrM1yj.pgp
Description: PGP signature

Re: gnulib-tool feature request

[Sam Steingold]

On 3/2/10, Simon Josefsson  wrote:
> Sam Steingold  writes:
>
> > On 3/1/10, Simon Josefsson  wrote:
>  >>
>  >> It would be nice it gnulib-tool could do it, but I have a hard time
>  >>  thinking how that would actually be implemented.  There are so many
>  >>  different ways you may want to organize your gnulib directories that
>  >>  having gnulib-tool support them all is probably going to overload the
>  >>  poor shell script.  Concrete ideas are always welcome, though...
>  >
>  > gnulib-tool accepts --source-base and --m4-base as destination for files.
>  > it should also accept --source-common and --m4-common which specify
>  > the location of modules which are in the "core" of the project, i.e., which
>  > can be assumed to be present.
>  > So I will write something like:
>  > gnulib-tool --source-base=src/gllib --m4-base=src/glm4 foo
>  > gnulib-tool --source-common=src/gllib --m4-common=src/glm4
>  >   --source-base=modules/syscalls/gllib --m4-base=modules/syscalls/glm4 bar
>  > and if bar depends on foo, then foo files will not be added to
>  > modules/syscalls/gllib
>  > and modules/syscalls/glm4, instead they will be found in src/gllib & 
> src/glm4
>  > by the second invocation of gnulib-tool.
>
>
> I don't think that approach works if modules/syscalls/ depends on a
>  gnulib module (e.g., chown or malloc) that needs to modify a system
>  header (i.e., unistd.h or stdlib.h) to work, and the system header is
>  already in src/.  Unfortunately, that is a quite common pattern in
>  gnulib.

this is precisely why I wrote that --macro-prefix patch!
it would be not necessary if the --*-common arguments are introduced
because then the macro prefix can be generated automatically.

>  Right now, I just let disk/CPU pay the price for this.
>  The runtime costs are quite small, if any, on any modern system.

1. how about embedded systems?

2. I find this approach to be highly unaesthetic.
One of the reasons we do FLOSS in our copious spare time is
the desire to _reduce_ the ugliness - and when clisp has to be distributed
with _several_ copies of most of gnu libc is ugly to the extreme.

-- 
Sam Steingold 

Re: [PATCH] Solaris ACL handling

[Eric Blake]

According to Ben Walton on 2/12/2010 6:32 AM:

Hi Ben,
> I'm building coreutils for OpenCSW (http://opencsw.org) and was seeing
> the test suite fail several tests with 'Operation not supported on
> tranport endpoint' messages.  After digging around a bit, I found that
> errno was EOPNOTSUPP in several ACL handling routines and that wasn't
> covered.

Sorry for the delayed review, but this looks correct.

> Gracefully handle EOPNOTSUPP in qcopy_acl and qset_acl.  These
> functions, as used in coreutils, were causing the test suite to fail
> on Solaris 8 i386 with ZFS-backed NFSv3 mounts.

I've applied your patch, after adding a ChangeLog entry.

-- 
Eric Blake   ebl...@redhat.com+1-801-349-2682
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: gnulib-tool feature request

[Simon Josefsson]

Sam Steingold  writes:

> On 3/2/10, Simon Josefsson  wrote:
>> Sam Steingold  writes:
>>
>> > On 3/1/10, Simon Josefsson  wrote:
>>  >>
>>  >> It would be nice it gnulib-tool could do it, but I have a hard time
>>  >>  thinking how that would actually be implemented.  There are so many
>>  >>  different ways you may want to organize your gnulib directories that
>>  >>  having gnulib-tool support them all is probably going to overload the
>>  >>  poor shell script.  Concrete ideas are always welcome, though...
>>  >
>>  > gnulib-tool accepts --source-base and --m4-base as destination for files.
>>  > it should also accept --source-common and --m4-common which specify
>>  > the location of modules which are in the "core" of the project, i.e., 
>> which
>>  > can be assumed to be present.
>>  > So I will write something like:
>>  > gnulib-tool --source-base=src/gllib --m4-base=src/glm4 foo
>>  > gnulib-tool --source-common=src/gllib --m4-common=src/glm4
>>  >   --source-base=modules/syscalls/gllib --m4-base=modules/syscalls/glm4 bar
>>  > and if bar depends on foo, then foo files will not be added to
>>  > modules/syscalls/gllib
>>  > and modules/syscalls/glm4, instead they will be found in src/gllib & 
>> src/glm4
>>  > by the second invocation of gnulib-tool.
>>
>>
>> I don't think that approach works if modules/syscalls/ depends on a
>>  gnulib module (e.g., chown or malloc) that needs to modify a system
>>  header (i.e., unistd.h or stdlib.h) to work, and the system header is
>>  already in src/.  Unfortunately, that is a quite common pattern in
>>  gnulib.
>
> this is precisely why I wrote that --macro-prefix patch!

Maybe I'm beginning to slowly catch up...

> it would be not necessary if the --*-common arguments are introduced
> because then the macro prefix can be generated automatically.

But I don't see how it would work because of module interdependency's.
I think there are two main cases:

1) There is only one configure.ac that runs both gnulib m4 scripts in
   src/ and in the modules/foo/ directories.  (I'm using this model in
   GNU Libidn, Shishi and GSS.)

   You want the modules/foo/ directory to --avoid all modules from src/.

   This will result in one config.h, and it will be used both for the
   gnulib system-replacement header file in src/ and in the modules/foo/
   copy.  But it won't work well because a module in modules/foo/ may
   demand changes to a system header in src/ that will break for
   modules/bar/.  For example, consider if modules/foo/ replaces system
   function FOO declared in src/stdlib.h because it needs some
   particular behaviour from FOO that's not widely needed.  Anything
   #include'ing the src/ header files will get the same replacement, and
   things will break because modules/bar/ doesn't link to modules/foo/'s
   gnulib library that provides rpl_FOO.

2) There are two configure.ac that has its own gnulib m4 scripts.  (I'm
   using this model in GnuTLS and GNU SASL.)

   You want the modules/foo/ directory to --avoid all modules from src/.

   This will result in two config.h files and two _different_ system
   replacement header files.  The system replacement header files will
   provide conflicting definitions.  For example, if sys_socket.h is in
   both src/ and modules/foo/ and modules/foo/ provides a declaration of
   struct sockaddr_storage, the #include_next that points to the src/
   copy also provides the same declaration, you'll get an error.

Which of these cases are you considering?  Or something else?

>>  Right now, I just let disk/CPU pay the price for this.
>>  The runtime costs are quite small, if any, on any modern system.
>
> 1. how about embedded systems?
>
> 2. I find this approach to be highly unaesthetic.
> One of the reasons we do FLOSS in our copious spare time is
> the desire to _reduce_ the ugliness - and when clisp has to be distributed
> with _several_ copies of most of gnu libc is ugly to the extreme.

Sure.  I'm not sure how to resolve it though, without shipping the
_entire_ glibc code together with all packages and make sure that the
glibc replacement is built and used by all the remaining code that may
need different parts of the glibc code.

/Simon

status of new 'c++defs' module?

[John W. Eaton]

Is there any reason not to commit the changes for the new c++defs
module?

Earlier, I wrote:

| How can one easily find all the places where the GNULIB_NAMESPACE tag
| is needed?  Is there some way we can get the compiler to help with
| this job?
| 
| The reason I liked the idea of having the gnulib headers automatically
| add the "using gnulib::FOO" directive was that it would require no
| change to my code.  One of the great things about gnulib with C is
| that it allows the system functions to be replaced when needed without
| having to change existing code.  Retrofitting a large project with
| namespace tags will be tedious at best, and likely to result in hard
| to find errors.
| 
| Is there some reason to not add using directives in the gnulib
| headers for each system function that is placed in the gnulib::
| namespace?
| 
| Oh, now I remember that doing this requires placing the system headers
| inside a namespace, and I guess that could cause some trouble.  But
| would you be interested in trying it?  I could do the testing and help
| with making the required changes.

Although I still think that it would be nice if we could find a way to
make gnulib work with C++ without having to add a namespace tag to
symbols, I can live with having to add the tag.  It is better than
having to try to avoid macro definitions in all source files that
might include gnulib headers.

If you would like to see more testing for the c++defs module before
you commit the changes, then I'd be happy to help with that, but I
could use a few pointers as to how to handle local changes to gnulib
in Octave's source files so that people who check out the Octave
sources from our Mercurial archive and the gnulib sources from the
gnulib git archive can use the patches you proposed.

Thanks,

jwe

Re: gnulib-tool feature request

[Sam Steingold]

On 3/2/10, Simon Josefsson  wrote:
>
>  > it would be not necessary if the --*-common arguments are introduced
>  > because then the macro prefix can be generated automatically.
>
>
> But I don't see how it would work because of module interdependency's.
>  I think there are two main cases:
>
>  1) There is only one configure.ac that runs both gnulib m4 scripts in
>src/ and in the modules/foo/ directories.  (I'm using this model in
>GNU Libidn, Shishi and GSS.)

this is not an option.
each module should be usable as is.
e.g., if you have clisp installed on your system, but, e.g., the
wildcard module is not supplied, you should be able to get the
wildcard module from the cvs (or tarball),
configure it, telling it which clisp installation to build against:
./configure --with-clisp=/usr/loca/bin/clisp
make && make install
and it is installed in your ~/.clisp/dynmod
and you can now use it like this:
$ clisp
> (require "wildcard")

Ergo: each module must have its own configure, which accepts --with-clisp
and uses build-aux &c based on that.

that's how it works now and that's how it should work in the future.

>  2) There are two configure.ac that has its own gnulib m4 scripts.  (I'm
>using this model in GnuTLS and GNU SASL.)
>
>You want the modules/foo/ directory to --avoid all modules from src/.
>
>This will result in two config.h files and two _different_ system
>replacement header files.  The system replacement header files will
>provide conflicting definitions.  For example, if sys_socket.h is in
>both src/ and modules/foo/ and modules/foo/ provides a declaration of
>struct sockaddr_storage, the #include_next that points to the src/
>copy also provides the same declaration, you'll get an error.

yes, alas, it appears that the headers would have to be duplicated.
however, the objects do not have to be.

>  > 2. I find this approach to be highly unaesthetic.
>  > One of the reasons we do FLOSS in our copious spare time is
>  > the desire to _reduce_ the ugliness - and when clisp has to be distributed
>  > with _several_ copies of most of gnu libc is ugly to the extreme.
>
> Sure.  I'm not sure how to resolve it though, without shipping the
>  _entire_ glibc code together with all packages and make sure that the
>  glibc replacement is built and used by all the remaining code that may
>  need different parts of the glibc code.

there is a more direct solution: make it illegal to use any OS other
than a recent linux.

-- 
Sam Steingold 

Re: Remove fts-lgpl?

[Eric Blake]

According to Ralf Wildenhues on 1/17/2010 2:40 AM:
>> I think you've nailed it - fts-lgpl isn't really being used anywhere.
>> We originally offered it as a way to port glibc's fts to platforms
>> that lacked fts altogether, but glibc's fts is so much worse than
>> gnulib's that it hasn't been worth trying to maintain.  Maybe we
>> should just delete the module, and require GPL?  Or maybe someone can
>> provide a better argument why library-safe directory recursion is
>> still an important goal, even if it is slower because it can't rely on
>> openat optimizations.
> 
> No further comments on this so far, so I'll propose this patch.
> 
> Cheers,
> Ralf
> 
> Remove fts-lgpl module.
> 
> * modules/fts-lgpl: Remove.
> * MODULES.html.sh (func_all_modules): Adjust.
> * check-module (find_included_lib_files): Adjust.
> * m4/fts.m4 (gl_FUNC_FTS_LGPL): Remove.

Digging through my backlog.  A couple day window waiting for complaints
has now turned into a month of silence, so I've now applied this patch.

-- 
Eric Blake   ebl...@redhat.com+1-801-349-2682
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: arg-nonnull.h & warn-on-use.h in build-aux

[Sam Steingold]

Sam Steingold wrote:
I updated gnulib in clisp and gnulib-tool installed arg-nonnull.h & 
warn-on-use.h in src/build-aux.


So, any chance these files could be restored to their proper place in gllib?

This location causes me huge problems because these files are not covered by 
sed_transform_lib_file and I would have to pull dirty tricks on them...


Thanks.

Re: arg-nonnull.h & warn-on-use.h in build-aux

[Sam Steingold]

On 3/2/10, Sam Steingold  wrote:
> Sam Steingold wrote:
>
> > I updated gnulib in clisp and gnulib-tool installed arg-nonnull.h &
> warn-on-use.h in src/build-aux.
> >
>
>  So, any chance these files could be restored to their proper place in
> gllib?

Alternatively, I would be reasonably happy with
sed -i -e 's/_GL_ARG_NONNULL/GL_ARG_NONNULL/' arg-nonnull.h
sed -i -e 's/_GL_WARN_ON_USE/GL_WARN_ON_USE/' warn-on-use.h
similar to GL_LINK_WARNING in link-warning.h.

thanks!

-- 
Sam Steingold 

Re: [PATCH] Solaris ACL handling

[Jim Meyering]

Eric Blake wrote:

> According to Ben Walton on 2/12/2010 6:32 AM:
>
> Hi Ben,
>> I'm building coreutils for OpenCSW (http://opencsw.org) and was seeing
>> the test suite fail several tests with 'Operation not supported on
>> tranport endpoint' messages.  After digging around a bit, I found that
>> errno was EOPNOTSUPP in several ACL handling routines and that wasn't
>> covered.
>
> Sorry for the delayed review, but this looks correct.
>
>> Gracefully handle EOPNOTSUPP in qcopy_acl and qset_acl.  These
>> functions, as used in coreutils, were causing the test suite to fail
>> on Solaris 8 i386 with ZFS-backed NFSv3 mounts.
>
> I've applied your patch, after adding a ChangeLog entry.

Thanks for handling that.