Re: [PATCH] Hurd: Enable ifunc by default

[Joseph Myers]

On Wed, 13 Jan 2021, Thomas Schwinge wrote:

> Hi!
> 
> Thanks (and sorry for the delay), pushed "Hurd: Enable ifunc by default"
> to master branch in commit e9cb89b936f831a02318d45fc4ddb06f7be55ae4, and
> cherry-picked into releases/gcc-10 branch in commit
> 92b131491c22eb4e4b663d226e9d97f1fd693063, releases/gcc-9 branch in commit
> 0313ce139f4ca3c96db9dc82125ec9e4a167a224, releases/gcc-8 branch in commit
> 975b0fa0f43e84bed3cb1b2b593132bc219f962c, see attached.

I'm not sure what toolchain component the underlying bug is in, but this 
GCC commit (verified in the releases/gcc-10 case) results in a glibc build 
failure for i686-gnu with build-many-glibcs.py.

https://sourceware.org/pipermail/libc-testresults/2021q1/007378.html

The error is:

/scratch/jmyers/glibc-bot/install/compilers/i686-gnu/lib/gcc/i686-glibc-gnu/11.0.0/../../../../i686-glibc-gnu/bin/ld:
 
/scratch/jmyers/glibc-bot/build/compilers/i686-gnu/glibc/i686-gnu/elf/librtld.os:
 in function `hurd_file_name_lookup_retry':
(.text+0x1e08e): undefined reference to `strncpy'
collect2: error: ld returned 1 exit status
Makefile:584: recipe for target 
'/scratch/jmyers/glibc-bot/build/compilers/i686-gnu/glibc/i686-gnu/elf/ld.so' 
failed
make[3]: *** 
[/scratch/jmyers/glibc-bot/build/compilers/i686-gnu/glibc/i686-gnu/elf/ld.so] 
Error 1

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: [RFC PATCH glibc 9/12] mach: Look for mach_i386.defs on x86_64 too

[Joseph Myers]

This commit broke the glibc build for i686-gnu.  There are errors of the 
form:

In file included from ../hurd/hurd/threadvar.h:23,
 from ../sysdeps/mach/hurd/mig-reply.c:20:
../sysdeps/mach/hurd/i386/tls.h:104:11: fatal error: mach/i386/mach_i386.h: No 
such file or directory
  104 | # include 
  |   ^~~

https://sourceware.org/pipermail/libc-testresults/2023q1/010861.html

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: [PATCH v3 0/6] The remaining x86_64-gnu patches

[Joseph Myers]

On Sat, 29 Apr 2023, Sergey Bugaev via Libc-alpha wrote:

> If these patches are pushed, it should be possible for anyone to build
> x86_64-gnu glibc just out of Git master, without having to dig through
> the mailing list archive for uncommited patches.

In that case I think there should be a patch to build-many-glibcs.py to 
add an x86_64-gnu configuration, as well as updates to NEWS and README to 
reflect the new support for such a configuration.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: [PATCH v3 0/6] The remaining x86_64-gnu patches

[Joseph Myers]

On Tue, 2 May 2023, Samuel Thibault via Libc-alpha wrote:

> Joseph Myers, le mar. 02 mai 2023 14:03:16 +, a ecrit:
> > On Sat, 29 Apr 2023, Sergey Bugaev via Libc-alpha wrote:
> > 
> > > If these patches are pushed, it should be possible for anyone to build
> > > x86_64-gnu glibc just out of Git master, without having to dig through
> > > the mailing list archive for uncommited patches.
> > 
> > In that case I think there should be a patch to build-many-glibcs.py to 
> > add an x86_64-gnu configuration,
> 
> I have it pending, just waiting for gcc 13.

It's enough for the x86_64-gnu support to be in GCC mainline when the 
patch to build-many-glibcs.py goes in; it doesn't need to be in the 
release branch (though being in the release branch may be helpful for 
users).

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: [PATCH v3 0/6] The remaining x86_64-gnu patches

[Joseph Myers]

On Tue, 2 May 2023, Sergey Bugaev via Libc-alpha wrote:

> It would also probably make sense to mention my other changes, of
> which there have been many, in the NEWS (would a simple "many fixes
> and improvements in the Hurd port" suffice?)

That may well be an appropriate way to describe them (if you haven't 
opened individual bug reports in Bugzilla; if a bug is reported in 
Bugzilla, then marked FIXED after the commit with the target milestone set 
to the first mainline release with the fix, it will automatically be 
listed in the list of fixed bugs in NEWS).

> Am I supposed to write a patch for this, or will somebody else do it?

The general expectation is that someone adding a significant new feature 
also does the NEWS update, and someone adding a new ABI does the 
build-many-glibcs.py update.

> Will adding it to build-many-glibcs.py automatically enable some sort
> of server-side CI for this configuration?

My build-many-glibcs.py bots automatically run the compilation parts of 
the testsuite for all configurations included in that script.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: [hurd,commited] hurd: Enable x86_64 build script

[Joseph Myers]

I'm observing that build-many-glibcs.py runs can end up with modifications 
to the source directory (which later cause problems with updating the 
glibc checkout):

diff --git a/sysdeps/mach/hurd/bits/errno.h b/sysdeps/mach/hurd/bits/errno.h
index 069865189f..3b6363568d 100644
--- a/sysdeps/mach/hurd/bits/errno.h
+++ b/sysdeps/mach/hurd/bits/errno.h
@@ -4,11 +4,11 @@
  libc-symbols.h
  mach/message.h
  mach/kern_return.h
- mach/i386/kern_return.h
+ mach/x86_64/kern_return.h
  mach/port.h
  mach/boolean.h
- mach/i386/boolean.h
- mach/i386/vm_types.h
+ mach/x86_64/boolean.h
+ mach/x86_64/vm_types.h
  stdint.h
  ../stdlib/stdint.h
  ../bits/libc-header-start.h

What this suggests to me is that the contents generated for 
sysdeps/mach/hurd/bits/errno.h depend on the system for which glibc is 
configured - which is never appropriate for a file in the source 
directory.  Either you need separate files for i386 and x86_64, or the 
contents need to avoid depending on the architecture, or the file needs to 
become purely a build-tree (and installed) file and not checked in.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: [RFC PATCH 10/10] hurd: Regenerate errno.h

[Joseph Myers]

On Wed, 17 May 2023, Sergey Bugaev via Libc-alpha wrote:

> Signed-off-by: Sergey Bugaev 
> ---
> My build keeps insisting that stdc-predef.h should be next to sysdeps and
> sys/cdefs.h; maybe it's right? If not, how do I stop it from changing?

I suggest changing the generator code not to put this list in the file at 
all.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: [RFC PATCH 06/10] hurd: Make sure to not use tcb->self

[Joseph Myers]

I suspect this of causing linknamespace test failures:

Contents of conform/POSIX2008/pthread.h/linknamespace.out:

[initial] pthread_create -> [libpthread.a(pt-create.o)] __pthread_setup -> 
[libpthread.a(pt-setup.o)] hurd_thread_self

(On x86_64 there's also a localplt test failure: "Extra PLT reference: 
libc.so: hurd_thread_self".  In addition, x86_64 has a c++-types-check 
failure.  Thus, neither Hurd configuration has clean results for 
compilation tests from build-many-glibcs.py at present.)

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: [RFC PATCH 06/10] hurd: Make sure to not use tcb->self

[Joseph Myers]

On Thu, 18 May 2023, Sergey Bugaev via Libc-alpha wrote:

> Hello,
> 
> On Thu, May 18, 2023 at 9:55 PM Joseph Myers  wrote:
> >
> > I suspect this of causing linknamespace test failures:
> >
> > Contents of conform/POSIX2008/pthread.h/linknamespace.out:
> >
> > [initial] pthread_create -> [libpthread.a(pt-create.o)] __pthread_setup -> 
> > [libpthread.a(pt-setup.o)] hurd_thread_self
> >
> > (On x86_64 there's also a localplt test failure: "Extra PLT reference:
> > libc.so: hurd_thread_self".  In addition, x86_64 has a c++-types-check
> > failure.  Thus, neither Hurd configuration has clean results for
> > compilation tests from build-many-glibcs.py at present.)
> 
> Thank you, and sorry :|
> 
> Do I understand it right that this is because of hurd_thread_self
> being publicly exported and interposable? A quick grep shows that
> nothing else inside glibc was using hurd_thread_self indeed.
> 
> Would the right way to resolve this be introducing a hidden
> __hurd_thread_self and using that? We could also make it

Yes.

Strictly there are two separate issues: (a) calling an exported symbol 
should be done via a hidden alias, to avoid going via the PLT, and (b) 
functions in a standard namespace should not call names in the user's 
namespace, which requires using a name such as __hurd_thread_self instead 
(which should also be marked hidden to make the call optimally efficient).

> What's the C++ type check failure? Surely this patch (nor
> 2f8ecb58a59eb82c43214d000842d99644a662d1 "hurd: Fix x86_64
> _hurd_tls_fork") has modified any public headers?

The C++ type check failure was already present before this patch.

--- sysdeps/mach/hurd/x86_64/c++-types.data 2023-05-02 09:14:30.246903708 
+
+++ -   2023-05-18 02:08:06.184068438 +
@@ -1 +1 @@
-blkcnt64_t:x
+blkcnt64_t:l
@@ -8 +8 @@
-dev_t:j
+dev_t:m
@@ -10 +10 @@
-fsblkcnt64_t:y
+fsblkcnt64_t:m
@@ -12 +12 @@
-fsfilcnt64_t:y
+fsfilcnt64_t:m
@@ -14 +14 @@
-fsid_t:y
+fsid_t:m
@@ -17 +17 @@
-ino64_t:y
+ino64_t:m
@@ -21 +21 @@
-int64_t:x
+int64_t:l
@@ -23 +23 @@
-intptr_t:i
+intptr_t:l
@@ -25 +25 @@
-loff_t:x
+loff_t:l
@@ -27,2 +27,2 @@
-nlink_t:j
-off64_t:x
+nlink_t:m
+off64_t:l
@@ -43,4 +43,4 @@
-pthread_t:i
-quad_t:x
-register_t:i
-rlim64_t:y
+pthread_t:l
+quad_t:l
+register_t:l
+rlim64_t:m
@@ -49 +49 @@
-size_t:j
+size_t:m
@@ -51 +51 @@
-ssize_t:i
+ssize_t:l
@@ -60 +60 @@
-u_int64_t:y
+u_int64_t:m
@@ -64 +64 @@
-u_quad_t:y
+u_quad_t:m

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: [RFC PATCH 06/10] hurd: Make sure to not use tcb->self

[Joseph Myers]

On Fri, 19 May 2023, Sergey Bugaev via Libc-alpha wrote:

> 'foo' is a public symbol, to be used by the user, and also
> interposable by the user. Always called via PLT (except from inside
> the user's code when redefined inside the executable, in which case
> the compiler/linker will see that no PLT is needed), and should not be
> called from inside glibc.

Should not be called from within glibc via the PLT within the same 
library.

It's OK for foo to be called from bar if foo is part of all the standards 
that contain bar.  But in that case, if the call is from the same library, 
*_hidden_def / *_hidden_proto should be used to avoid calling via the PLT.

If foo is not part of all the standards that contain bar, then glibc code 
for bar should use __foo instead to avoid namespace issues, especially for 
static linking.

If __foo is needed by executables, *_nonshared.a or other shared 
libraries, then it also needs to be exported from the library it's in (in 
which case PLT avoidance is also relevant for __foo, so *_hidden_* should 
be used for __foo).

If __foo is only used within a single library and doesn't need to be 
exported, it's OK to declare it directly with attribute_hidden rather than 
using *_hidden_* to create __GI___foo as an alias.  (In this case, if you 
forget to use attribute_hidden, you won't get test failures - the symbol 
in fact won't get exported, because only symbols explicitly listed in the 
version maps get exported, and so it won't get a PLT entry.  But in some 
cases, code generation is more efficient if the compiler knows that the 
symbol is hidden.  So when you're calling an unexported symbol in the same 
library, it's still desirable for it to be declared as hidden in a way 
visible to the compiler.)  The more complicated hidden_proto / hidden_def 
approach also works for unexported symbols used within a single library, 
it's just more complicated than necessary in that case.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

Please note (as a reminder from past discussions) that the Hurd libpthread 
will need to be included as part of glibc, much like NPTL is a 
fully-integrated part of glibc - not a separate package (support for 
add-ons has been removed from glibc).  (I'd also suggest that it *not* use 
a top-level directory called simply "libpthread" or similar without 
mention of Hurd, as that would be too confusing to people looking at the 
source tree for pthreads for other platforms.)

Also as per previous discussions: Hurd port maintainers can put in changes 
to Hurd-specific files at any time (regardless of release freeze state) 
until the port is actually working.  All the usual coding standards of 
course apply, and changes to files that might affect non-Hurd may need 
review and not be appropriate at some times, depending on freeze state and 
the risks associated with such patches.

In my view, build-many-glibcs.py support for Hurd would be appropriate 
even before the Hurd port builds (and certainly before it cleanly passes 
the compilation tests).  That support will definitely need patch review.

Before Hurd support is fully integrated in glibc, I'd encourage having a 
branch *in the glibc repository* that contains such support, so we can 
more readily see what the changes yet to be merged are (and possibly 
comment on issues that will need addressing when integrating them in 
glibc).

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

Incidentally, Samuel did post the Hurd TLS support three years ago 
.  Roland's 
review comments should of course be considered, though the current Hurd 
port maintainers are free to decide they disagree with particular aspects 
of those comments.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Thu, 18 Jan 2018, Samuel Thibault wrote:

> > Before Hurd support is fully integrated in glibc, I'd encourage having a 
> > branch *in the glibc repository* that contains such support, so we can 
> > more readily see what the changes yet to be merged are (and possibly 
> > comment on issues that will need addressing when integrating them in 
> > glibc).
> 
> I can pile-commit what we currently have, with all the XXXs, FIXMEs,
> etc., if that can help the glibc side.

I think having such a branch (rebased from time to time, especially as 
Hurd changes make their way into master or get cleaned up for issues 
raised, so it doesn't fall too far behind master) would be helpful so we 
can see the changes and quite likely point out global issues to be 
addressed throughout.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Thu, 18 Jan 2018, Samuel Thibault wrote:

> Joseph Myers, on jeu. 18 janv. 2018 13:47:42 +, wrote:
> > (I'd also suggest that it *not* use a top-level directory called
> > simply "libpthread" or similar without mention of Hurd, as that would
> > be too confusing to people looking at the source tree for pthreads for
> > other platforms.)
> 
> Sure :)
> 
> kFreeBSD/GNU used fbtl (FreeBSD Thread Library), perhaps htl? (Hurd
> Thread Library)

Seems reasonable to me.

Longer-term, once the support is in, it would be good to know how the API 
exposed for pthreads on Hurd compares to the NPTL API.  Right now, we have 
very little documentation for threads, but in principle it would be good 
to have thorough documentation for the pthreads API, and for the API to be 
the same between different OSes rather than having some interfaces only 
supported on some OSes.  (Then most of pthread.h could be shared between 
the different implementations - duplicating all the function declarations 
and associated feature test macro conditionals for them is not ideal.)

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Thu, 18 Jan 2018, Samuel Thibault wrote:

> Coding standards can be worked on by anybody, this is really something
> that bug-hurd people can unload us from.

Which is also something that having a branch with the patches is helpful 
for - it allows glibc people to look over them and point out coding 
standards issues to be addressed globally by people less familiar with the 
conventions used in glibc.  (Though when those issues are addressed the 
fixes should be integrated into each patch as part of a rebase, rather 
than simply adding another patch to the branch that fixes coding standards 
issues with the previous patches.)

(A branch close to current master also provides a basis for anyone working 
on build-many-glibcs.py support for Hurd, if you don't already have such 
support among your glibc patches.)

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Thu, 18 Jan 2018, Samuel Thibault wrote:

> Joseph Myers, on jeu. 18 janv. 2018 15:34:56 +, wrote:
> > On Thu, 18 Jan 2018, Samuel Thibault wrote:
> > > Coding standards can be worked on by anybody, this is really something
> > > that bug-hurd people can unload us from.
> > 
> > Which is also something that having a branch with the patches is helpful 
> > for -
> 
> Well, we already have that on
> 
> git clone git.savannah.gnu.org:/srv/git/hurd/glibc.git/
> 
> with almost a hundred branches to be looked after...

A hundred branches with many different purposes is not something at all 
convenient for glibc people to look over!

> Not all of them are necessary for managing to build glibc. Some of them
> are just hacking, others are perhaps almost ready to commit, just
> missing changelogs & formatting. That's the triaging thing that takes
> time, and having to do all the work including changelogs & formatting
> makes it get lower in my global TODOlist.

I'm still arguing on bug-standards for removing the requirement for 
ChangeLog format (given a public distributed version control system), but 
that wouldn't remove the requirement for proper explanations of patches, 
just mean that explanations intended to be read together with the patches 
themselves would suffice, without needing to duplicate the low-level 
details of how the patches change each affected named entity.

> > (A branch close to current master also provides a basis for anyone working 
> > on build-many-glibcs.py support for Hurd, if you don't already have such 
> > support among your glibc patches.)
> 
> That's why I suggested just committing the required patches to a branch
> that we rebase regularly, so there's a master that does build.

Yes, that's what I'd like to see.  From the perspective of people wanting 
to do global changes to glibc I think the priorities would be (a) building 
cleanly for Hurd; (b) building for Hurd with the intended ABI exported 
from each library at each symbol version; (c) having ABI test baselines to 
verify the ABI stays as expected; (d) having all the rest of the 
compilation parts of the testsuite passing.  Even given just (a) I might 
look at setting up build-many-glibcs.py support.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Thu, 18 Jan 2018, Samuel Thibault wrote:

> That's why I meant work is needed to sort out what we want to show
> people. Which takes triaging time, back to square 0. I sent to the
> bug-hurd list the list of patches which was enough at some point to get
> glibc buildable, that's what should be worked on and what I plan to
> commit to a single branch on the upstream repo.

Thanks for the changes pushed to sthibaul/hurd-builds so far (I realise 
there will be more to get it into a buildable state, e.g. the actual 
libpthread implementation).

General observations on mainly coding style issues that should be cleaned 
up for actual submission (there are also obvious issues with changes to 
non-Hurd-specific files needing cleaning up / justifying, which I didn't 
list individually):

* Avoid __GNU__ conditionals in generic files (as per Roland's comments on 
the TLS patch).

* Use -2018 in copyright notices.

* There should be a descriptive comment on the first line of each file 
before the copyright notice.  Where you have such comments, make sure they 
don't say something is a generic version when actually it's a 
Hurd-specific version.

* Avoid FSF postal addresses in license notices; use a URL instead, as in 
existing files in glibc (using an https URL will avoid needing to be 
included in an http -> https fix for such URLs in future).

* No "Contributed by" notices in new files.

* verify.h should be LGPLv2.1+ for glibc use, but actually I think you 
should just use _Static_assert instead of verify.h, since the minimum 
compiler version for building glibc supports _Static_assert.

* sysdeps/mach/hurd/gai_misc.h needs a copyright / license notice (and is 
missing spaces before '(' in some places).

* In a few places, missing preprocessor indentation inside #if (as per 
,
 
should be used except for a multiple include guard round a whole file).

* Why are some symbols added in sysdeps/mach/hurd/bits/posix_opt.h with 
value 200112L, when 200809L corresponds to the POSIX version generally 
supported by glibc?

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Fri, 19 Jan 2018, Samuel Thibault wrote:

> Joseph Myers, on jeu. 18 janv. 2018 23:15:59 +, wrote:
> > Thanks for the changes pushed to sthibaul/hurd-builds so far (I realise 
> > there will be more to get it into a buildable state, e.g. the actual 
> > libpthread implementation).
> 
> What I have pushed is basically only missing the libpthread
> implementation, so you already have an idea of the minimal set of
> modifications to get something building (and IIRC essentially passing
> the testsuite).

I'd still like to have the libpthread implementation there (with a view to 
seeing if I can get build-many-glibcs.py working for Hurd with this branch 
- if the branch has sources that build for Hurd, I should have some chance 
of using 
https://git.savannah.gnu.org/cgit/hurd/incubator.git/tree/cross-gnu?h=cross-gnu/master
 
to figure out how to do the Hurd-specific pieces that will need adding to 
build-many-glibcs.py).

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Fri, 19 Jan 2018, Thomas Schwinge wrote:

> > - if the branch has sources that build for Hurd, I should have some chance 
> > of using 
> > https://git.savannah.gnu.org/cgit/hurd/incubator.git/tree/cross-gnu?h=cross-gnu/master
> >  
> > to figure out how to do the Hurd-specific pieces that will need adding to 
> > build-many-glibcs.py).
> 
> Beware that this script is from many years ago -- from times before my
> dear CodeSourcery colleagues taught be how to ;-) properly build
> cross-compilers.  But yes, cross-gnu did work back then.
> 
> Another thing to look into nowadays is David's (CCed) gnuxc,
> <87r3vaw4os.fsf@gmail.com">http://mid.mail-archive.com/87r3vaw4os.fsf@gmail.com> "Scripts to build
> a Hurd distro",  "GNU OS Cross-Compiler".
> (I have not yet looked into that one myself.)
> 
> And, the Guix/Hurd effort also is capable of cross-compilation, as far as
> I remember.  But probably cross-gnu/gnuxc will already give you enough
> pointers of what needs to be done.

Do you have advice on versions to use of mig / gnumach / hurd?

Currently, build-many-glibcs.py uses (by default) release versions of most 
components other than glibc itself, release branches for gcc / binutils.  
Should the most recent releases from ftp.gnu.org of mig / gnumach / hurd 
be used for building current glibc for Hurd, or is it preferable or 
necessary to use newer development versions of those components?

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Fri, 19 Jan 2018, Thomas Schwinge wrote:

> Hi!
> 
> On Fri, 19 Jan 2018 13:08:01 +0000, Joseph Myers  
> wrote:
> > On Fri, 19 Jan 2018, Thomas Schwinge wrote:
> > > > [build-many-glibcs.py]
> 
> > Do you have advice on versions to use of mig / gnumach / hurd?
> > 
> > Currently, build-many-glibcs.py uses (by default) release versions of most 
> > components other than glibc itself, release branches for gcc / binutils.  
> > Should the most recent releases from ftp.gnu.org of mig / gnumach / hurd 
> > be used for building current glibc for Hurd, or is it preferable or 
> > necessary to use newer development versions of those components?
> 
> The current releases should generally be fine, yes.

The source file sysdeps/mach/hurd/bits/errno.h is generated from sources 
including some headers from those components.  I don't know how often 
those may change in ways that affect that header, or what versions the 
current header corresponds to, but in any case it's a known issue that 
there are several generated files in the source tree that 
build-many-glibcs.py doesn't yet touch on checkout.

> (This reminds me that I wanted to publish new GNU Mach, MIG, Hurd,
> libpthread releases, but no need for you to wait for these, as far as I
> know.)

Regarding libpthread, I don't propose build-many-glibcs.py support for a 
separate libpthread component; what's relevant is whatever version goes on 
the sthibaul/hurd-builds branch of glibc (set up to build without the old 
add-ons mechanism), and in due course on master.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Fri, 19 Jan 2018, Thomas Schwinge wrote:

> Hi Joseph!
> 
> On Fri, 19 Jan 2018 00:34:42 +0000, Joseph Myers  
> wrote:
> > On Fri, 19 Jan 2018, Samuel Thibault wrote:
> > 
> > > Joseph Myers, on jeu. 18 janv. 2018 23:15:59 +, wrote:
> > > > Thanks for the changes pushed to sthibaul/hurd-builds so far (I realise 
> > > > there will be more to get it into a buildable state, e.g. the actual 
> > > > libpthread implementation).
> > > 
> > > What I have pushed is basically only missing the libpthread
> > > implementation, so you already have an idea of the minimal set of
> > > modifications to get something building (and IIRC essentially passing
> > > the testsuite).
> > 
> > I'd still like to have the libpthread implementation there (with a view to 
> > seeing if I can get build-many-glibcs.py working for Hurd with this branch 
> 
> Many thanks for your offer!  As far as I'm aware indeed nobody from the
> Hurd team has spent time on that yet.

This patch adds build-many-glibcs.py support for GNU Hurd.  It is
intended for master, where the builds of the i686-gnu configuration
would fail until sufficient support is merged to master, so completing
build-many-glibcs.py coverage of all glibc ABIs and making results
accurately reflect the broken state of builds for Hurd.  Using
sthibaul/hurd-builds branch, it reaches the glibc build, which then
falls over with errors starting:

In file included from mutex-init.c:19:0:
../mach/lock-intern.h: In function '__spin_lock_init':
../mach/lock-intern.h:41:13: error: 'LLL_INITIALIZER' undeclared (first use in 
this function)
   *__lock = LLL_INITIALIZER;
 ^~~
../mach/lock-intern.h:41:13: note: each undeclared identifier is reported only 
once for each function it appears in
../mach/lock-intern.h: In function '__spin_lock':
../mach/lock-intern.h:50:3: error: implicit declaration of function 'lll_lock' 
[-Werror=implicit-function-declaration]
   lll_lock (__lock, 0);
   ^~~~
In file included from mutex-init.c:20:0:
../mach/lowlevellock.h: At top level:
../mach/lowlevellock.h:21:10: fatal error: mach/gnumach.h: No such file or 
directory
 #include 
  ^~~~

Could Hurd people review how this handles building for Hurd, and
indicate whether the above errors indicate problems with these
changes, or simply incompleteness of the build support on the
sthibaul/hurd-builds branch at present?

2018-01-19  Joseph Myers  

* scripts/build-many-glibcs.py (Context.add_all_configs): Add
i686-gnu configurations.
(Context.run_builds): Include mig, gnumach and hurd in components
considered.
(Context.checkout): Add mig, gnumach and hurd to components.
(Context.checkout_tar): Add URL mappings for mig, gnumach and
hurd.
(Context.bot_cycle): Check for changes to mig, gnumach and hurd.
(Config.build): Install gnumach headers, build mig and install
hurd headers for 'gnu' OS.
(Config.install_gnumach_headers): New function.
(Config.install_hurd_headers): Likewise.
(Glibc.build_glibc): Do not use /usr for 'gnu' OS.  Specifiy MIG
when building for 'gnu' OS.

diff --git a/scripts/build-many-glibcs.py b/scripts/build-many-glibcs.py
index 8849574..6095584 100755
--- a/scripts/build-many-glibcs.py
+++ b/scripts/build-many-glibcs.py
@@ -193,6 +193,8 @@ class Context(object):
  '--with-fpu=vfpv3'])
 self.add_config(arch='hppa',
 os_name='linux-gnu')
+self.add_config(arch='i686',
+os_name='gnu')
 self.add_config(arch='ia64',
 os_name='linux-gnu',
 first_gcc_cfg=['--with-system-libunwind'])
@@ -461,13 +463,15 @@ class Context(object):
 old_versions = {}
 self.build_host_libraries()
 elif action == 'compilers':
-build_components = ('binutils', 'gcc', 'glibc', 'linux')
+build_components = ('binutils', 'gcc', 'glibc', 'linux', 'mig',
+'gnumach', 'hurd')
 old_components = ('gmp', 'mpfr', 'mpc')
 old_versions = self.build_state['host-libraries']['build-versions']
 self.build_compilers(configs)
 else:
 build_components = ('glibc',)
-old_components = ('gmp', 'mpfr', 'mpc', 'binutils', 'gcc', 'linux')
+ 

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

Incidentally, building for GNU/Linux (at least x86_64) is broken on the 
sthibaul/hurd-builds branch (I suspect, based on the error messages, that 
"t2.26/sched_param" was responsible, but didn't verify that).  (I was 
testing to make sure my build-many-glibcs.py changes didn't affect 
GNU/Linux builds and so are safe to apply to master.  Building for 
GNU/Linux is not of course a priority for the branch, but any such issues 
with changes to non-OS-specific code will need fixing for merging to 
master.)

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Fri, 19 Jan 2018, Zack Weinberg wrote:

> On Fri, Jan 19, 2018 at 10:11 AM, Joseph Myers  
> wrote:
> > On Fri, 19 Jan 2018, Thomas Schwinge wrote:
> >
> > The source file sysdeps/mach/hurd/bits/errno.h is generated from sources
> > including some headers from those components.  I don't know how often
> > those may change in ways that affect that header, or what versions the
> > current header corresponds to, but in any case it's a known issue that
> > there are several generated files in the source tree that
> > build-many-glibcs.py doesn't yet touch on checkout.
> 
> Note that there is no particularly good reason to have this file
> checked in.  It's generated with an awk script, and the files it's

I'm not sure if having a non-checked-in generated file override another 
file in the source tree works correctly, the way a checked-in sysdeps file 
does override other files in the source tree.  (sysdeps/gnu/errlist.c is 
another case of a sysdeps generated file - it needs to override 
stdio-common/errlist.c, and for that matter gets included by 
sysdeps/mach/hurd/errlist.c.)

If the search orderings work correctly - both for finding files during the 
build, and for installing the correct bits/errno.h header (and the 
build-tree errlist.c can be properly included from the source-tree 
sysdeps/mach/hurd/errlist.c) - then there shouldn't be any need for either 
sysdeps/mach/hurd/bits/errno.h or sysdeps/gnu/errlist.c to be checked in.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Wed, 24 Jan 2018, Samuel Thibault wrote:

> Hello,
> 
> Joseph Myers, on ven. 19 janv. 2018 17:23:29 +, wrote:
> > Could Hurd people review how this handles building for Hurd,
> 
> Yes, it looks good.

Thanks, I've committed it to master.

> > and indicate whether the above errors indicate problems with these
> > changes, or simply incompleteness of the build support on the
> > sthibaul/hurd-builds branch at present?
> 
> They are not related to the changes.  I have pushed fix updates against
> them.

Now I see errors about missing libc-lockP.h, and "implicit declaration of 
function '__spin_lock_solid'", and the discarded qualifiers errors you 
note below.

> There is just one missing thing: mig prototypes etc. are not perfect,
> and we thus have some nasty warnings about void * vs const void *. We'd
> have to use
> 
>  if self.os == 'gnu':
>  cfg_cmd += ['MIG=%s' % self.tool_name('mig')]
> +cfg_cmd += ['--disable-werror']
> 
> for now until we fix that.

A properly upstreamed port should not need --disable-werror.  If 
workarounds are needed until there's a fixed mig release (or whatever), 
they could take the form of explicit casts, or CFLAGS- += 
-Wno-error=discarded-qualifiers - disabling a particular warning being an 
error in a particular file is much better than globally disabling all 
warnings as errors.  (Strictly -Wno-error=discarded-qualifiers shouldn't 
be used unconditionally since that warning option is new in GCC 5 and 
glibc supports building with GCC 4.9, but in the current state of the Hurd 
port I think we can ignore that in Hurd-specific Makefiles.)

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Thu, 25 Jan 2018, Samuel Thibault wrote:

> Samuel Thibault, on mer. 24 janv. 2018 02:27:26 +0100, wrote:
> > Samuel Thibault, on mer. 24 janv. 2018 02:10:51 +0100, wrote:
> > > Joseph Myers, on ven. 19 janv. 2018 17:23:29 +, wrote:
> > > > and indicate whether the above errors indicate problems with these
> > > > changes, or simply incompleteness of the build support on the
> > > > sthibaul/hurd-builds branch at present?
> > > 
> > > They are not related to the changes.  I have pushed fix updates against
> > > them.
> > 
> > The next error is about inclusion of libc-lockP.h.  That's where I'll
> > have to import libpthread.
> 
> Which I have now done.  I see that it now fails on missing reference to
> __file_exec_paths.  That's indeed a new RPC which hasn't been released
> yet, so we'd need to use a git snapshot.

If a git version of Hurd (or some other component) is needed, it would be 
reasonable to make vcs-mainline the default version of Hurd until there's 
a new release.  Longer-term (once Hurd is working on master) I'd hope 
there would be a clear (and documented) notion of minimum compile-time and 
run-time Hurd versions, as there is for glibc using the Linux kernel, with 
any features of newer versions being used only conditionally, so that the 
most recent releases are generally sufficient to build and use glibc for 
Hurd.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

Regarding the libpthread / htl code, and getting it ready for inclusion in 
glibc master:

Obviously my general coding style comments at 
 apply equally 
to this code.  Apart from that:

* Remove htl/ChangeLog.  We don't have subdirectory ChangeLogs now.  
Everything goes in the top-level ChangeLog, with older ChangeLogs (that 
have actual ChangeLog content) being in the ChangeLog.old/ directory.  
When importing files / patches previously maintained elsewhere, I *do* 
think the ChangeLog header for those patches should name all authors who 
contributed to them (which may make for a long header with lots of authors 
listed).  (If the GNU Coding Standards change to stop requiring ChangeLog 
format and we decide to stop using it, we'll still need some convention 
for how to credit multi-author patches.)

* Remove htl/tests/.cvsignore.

* Support for building htl outside the glibc source tree (or with older 
glibc versions) should not be included.

* Files in htl/sysdeps should be moved into appropriate locations in the 
main sysdeps tree, just as was done with nptl/sysdeps when nptl ceased to 
be an add-on.

* You seem to have some custom system for building tests in htl/tests.  
Tests should be built and run using the normal glibc testsuite machinery 
(and should use ).

* You have four installed non-bits/ API headers: pthread.h 
pthread/pthread.h pthread/pthreadtypes.h semaphore.h.  NPTL has just 
pthread.h and semaphore.h.  Do you really need pthread/pthread.h and 
pthread/pthreadtypes.h as installed public API headers?  I'd expect the 
same two API headers as NPTL has.

* The bits/ convention is for headers that are (a) installed, (b) only for 
use by other installed headers, not for direct inclusion by users (and 
sometimes have #error conditionals to guard against direct inclusion).  
Within that convention, there's a newer bits/types/ convention for headers 
defining a single type, which is intended as a replacement for the older 
__need_* convention.  Now, this brings up three points regarding the htl 
code (and possibly other pieces in the Hurd port).

(i) You still have some headers that define or use __need_*; those should 
be changed to use bits/types/ headers (__need_* has generally been 
eliminated from glibc, except where it's defined before including *GCC* 
headers, not glibc ones).

(ii) Some of the bits/ headers in htl look like they are just defining a 
single type, so should actually be bits/types/ headers named according to 
the new convention.

(iii) bits/memory.h and bits/pt-atomic.h don't appear to be installed 
headers, meaning they should not have bits/ names.  Actually, uses of 
those headers should probably be changed to use glibc's existing atomic.h 
interfaces, and those htl-specific headers removed - if for some reason 
the existing atomic.h interfaces are insufficient, maybe those interfaces 
need to be extended.


The following would be desirable cleanups, but maybe for after the code is 
in master:

* I'd expect that NPTL tests (and likewise HTL tests) are largely tests of 
POSIX (or GNU extension) threading functionality, not of features specific 
to one threading implementation.  Thus, they should as far as possible be 
shared between the different threading implementations.  I don't know the 
best directory arrangements for achieving that.

* Hopefully C11 threads support will be added for 2.28 (didn't get 
reviewed in time for 2.27).  The existing patches put it in the nptl/ 
directory, but again, to the extent that it actually builds on generic 
pthreads functionality, the code and its tests should be shared as much as 
possible between NPTL and HTL.

* Likewise for the pthread.h and semaphore.h API headers - only parts 
genuinely specific to a given implementation should be split out into 
implementation-specific bits/ headers.  (Indeed, the NPTL semaphore.h is 
already in sysdeps/pthread/ not an NPTL-specific directory - do you really 
need a different one for HTL?)

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Mon, 2 Apr 2018, Samuel Thibault wrote:

> - header standard conformity issues: These will be hard to fix.

What are the issues here?

> - elf/check-localplt: There will always be PLTs to libhurd/machuser.so
>   anyway.

If a library has *local* PLT entries - PLT entries for within-library 
calls to other functions in that shared library - that are hard to fix to 
use hidden aliases, it's expected that the localplt.data files will list 
those as expected (with a "?" if the PLT entry may or may not be present 
depending on compiler version etc., as is the case for some configurations 
where functions get exported from both libgcc and glibc).

> - elf/check-execstack: We have nested functions which make the stack
>   executable indeed.

That's pointers to nested functions involving creation of trampolines?  
Do those nested functions actually improve the code or would it be cleaner 
(have cleaner internal interfaces etc.) without them?  Do all libraries 
have these or only some?

> - check-abi-libmachuser, check-abi-libhurduser: These actually depend on
>   .defs files in gnumach and hurd, so we can't really define ABI files.

Depend in what way?  Are you saying they export different symbols 
depending on the versions of gnumach and hurd glibc is built with?  How 
are symbol versions for such extra symbols determined - based on gnumach 
and hurd versions instead of glibc versions?

It's clearly desirable to be able to make sure that old symbol versions 
don't change.  But if the contents of those versions are determined by 
gnumach and hurd, maybe those packages need to install ABI baselines for 
the glibc tests (or something like that).

The nearest analogue I see in glibc for systems using the Linux kernel 
(without more information on how the gnumach/hurd dependency works) is the 
syscall lists - where we have a list in glibc of all possible syscall 
names (sysdeps/unix/sysv/linux/syscall-names.list), and a test will fail 
if the kernel has additional syscalls and is not newer than the version 
listed in that list (so using a different kernel version does not result 
in test failures, but the test is fully effective if the listed kernel 
version or an older kernel version is used).

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Mon, 2 Apr 2018, Samuel Thibault wrote:

> This means that build-glibcs i686-gnu now builds fine.  Among the
> remaining TODOs, there are

Thanks!  I'd add: the "requires out-of-tree patches" statement in README 
needs to be removed, and a NEWS entry is needed.  Will you be able to 
provide full execution test results for the testsuite on the per-release 
wiki pages for 2.28 and later releases, during each release freeze period?

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Mon, 2 Apr 2018, Samuel Thibault wrote:

> Others really pose problem, like ./sysdeps/mach/hurd/bits/fcntl.h's
> l_type/l_whence being int instead of short.

Where something is problematic to fix, because a fix would break the ABI 
or needs some external feature, there is an xfail mechanism internal to 
the conform tests.  It involves a bug being filed in Bugzilla for the 
issue in question, an addition to conformtest-xfail-conds (conditional on 
ifeq ($(subdir),conform)) in an appropriate sysdeps Makefile, with a 
comment referencing the bug, and xfail[cond]- on the relevant expectation 
in the relevant -data file, again with a comment referencing the bug.  
For example:

// Bug 17786: st_dev has wrong type.
xfail[mips-o32-linux]-element {struct stat} dev_t st_dev

Before doing any such XFAILing, you should check that the expectation is 
actually backed up by the relevant standard and that a fix really does 
need an ABI change or a new feature.  Also, this XFAIL mechanism can only 
be used for positive expectations that are failing - it can't be used for 
cases where the header violates the expected namespace.

> There is also sys/un.h which defines a sun_len field, which IIRC is to
> be expected on BSD systems, but not defined in Posix?

Well, in fact POSIX reserves sun_* for sys/un.h (note the reservations are 
in a separate part of the standard from the main definitions of header 
contents), so there's a bug in the expectations not allowing for it.  This 
illustrates the need for checking such failures against the standards to 
see where the bug is.

> Also, ioctl takes (int, unsigned long int, ...) but
> ./conform/XPG42/stropts.h/conform.out wants (int, int, ...)?

That's already generically XFAILed with reference to bug 14362.

> > Do those nested functions actually improve the code
> 
> Yes.  There are notably callbacks whose parameters don't permit to get
> the context parameters etc.
> 
> >  Do all libraries have these or only some?
> 
> Only libc.so, ld.so and libpthread.so have them.

Then maybe some mechanism is needed for sysdeps Makefiles to define 
libraries expected to have executable stacks.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Mon, 2 Apr 2018, Samuel Thibault wrote:

> Samuel Thibault, on lun. 02 avril 2018 17:50:17 +0200, wrote:
> > There are a few remaining namespace issues due to missing __ marking or
> > spurious #includes.
> 
> One issue is with struct sched_param.  The __sched_param definition
> was removed in glibc while htl's pthread_attr uses it.  For now I just
> made struct___pthread_attr.h include  but that brings things
> unwanted for  (which is supposed to pull pthread_attr).
> 
> So I guess we should revert to defining __sched_param?

If you need, on Hurd, in installed headers, a type "struct __sched_param", 
that would most naturally go in a Hurd-specific 
bits/types/struct___sched_param.h.  If that's not appropriate, please 
explain the issue in more detail.  Anything restoring the old __need_* 
mechanism for any glibc-internal definitions is not appropriate (struct 
__sched_param was removed as part of removing __need_*).

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Mon, 2 Apr 2018, Samuel Thibault wrote:

> This means that build-glibcs i686-gnu now builds fine.  Among the
> remaining TODOs, there are

If you use mainline GCC, however, it fails:

../sysdeps/mach/hurd/if_index.c: In function '__if_nametoindex':
../sysdeps/mach/hurd/if_index.c:40:3: error: 'strncpy' specified bound 16 
equals destination size [-Werror=stringop-truncation]
   strncpy (ifr.ifr_name, ifname, IFNAMSIZ);
   ^~~~

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

The build for i686-gnu also fails using GCC 6 branch with 
build-many-glibcs.py:

hurdsig.c: In function 'interrupted_reply_port_location.isra.1':
hurdsig.c:250:39: error: 'portloc' may be used uninitialized in this function 
[-Werror=maybe-uninitialized]
   *(volatile mach_port_t *) portloc = *portloc;
   ^~~~

I haven't investigated whether this warning is valid, or whether it's 
bogus (in which case we don't add initializations just to fix warnings, 
but may used DIAG_*_NEEDS_COMMENT with appropriate explanations of what 
warning appears and why it is bogus).

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Tue, 3 Apr 2018, Zack Weinberg wrote:

> On Tue, Apr 3, 2018 at 5:58 PM, Samuel Thibault  
> wrote:
> > Joseph Myers, on mar. 03 avril 2018 21:48:32 +, wrote:
> >> The build for i686-gnu also fails using GCC 6 branch with
> >> build-many-glibcs.py:
> >>
> >> hurdsig.c: In function 'interrupted_reply_port_location.isra.1':
> >> hurdsig.c:250:39: error: 'portloc' may be used uninitialized in this 
> >> function [-Werror=maybe-uninitialized]
> >>*(volatile mach_port_t *) portloc = *portloc;
> >>^~~~
> >>
> >> I haven't investigated whether this warning is valid, or whether it's
> >> bogus
> >
> > Well, it's completely bogus: portloc is initialized just above. I don't
> > understand why it wouldn't see that.
> 
> I think it thinks the *value pointed-to by portloc* is uninitialized.

This is just after a call to _hurdsig_catch_memory_fault, which is a macro 
involving a call to setjmp.  I think the returns-twice nature of setjmp is 
confusing the compiler into thinking portloc itself is uninitialized (I 
don't know why this particular warning) - in fact, the second return of 
setjmp will never reach the code getting the warning (as setjmp will 
return nonzero, so _hurdsig_catch_memory_fault will).

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Wed, 18 Apr 2018, Samuel Thibault wrote:

> The patch below would just introduce bits/types/struct___sched_param.h.
> and bits/types/struct_sched_param.h for all ports since it's the same.

A bits/types/struct_sched_param.h that does "#define sched_param 
__sched_param" is not appropriate for Linux, because it would change the 
C++ mangling of struct sched_param.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Wed, 18 Apr 2018, Samuel Thibault wrote:

> Joseph Myers, le mar. 17 avril 2018 23:02:45 +, a ecrit:
> > On Wed, 18 Apr 2018, Samuel Thibault wrote:
> > 
> > > The patch below would just introduce bits/types/struct___sched_param.h.
> > > and bits/types/struct_sched_param.h for all ports since it's the same.
> > 
> > A bits/types/struct_sched_param.h that does "#define sched_param 
> > __sched_param" is not appropriate for Linux, because it would change the 
> > C++ mangling of struct sched_param.
> 
> Oh.
> 
> So that could be as per below?

That seems plausible (this is not a review of this patch).

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: [RFC PATCH 00/23] aarch64-gnu port

[Joseph Myers]

On Wed, 3 Jan 2024, Sergey Bugaev wrote:

> To build this, you need an aarch64-gnu toolchain (binutils, GCC, MIG),
> and GNU Mach headers for AArch64. I have posted the patches for
> binutils, GCC, and GNU Mach to the bug-hurd mailing list; no patches
> are required to build aarch64-gnu-mig.

> There is no AArch64 port of GNU Mach yet; I'm trying to get the ball
> rolling by making this port of glibc. I have only added some AArch64

I think the same principle applies for ports to new (architecture, Hurd) 
pairs as for new (architecture, Linux) pairs: the relevant code needs to 
be in upstream mainline of all components on which glibc build-depends 
before the port can go into glibc (and thus before the symbol versions for 
the port can be determined) - and at the point where it's ready to go in, 
a corresponding build-many-glibcs.py entry should be added as part of the 
patch series.  Parts of  may 
be applicable, though that's more oriented to the case where the 
architecture support itself is entirely new.

You can of course do refactors of existing files not specific to 
aarch64-gnu, in preparation for the port, before then, to reduce the size 
of the eventual patch series proposing adding the port to glibc.

-- 
Joseph S. Myers
j...@polyomino.org.uk