Re: GCC Multi-Threading Ideas
On Freitag, 24. Januar 2020 04:38:48 CET Nicholas Krause wrote: > On 1/23/20 12:19 PM, Nicholas Krause wrote: > > On 1/23/20 3:39 AM, Allan Sandfeld Jensen wrote: > >> On Montag, 20. Januar 2020 20:26:46 CET Nicholas Krause wrote: > >>> Greetings All, > >>> > >>> Unfortunately due to me being rather busy with school and other > >>> things I > >>> will not be able to post my article to the wiki for awhile. However > >>> there is a rough draft here: > >>> https://docs.google.com/document/d/1po_RRgSCtRyYgMHjV0itW8iOzJXpTdHYIpC9 > >>> gUMj > >>> > >>> Oxk/edit that may change a little for people to read in the meantime. > >> > >> This comment might not be suited for your project, but now that I > >> think about > >> it: If we want to improve gcc toolchain buildspeed with better > >> multithreading. > >> I think the most sensible would be fixing up gold multithreading and > >> enabling > >> it by default. We already get most of the benefits of multicore > >> architectures > >> by running multiple compile jobs in parallel (yes, I know you are > >> focusing on > >> cases where that for some reason doesn't work, but it is still the > >> case in > >> most situations). The main bottleneck is linking. The code is even > >> already > >> there in gold and have been for years, it just haven't been deemed > >> ready for > >> being enabled by default. > >> > >> Is anyone even working on that? > >> > >> Best regards > >> Allan > > > > Allan, > > You would need both depending on the project, some are more compiler > > bottle necked and others linker. I mentioned that issue at Cauldron as > > the other side would be the linker. > > > > Nick > > Sorry for the second message Allan but make -j does not scale well > beyond 4 or > 8 threads and that's considering a 4 core or 8 machine. It doesn't? I generally build with -j100, but then also use icecream to distribute builds to multiple machines in the office. That probably also makes linking times more significant to my case. 'Allan
Re: GCC Multi-Threading Ideas
On Fri, 24 Jan 2020 at 03:39, Nicholas Krause wrote: > Sorry for the second message Allan but make -j does not scale well > beyond 4 or > 8 threads and that's considering a 4 core or 8 machine. The problem has to > do with large build machines with CPUs with more cores than this or as > is becoming > more common on mainstream systems. And make scales well beyond 8 processes (not threads) on such machines.
Re: Merges from release branches to vendor tracking branches
On Thu, 23 Jan 2020 at 23:15, Peter Bergner wrote: > > On 1/23/20 12:09 PM, Peter Bergner wrote: > > On 1/23/20 4:29 AM, Jakub Jelinek wrote: > >> so it is not a fast forward merge and we have the requirement that > >> From-SVN: shouldn't appear in commit logs of new commits. > > > > So I just did "git merge releases/gcc-9" into our branch and I'm not > > seeing any From-SVN: in any of the commit messages. Where/how are > > you seeing those? > > Actually, I see them now. I'm not sure what happened before. > > So Joseph said these are actually ok on the vendor branches, > but what was the original concern with them being there in the > commit logs? We don't want a commit that has been cherry-picked onto a branch (and possibly edited to resolve conflicts) to have "From-SVN" in the log, because that would confuse various scripts into thinking that your new commit on the branch *is* the commit imported from SVN. If it's a different commit, it shouldn't have the From-SVN marker from the original commit. Only commits that actually came from SVN should say From-SVN in their logs. > Is it still useful to remove them? If the hooks don't reject it, I'd say no. If the commit really *is* a merge commit, i.e. you do a 'git merge' to add the same commit with the original hash to your vendor branch, then it's not useful to remove them. Because in that case it really *is* the commit that was imported from SVN (and removing the From-SVN would just alter the commit and give it a different hash, so now it would be a different commit, for no good reason).
Re: [PATCH] wwwdocs: document scripts to access personal and vendor spaces
On 23/01/20 16:23 +, Richard Earnshaw (lists) wrote: On 21/01/2020 18:58, Richard Earnshaw (lists) wrote: This patch documents some of the scripts that I've published for managing the personal and vendor spaces on the server. It also covers some of the other features that those scripts enable, so that it's all in one place. This is a complete rewrite of the material I had written previously since before we did not have these scripts to make use of. I've not filled in the documentation for gcc-descr and gcc-undescr, Jakub has agreed to provide that at a later date. R. I've pushed this. I think it's better to have this in than nothing at all. We can iterate on it as required. I've pushed the attached fix for a couple of typos. Do we want to repeat the "do not push changes to that space without their express permission" for user branches? There are no access checks to prevent it, but I hsouldn't be changing things in your branches without your permission. Finally, why does this qualify it as "If you have multiple clones"? If you have multiple clones of the gcc repository you can fetch updates from your personal space by running git fetch me Isn't that the right command even if you only have one clone? commit 182ab16b43b0b40b985e0678891b29debef2c9a2 Author: Jonathan Wakely Date: Fri Jan 24 10:59:14 2020 + Fix typos in gitwrite.html diff --git a/htdocs/gitwrite.html b/htdocs/gitwrite.html index ad073b53..c0fe8526 100644 --- a/htdocs/gitwrite.html +++ b/htdocs/gitwrite.html @@ -391,7 +391,7 @@ configuration steps can give access to them. in refs/vendors/vendor-name/tags. -Scripts exist the contrib directory to help manage these spaces. +Scripts exist in the contrib directory to help manage these spaces. contrib/gcc-git-customization.sh @@ -413,7 +413,7 @@ some aliases that might be useful when developing GCC. The script will work this out based on the URL used to fetch from the git server, or fall back to your local user name if that fails. Using this name on the server for your personal space ensures that your branches will -not conflict with anybody elses. +not conflict with anybody else's. The prefix to use for your personal branches - the default is me, but you can change this if you prefer. The script will add configuration information to allow local branches that
Re: Wrong GCC PR2020 annotated for "[committed, libgomp,amdgcn] Fix plugin-gcn.c bug"
On Thu, 23 Jan 2020 at 16:57, Andrew Stubbs wrote: > Indeed, PR2019 has a number of unrelated commits referenced, and pr2018 > has one too. The years before that appear to have escaped the problem. The 2019 ones are even more silly, as the unrelated commits were correcting the PR number in some filenames! I'll tag those bugzilla comments as "obsolete" so they're hidden by default and don't confuse people.
Re: [PATCH] wwwdocs: document scripts to access personal and vendor spaces
On 24/01/2020 11:04, Jonathan Wakely wrote: On 23/01/20 16:23 +, Richard Earnshaw (lists) wrote: On 21/01/2020 18:58, Richard Earnshaw (lists) wrote: This patch documents some of the scripts that I've published for managing the personal and vendor spaces on the server. It also covers some of the other features that those scripts enable, so that it's all in one place. This is a complete rewrite of the material I had written previously since before we did not have these scripts to make use of. I've not filled in the documentation for gcc-descr and gcc-undescr, Jakub has agreed to provide that at a later date. R. I've pushed this. I think it's better to have this in than nothing at all. We can iterate on it as required. I've pushed the attached fix for a couple of typos. Do we want to repeat the "do not push changes to that space without their express permission" for user branches? There are no access checks to prevent it, but I hsouldn't be changing things in your branches without your permission. Absolutely. I guess I took this as read in the personal spaces. Finally, why does this qualify it as "If you have multiple clones"? If you have multiple clones of the gcc repository you can fetch updates from your personal space by running git fetch me Isn't that the right command even if you only have one clone? Well, if you only have one clone, why would you need to pull branches from upstream that you pushed yourself? In fact, why would you even need to push them in that case, unless it's for backup? ... and if you're restoring your backup, then that's a new clone. You might temporally have only only one clone, but across all time you have more than one. It's pedantry, though, so feel free to re-word it. Thanks for the tweaks, anyway. R.
Re: [PATCH] wwwdocs: document scripts to access personal and vendor spaces
On 24/01/20 11:48 +, Richard Earnshaw (lists) wrote: On 24/01/2020 11:04, Jonathan Wakely wrote: On 23/01/20 16:23 +, Richard Earnshaw (lists) wrote: On 21/01/2020 18:58, Richard Earnshaw (lists) wrote: This patch documents some of the scripts that I've published for managing the personal and vendor spaces on the server. It also covers some of the other features that those scripts enable, so that it's all in one place. This is a complete rewrite of the material I had written previously since before we did not have these scripts to make use of. I've not filled in the documentation for gcc-descr and gcc-undescr, Jakub has agreed to provide that at a later date. R. I've pushed this. I think it's better to have this in than nothing at all. We can iterate on it as required. I've pushed the attached fix for a couple of typos. Do we want to repeat the "do not push changes to that space without their express permission" for user branches? There are no access checks to prevent it, but I hsouldn't be changing things in your branches without your permission. Absolutely. I guess I took this as read in the personal spaces. If we don't document it, somebody will get it wrong :-) Maybe like so, after the "To create a new personal branch" paragraph: Personal spaces are controlled by the individual user. Do not push changes to somebody else's space without their express permission. (Rather than pushing to somebody else's personal branch, consider pushing to your own personal branch and having collaborators pull from there). Is the parenthesis going into too much detail, too early? This page doesn't describe how to pull from somebody else's personal branch (and arguably that would be too much info here anyway, but it might make sense for https://gcc.gnu.org/wiki/GitCookbook instead). So maybe just the part in . Finally, why does this qualify it as "If you have multiple clones"? If you have multiple clones of the gcc repository you can fetch updates from your personal space by running git fetch me Isn't that the right command even if you only have one clone? Well, if you only have one clone, why would you need to pull branches from upstream that you pushed yourself? Good point. In fact, why would you even need to push them in that case, unless it's for backup? ... and if So other people can see your work in progress (without the overhead of setting up a devel/* branch for a short-lived topic branch). you're restoring your backup, then that's a new clone. You might temporally have only only one clone, but across all time you have more than one. Right. I was stuck in small-minded, non-distributed thinking :-) It's pedantry, though, so feel free to re-word it. How about: "You can fetch updates from your personal space into some other clone of the repository (e.g. on a machine used for testing) by running:"
Re: [PATCH] wwwdocs: document scripts to access personal and vendor spaces
On 24/01/2020 12:19, Jonathan Wakely wrote: On 24/01/20 11:48 +, Richard Earnshaw (lists) wrote: On 24/01/2020 11:04, Jonathan Wakely wrote: On 23/01/20 16:23 +, Richard Earnshaw (lists) wrote: On 21/01/2020 18:58, Richard Earnshaw (lists) wrote: This patch documents some of the scripts that I've published for managing the personal and vendor spaces on the server. It also covers some of the other features that those scripts enable, so that it's all in one place. This is a complete rewrite of the material I had written previously since before we did not have these scripts to make use of. I've not filled in the documentation for gcc-descr and gcc-undescr, Jakub has agreed to provide that at a later date. R. I've pushed this. I think it's better to have this in than nothing at all. We can iterate on it as required. I've pushed the attached fix for a couple of typos. Do we want to repeat the "do not push changes to that space without their express permission" for user branches? There are no access checks to prevent it, but I hsouldn't be changing things in your branches without your permission. Absolutely. I guess I took this as read in the personal spaces. If we don't document it, somebody will get it wrong :-) Maybe like so, after the "To create a new personal branch" paragraph: Personal spaces are controlled by the individual user. Do not push changes to somebody else's space without their express permission. (Rather than pushing to somebody else's personal branch, consider pushing to your own personal branch and having collaborators pull from there). Is the parenthesis going into too much detail, too early? This page doesn't describe how to pull from somebody else's personal branch (and arguably that would be too much info here anyway, but it might make sense for https://gcc.gnu.org/wiki/GitCookbook instead). I think the 'pull' model is probably preferable for personal spaces. At some point we might need to enforce something like that anyway. So maybe just the part in . Finally, why does this qualify it as "If you have multiple clones"? If you have multiple clones of the gcc repository you can fetch updates from your personal space by running git fetch me Isn't that the right command even if you only have one clone? Well, if you only have one clone, why would you need to pull branches from upstream that you pushed yourself? Good point. In fact, why would you even need to push them in that case, unless it's for backup? ... and if So other people can see your work in progress (without the overhead of setting up a devel/* branch for a short-lived topic branch). you're restoring your backup, then that's a new clone. You might temporally have only only one clone, but across all time you have more than one. Right. I was stuck in small-minded, non-distributed thinking :-) It's pedantry, though, so feel free to re-word it. How about: "You can fetch updates from your personal space into some other clone of the repository (e.g. on a machine used for testing) by running:" Yes, that's fine. Note, that I spotted a problem with the simple "me" remote, in that git gets confused if you have both refs/heads// and refs/remotes// (checking out me/ results in complaints about an ambiguous ref). I'm currently reworking my scripts to use users/ for the remote, much like the 'vendors' space. Watch this space... R.
Re: [PATCH] wwwdocs: document scripts to access personal and vendor spaces
On 24/01/20 13:55 +, Richard Earnshaw (lists) wrote: On 24/01/2020 12:19, Jonathan Wakely wrote: On 24/01/20 11:48 +, Richard Earnshaw (lists) wrote: On 24/01/2020 11:04, Jonathan Wakely wrote: On 23/01/20 16:23 +, Richard Earnshaw (lists) wrote: On 21/01/2020 18:58, Richard Earnshaw (lists) wrote: This patch documents some of the scripts that I've published for managing the personal and vendor spaces on the server. It also covers some of the other features that those scripts enable, so that it's all in one place. This is a complete rewrite of the material I had written previously since before we did not have these scripts to make use of. I've not filled in the documentation for gcc-descr and gcc-undescr, Jakub has agreed to provide that at a later date. R. I've pushed this. I think it's better to have this in than nothing at all. We can iterate on it as required. I've pushed the attached fix for a couple of typos. Do we want to repeat the "do not push changes to that space without their express permission" for user branches? There are no access checks to prevent it, but I hsouldn't be changing things in your branches without your permission. Absolutely. I guess I took this as read in the personal spaces. If we don't document it, somebody will get it wrong :-) Maybe like so, after the "To create a new personal branch" paragraph: Personal spaces are controlled by the individual user. Do not push changes to somebody else's space without their express permission. (Rather than pushing to somebody else's personal branch, consider pushing to your own personal branch and having collaborators pull from there). Is the parenthesis going into too much detail, too early? This page doesn't describe how to pull from somebody else's personal branch (and arguably that would be too much info here anyway, but it might make sense for https://gcc.gnu.org/wiki/GitCookbook instead). I think the 'pull' model is probably preferable for personal spaces. At some point we might need to enforce something like that anyway. So maybe just the part in . Finally, why does this qualify it as "If you have multiple clones"? If you have multiple clones of the gcc repository you can fetch updates from your personal space by running git fetch me Isn't that the right command even if you only have one clone? Well, if you only have one clone, why would you need to pull branches from upstream that you pushed yourself? Good point. In fact, why would you even need to push them in that case, unless it's for backup? ... and if So other people can see your work in progress (without the overhead of setting up a devel/* branch for a short-lived topic branch). you're restoring your backup, then that's a new clone. You might temporally have only only one clone, but across all time you have more than one. Right. I was stuck in small-minded, non-distributed thinking :-) It's pedantry, though, so feel free to re-word it. How about: "You can fetch updates from your personal space into some other clone of the repository (e.g. on a machine used for testing) by running:" Yes, that's fine. OK, I've pushed the attached patch. commit 0bce4cca846fa1d79f080d2784f90aaa5c447e2b Author: Jonathan Wakely Date: Fri Jan 24 15:00:57 2020 + Adjust docs on personal Git branches diff --git a/htdocs/gitwrite.html b/htdocs/gitwrite.html index c0fe8526..5b631607 100644 --- a/htdocs/gitwrite.html +++ b/htdocs/gitwrite.html @@ -424,8 +424,8 @@ some aliases that might be useful when developing GCC. The script will settings configured by the script will still be useful. -If you have multiple clones of the gcc repository you can fetch -updates from your personal space by running +You can fetch updates from your personal space into some other clone +of the repository (e.g. on a machine used for testing) by running git fetch me (or whatever personal prefix you've chosen). You can also push an already existing branch using git push me me/branch. @@ -444,6 +444,12 @@ used: If you've used a different personal prefix to 'me' then use that in the sequence described above. +Personal spaces are controlled by the individual user. Do not push +changes to somebody else's space without their express permission. +(Rather than pushing to somebody else's personal branch, consider pushing +to your own personal branch and having collaborators pull from there.) + + The script also defines a few useful aliases that can be used with the repository:
Re: [PATCH] wwwdocs: document scripts to access personal and vendor spaces
On 24/01/20 15:09 +, Jonathan Wakely wrote: On 24/01/20 13:55 +, Richard Earnshaw (lists) wrote: On 24/01/2020 12:19, Jonathan Wakely wrote: On 24/01/20 11:48 +, Richard Earnshaw (lists) wrote: On 24/01/2020 11:04, Jonathan Wakely wrote: On 23/01/20 16:23 +, Richard Earnshaw (lists) wrote: On 21/01/2020 18:58, Richard Earnshaw (lists) wrote: This patch documents some of the scripts that I've published for managing the personal and vendor spaces on the server. It also covers some of the other features that those scripts enable, so that it's all in one place. This is a complete rewrite of the material I had written previously since before we did not have these scripts to make use of. I've not filled in the documentation for gcc-descr and gcc-undescr, Jakub has agreed to provide that at a later date. R. I've pushed this. I think it's better to have this in than nothing at all. We can iterate on it as required. I've pushed the attached fix for a couple of typos. Do we want to repeat the "do not push changes to that space without their express permission" for user branches? There are no access checks to prevent it, but I hsouldn't be changing things in your branches without your permission. Absolutely. I guess I took this as read in the personal spaces. If we don't document it, somebody will get it wrong :-) Maybe like so, after the "To create a new personal branch" paragraph: Personal spaces are controlled by the individual user. Do not push changes to somebody else's space without their express permission. (Rather than pushing to somebody else's personal branch, consider pushing to your own personal branch and having collaborators pull from there). Is the parenthesis going into too much detail, too early? This page doesn't describe how to pull from somebody else's personal branch (and arguably that would be too much info here anyway, but it might make sense for https://gcc.gnu.org/wiki/GitCookbook instead). I think the 'pull' model is probably preferable for personal spaces. At some point we might need to enforce something like that anyway. So maybe just the part in . Finally, why does this qualify it as "If you have multiple clones"? If you have multiple clones of the gcc repository you can fetch updates from your personal space by running git fetch me Isn't that the right command even if you only have one clone? Well, if you only have one clone, why would you need to pull branches from upstream that you pushed yourself? Good point. In fact, why would you even need to push them in that case, unless it's for backup? ... and if So other people can see your work in progress (without the overhead of setting up a devel/* branch for a short-lived topic branch). you're restoring your backup, then that's a new clone. You might temporally have only only one clone, but across all time you have more than one. Right. I was stuck in small-minded, non-distributed thinking :-) It's pedantry, though, so feel free to re-word it. How about: "You can fetch updates from your personal space into some other clone of the repository (e.g. on a machine used for testing) by running:" Yes, that's fine. OK, I've pushed the attached patch. Oops, wrong patch, *this* is what I pushed. commit f526c4724b183482fc7fe1a5ce78e1597fd1005f Author: Jonathan Wakely Date: Fri Jan 24 15:00:57 2020 + Adjust docs on personal Git branches diff --git a/htdocs/gitwrite.html b/htdocs/gitwrite.html index 55667a2d..b5d7ff35 100644 --- a/htdocs/gitwrite.html +++ b/htdocs/gitwrite.html @@ -436,6 +436,12 @@ you intend. The script contrib/git-add-user-branch.sh can be used to create a new personal branch which can be pushed and pulled from the users/me remote. +Personal spaces are controlled by the individual user. Do not push +changes to somebody else's space without their express permission. +(Rather than pushing to somebody else's personal branch, consider pushing +to your own personal branch and having collaborators pull from there.) + + The script also defines a few useful aliases that can be used with the repository:
Re: [PATCH] wwwdocs: document scripts to access personal and vendor spaces
On 24/01/2020 15:12, Jonathan Wakely wrote: On 24/01/20 15:09 +, Jonathan Wakely wrote: On 24/01/20 13:55 +, Richard Earnshaw (lists) wrote: On 24/01/2020 12:19, Jonathan Wakely wrote: On 24/01/20 11:48 +, Richard Earnshaw (lists) wrote: On 24/01/2020 11:04, Jonathan Wakely wrote: On 23/01/20 16:23 +, Richard Earnshaw (lists) wrote: On 21/01/2020 18:58, Richard Earnshaw (lists) wrote: This patch documents some of the scripts that I've published for managing the personal and vendor spaces on the server. It also covers some of the other features that those scripts enable, so that it's all in one place. This is a complete rewrite of the material I had written previously since before we did not have these scripts to make use of. I've not filled in the documentation for gcc-descr and gcc-undescr, Jakub has agreed to provide that at a later date. R. I've pushed this. I think it's better to have this in than nothing at all. We can iterate on it as required. I've pushed the attached fix for a couple of typos. Do we want to repeat the "do not push changes to that space without their express permission" for user branches? There are no access checks to prevent it, but I hsouldn't be changing things in your branches without your permission. Absolutely. I guess I took this as read in the personal spaces. If we don't document it, somebody will get it wrong :-) Maybe like so, after the "To create a new personal branch" paragraph: Personal spaces are controlled by the individual user. Do not push changes to somebody else's space without their express permission. (Rather than pushing to somebody else's personal branch, consider pushing to your own personal branch and having collaborators pull from there). Is the parenthesis going into too much detail, too early? This page doesn't describe how to pull from somebody else's personal branch (and arguably that would be too much info here anyway, but it might make sense for https://gcc.gnu.org/wiki/GitCookbook instead). I think the 'pull' model is probably preferable for personal spaces. At some point we might need to enforce something like that anyway. So maybe just the part in . Finally, why does this qualify it as "If you have multiple clones"? If you have multiple clones of the gcc repository you can fetch updates from your personal space by running git fetch me Isn't that the right command even if you only have one clone? Well, if you only have one clone, why would you need to pull branches from upstream that you pushed yourself? Good point. In fact, why would you even need to push them in that case, unless it's for backup? ... and if So other people can see your work in progress (without the overhead of setting up a devel/* branch for a short-lived topic branch). you're restoring your backup, then that's a new clone. You might temporally have only only one clone, but across all time you have more than one. Right. I was stuck in small-minded, non-distributed thinking :-) It's pedantry, though, so feel free to re-word it. How about: "You can fetch updates from your personal space into some other clone of the repository (e.g. on a machine used for testing) by running:" Yes, that's fine. OK, I've pushed the attached patch. Oops, wrong patch, *this* is what I pushed. Thanks. On reflection, I've reworded this slightly and moved it up to the introduction to the personal and vendor spaces. The same principle applies to both. Pushed. R. diff --git a/htdocs/gitwrite.html b/htdocs/gitwrite.html index b5d7ff35..97d60870 100644 --- a/htdocs/gitwrite.html +++ b/htdocs/gitwrite.html @@ -378,7 +378,7 @@ The GCC git repository is used by many people and the branch and tag namespace would become very polluted if all branches lived at the top-level naming space. To help minimise the amount of data that needs to be fetched the git repository on gcc.gnu.org supports -having personal and vendor branches that developers use to +having personal and vendor spaces that developers use to share their work. These are not pulled by default, but simple configuration steps can give access to them. @@ -391,7 +391,14 @@ configuration steps can give access to them. in refs/vendors/vendor-name/tags. -Scripts exist in the contrib directory to help manage these spaces. +Personal and vendor spaces are controlled by the individual +user or vendor. Do not push changes to somebody else's space without +their express permission. Rather than pushing to somebody +else's branch, consider pushing to your own personal branch and having +collaborators pull from there. + + +Scripts exist in the contrib directory to help manage these spaces. contrib/gcc-git-customization.sh @@ -436,12 +443,6 @@ you intend. The script contrib/git-add-user-branch.sh can be used to create a new personal branch which can be pushed and pulled from the users/me remote. -Personal spaces are controlled by t
Re: GCC Multi-Threading Ideas
On 1/24/20 3:18 AM, Allan Sandfeld Jensen wrote: On Freitag, 24. Januar 2020 04:38:48 CET Nicholas Krause wrote: On 1/23/20 12:19 PM, Nicholas Krause wrote: On 1/23/20 3:39 AM, Allan Sandfeld Jensen wrote: On Montag, 20. Januar 2020 20:26:46 CET Nicholas Krause wrote: Greetings All, Unfortunately due to me being rather busy with school and other things I will not be able to post my article to the wiki for awhile. However there is a rough draft here: https://docs.google.com/document/d/1po_RRgSCtRyYgMHjV0itW8iOzJXpTdHYIpC9 gUMj Oxk/edit that may change a little for people to read in the meantime. This comment might not be suited for your project, but now that I think about it: If we want to improve gcc toolchain buildspeed with better multithreading. I think the most sensible would be fixing up gold multithreading and enabling it by default. We already get most of the benefits of multicore architectures by running multiple compile jobs in parallel (yes, I know you are focusing on cases where that for some reason doesn't work, but it is still the case in most situations). The main bottleneck is linking. The code is even already there in gold and have been for years, it just haven't been deemed ready for being enabled by default. Is anyone even working on that? Best regards Allan Allan, You would need both depending on the project, some are more compiler bottle necked and others linker. I mentioned that issue at Cauldron as the other side would be the linker. Nick Sorry for the second message Allan but make -j does not scale well beyond 4 or 8 threads and that's considering a 4 core or 8 machine. It doesn't? I generally build with -j100, but then also use icecream to distribute builds to multiple machines in the office. That probably also makes linking times more significant to my case. 'Allan Allan, I ran a gcc build on a machine with make -j32 and -j64 that had 64 cores. There was literally only a 4 minute increase in build speed. Good question through. Nick
Re: GCC Multi-Threading Ideas
On 24/01/2020 10:27, Jonathan Wakely wrote: On Fri, 24 Jan 2020 at 03:39, Nicholas Krause wrote: Sorry for the second message Allan but make -j does not scale well beyond 4 or 8 threads and that's considering a 4 core or 8 machine. The problem has to do with large build machines with CPUs with more cores than this or as is becoming more common on mainstream systems. And make scales well beyond 8 processes (not threads) on such machines. The problem isn't make, per se, or even gcc. It's the build system as a whole. On a highly multi-core machine, gcc itself hits the bottle-neck called configure. That's serial, run *many* times (especially when there are many multilibs) and dominates build time. On high multi-core machines, gcc's 15-minute system load gets no-where near to the number of threads on the machine because of this. R.
Re: GCC Multi-Threading Ideas
On Freitag, 24. Januar 2020 17:29:06 CET Nicholas Krause wrote: > On 1/24/20 3:18 AM, Allan Sandfeld Jensen wrote: > > On Freitag, 24. Januar 2020 04:38:48 CET Nicholas Krause wrote: > >> On 1/23/20 12:19 PM, Nicholas Krause wrote: > >>> On 1/23/20 3:39 AM, Allan Sandfeld Jensen wrote: > On Montag, 20. Januar 2020 20:26:46 CET Nicholas Krause wrote: > > Greetings All, > > > > Unfortunately due to me being rather busy with school and other > > things I > > will not be able to post my article to the wiki for awhile. However > > there is a rough draft here: > > https://docs.google.com/document/d/1po_RRgSCtRyYgMHjV0itW8iOzJXpTdHYIp > > C9 > > gUMj > > > > Oxk/edit that may change a little for people to read in the meantime. > > This comment might not be suited for your project, but now that I > think about > it: If we want to improve gcc toolchain buildspeed with better > multithreading. > I think the most sensible would be fixing up gold multithreading and > enabling > it by default. We already get most of the benefits of multicore > architectures > by running multiple compile jobs in parallel (yes, I know you are > focusing on > cases where that for some reason doesn't work, but it is still the > case in > most situations). The main bottleneck is linking. The code is even > already > there in gold and have been for years, it just haven't been deemed > ready for > being enabled by default. > > Is anyone even working on that? > > Best regards > Allan > >>> > >>> Allan, > >>> You would need both depending on the project, some are more compiler > >>> bottle necked and others linker. I mentioned that issue at Cauldron as > >>> the other side would be the linker. > >>> > >>> Nick > >> > >> Sorry for the second message Allan but make -j does not scale well > >> beyond 4 or > >> 8 threads and that's considering a 4 core or 8 machine. > > > > It doesn't? I generally build with -j100, but then also use icecream to > > distribute builds to multiple machines in the office. That probably also > > makes linking times more significant to my case. > > > > 'Allan > > Allan, > > I ran a gcc build on a machine with make -j32 and -j64 that had 64 cores. > There was literally only a 4 minute increase in build speed. Good question > through. > Right. I guess it entirely depends on what you are building. If you are building gcc, it is probably bound by multiple configure runs, and separate iterations. What I usually build is Qt and Chromium, where thousands of files can be compiled from a single configure run (more than 2 in the case of Chromium), plus those configure runs are much faster. For Chromium there is almost a linear speed up with the number of parallel jobs you run up to around 100. With -j100 I can build Chromium in 10 minutes, with 2 minutes being linking time (5 minutes linking if using bfd linker). With -j8 it takes 2 hours. But I guess that means multithreading the compiler can make sense to your case, even if it doesn't to mine. Regards 'Allan
Re: GCC Multi-Threading Ideas
On 1/24/20 1:28 PM, Allan Sandfeld Jensen wrote: On Freitag, 24. Januar 2020 17:29:06 CET Nicholas Krause wrote: On 1/24/20 3:18 AM, Allan Sandfeld Jensen wrote: On Freitag, 24. Januar 2020 04:38:48 CET Nicholas Krause wrote: On 1/23/20 12:19 PM, Nicholas Krause wrote: On 1/23/20 3:39 AM, Allan Sandfeld Jensen wrote: On Montag, 20. Januar 2020 20:26:46 CET Nicholas Krause wrote: Greetings All, Unfortunately due to me being rather busy with school and other things I will not be able to post my article to the wiki for awhile. However there is a rough draft here: https://docs.google.com/document/d/1po_RRgSCtRyYgMHjV0itW8iOzJXpTdHYIp C9 gUMj Oxk/edit that may change a little for people to read in the meantime. This comment might not be suited for your project, but now that I think about it: If we want to improve gcc toolchain buildspeed with better multithreading. I think the most sensible would be fixing up gold multithreading and enabling it by default. We already get most of the benefits of multicore architectures by running multiple compile jobs in parallel (yes, I know you are focusing on cases where that for some reason doesn't work, but it is still the case in most situations). The main bottleneck is linking. The code is even already there in gold and have been for years, it just haven't been deemed ready for being enabled by default. Is anyone even working on that? Best regards Allan Allan, You would need both depending on the project, some are more compiler bottle necked and others linker. I mentioned that issue at Cauldron as the other side would be the linker. Nick Sorry for the second message Allan but make -j does not scale well beyond 4 or 8 threads and that's considering a 4 core or 8 machine. It doesn't? I generally build with -j100, but then also use icecream to distribute builds to multiple machines in the office. That probably also makes linking times more significant to my case. 'Allan Allan, I ran a gcc build on a machine with make -j32 and -j64 that had 64 cores. There was literally only a 4 minute increase in build speed. Good question through. Right. I guess it entirely depends on what you are building. If you are building gcc, it is probably bound by multiple configure runs, and separate iterations. What I usually build is Qt and Chromium, where thousands of files can be compiled from a single configure run (more than 2 in the case of Chromium), plus those configure runs are much faster. For Chromium there is almost a linear speed up with the number of parallel jobs you run up to around 100. With -j100 I can build Chromium in 10 minutes, with 2 minutes being linking time (5 minutes linking if using bfd linker). With -j8 it takes 2 hours. But I guess that means multithreading the compiler can make sense to your case, even if it doesn't to mine. Regards 'Allan Allan, The question I would have is make -j on one machine as that's my point. You can distribute it out with icecream or other thinks but again that doesn't always help. In the paper in does mention interaction with make and build systems to make it scale better if possible. Not sure how QT or Chromium would be directly affected. However its a considered use case where make scales fine but may do better with coupling into the internal compiler multi-threading. Again even 10 minutes should be make better if make interacted with gcc multi-threading well. Nick
Git ChangeLog policy for GCC Testsuite inquiry
>> > On 1/24/20 8:45 AM, David Edelsohn wrote: >> > > There is no ChangeLog entry for the testsuite changes. >> > >> > I don't believe in ChangeLog entries for testcases, but I'll add one for >> > the target-supports.exp change, thanks. >> >> Is this a general policy change that we want to make? Current we >> still have gcc/testsuite/ChangeLog and developers are updating that >> file. > > I would support formalizing that as policy; currently there is no policy. > > https://gcc.gnu.org/codingconventions.html#ChangeLogs > > "There is no established convention on when ChangeLog entries are to be made > for testsuite changes." Do we want to continue with ChangeLog entries for testsuite changes or only rely on Git log? Thanks, David
Re: Git ChangeLog policy for GCC Testsuite inquiry
On Fri, 2020-01-24 at 13:49 -0500, David Edelsohn wrote: > > > > On 1/24/20 8:45 AM, David Edelsohn wrote: > > > > > There is no ChangeLog entry for the testsuite changes. > > > > > > > > I don't believe in ChangeLog entries for testcases, but I'll add one for > > > > the target-supports.exp change, thanks. > > > > > > Is this a general policy change that we want to make? Current we > > > still have gcc/testsuite/ChangeLog and developers are updating that > > > file. > > > > I would support formalizing that as policy; currently there is no policy. > > > > https://gcc.gnu.org/codingconventions.html#ChangeLogs > > > > "There is no established convention on when ChangeLog entries are to be > > made for testsuite changes." > > Do we want to continue with ChangeLog entries for testsuite changes or > only rely on Git log? I strongly prefer to move towards relying on the git log. jeff
Re: Git ChangeLog policy for GCC Testsuite inquiry
> I strongly prefer to move towards relying on the git log. In my experience the output of git log is a total mess so cannot replace ChangeLogs. But we can well decide to drop ChangeLog for the testsuite. -- Eric Botcazou
Re: Git ChangeLog policy for GCC Testsuite inquiry
* Eric Botcazou: >> I strongly prefer to move towards relying on the git log. > > In my experience the output of git log is a total mess so cannot replace > ChangeLogs. That's fixable if the commit message is part of the patch review (just like the source code comments).
Re: Git ChangeLog policy for GCC Testsuite inquiry
On Fri, 2020-01-24 at 20:32 +0100, Eric Botcazou wrote: > > I strongly prefer to move towards relying on the git log. > > In my experience the output of git log is a total mess so cannot replace > ChangeLogs. But we can well decide to drop ChangeLog for the testsuite. Well, glibc has moved to extracting them from git, building policies and scripts around that. I'm pretty sure other significant projecs are also extracting their ChangeLogs from git. We could do the same, selecting some magic date as the cutover point after which future ChangeLogs are extracted from GIT. In fact, that's precisely what I'd like to see us do. jeff
Re: Git ChangeLog policy for GCC Testsuite inquiry
On Fri, Jan 24, 2020 at 02:36:31PM -0700, Jeff Law wrote: > On Fri, 2020-01-24 at 20:32 +0100, Eric Botcazou wrote: > > > I strongly prefer to move towards relying on the git log. > > > > In my experience the output of git log is a total mess so cannot replace > > ChangeLogs. But we can well decide to drop ChangeLog for the testsuite. > Well, glibc has moved to extracting them from git, building policies > and scripts around that. I'm pretty sure other significant projecs are > also extracting their ChangeLogs from git. > > We could do the same, selecting some magic date as the cutover point > after which future ChangeLogs are extracted from GIT. In fact, that's > precisely what I'd like to see us do. We don't have a tool that can do it, not even get the boilerplate right. Yes, mklog helps, but it very often gets stuff wrong. Not to mention that the text what actually changed can't be generated very easily. Jakub
Re: Git ChangeLog policy for GCC Testsuite inquiry
On Fri, 2020-01-24 at 22:45 +0100, Jakub Jelinek wrote: > > > In my experience the output of git log is a total mess so cannot > > > replace ChangeLogs. But we can well decide to drop ChangeLog for > > > the testsuite. > > > > Well, glibc has moved to extracting them from git, building > > policies and scripts around that. I'm pretty sure other > > significant projecs are also extracting their ChangeLogs from git. > > > > We could do the same, selecting some magic date as the cutover > > point after which future ChangeLogs are extracted from GIT. In > > fact, that's precisely what I'd like to see us do. > > We don't have a tool that can do it, not even get the boilerplate > right. Yes, mklog helps, but it very often gets stuff wrong. Not to > mention that the text what actually changed can't be generated very > easily. I don't know if it counts as a significant project, but GNU make has been doing this for years. What I did was take the existing ChangeLogs and rename them to ChangeLog.1 or whatever, then started with a new ChangeLog generated from scratch from Git messages. I use the gnulib build-aux/gitlog-to-changelog script to do it. It requires a little bit of discipline to get right; in particular you have to remember that the Git commit message will be indented 8 spaces in the ChangeLog, so you have to be careful that your commit messages wrap at char 70 (or less) in your Git commit. If you have Git hooks you could enforce a bit of formatting; for example any line not indented by space must be <=70 chars long; this allows people to use long lines for formatted content if they indent it with a space or two. Otherwise, it's the same as writing the ChangeLog and you only have to do it once. Just to note, the above script simply transcribes the commit message into ChangeLog format. It does NOT try to auto-generate ChangeLog- style content (files that changed, functions, etc.) from the Git diff or whatever. There are a few special tokens you can add to your Git commit message that get reformated to special changelog tokens like "(tiny change)" etc. As mentioned previously, it's very important that the commit message be provided as part of the code review, and it is very much fair game for review comments. This is common practice, and a good idea because bad commit messages are always a bummer, ChangeLog or not.
gcc-8-20200124 is now available
Snapshot gcc-8-20200124 is now available on https://gcc.gnu.org/pub/gcc/snapshots/8-20200124/ and on various mirrors, see http://gcc.gnu.org/mirrors.html for details. This snapshot has been generated from the GCC 8 git branch with the following options: git://gcc.gnu.org/git/gcc.git branch releases/gcc-8 revision 7175893ad8010df1270af884a1810476e51a8cd9 You'll find: gcc-8-20200124.tar.xzComplete GCC SHA256=1ef152ee8ec5956bd544becd70e89c42bc56b9a2ef017ce3f6cd2469873a5fcc SHA1=c82c7f4d17f7ddd4acf10144fb37a4ad82b8cb16 Diffs from 8-20200117 are available in the diffs/ subdirectory. When a particular snapshot is ready for public consumption the LATEST-8 link is updated and a message is sent to the gcc list. Please do not use a snapshot before it has been announced that way.
Re: Git ChangeLog policy for GCC Testsuite inquiry
On Fri, Jan 24, 2020 at 2:39 PM Paul Smith wrote: > > On Fri, 2020-01-24 at 22:45 +0100, Jakub Jelinek wrote: > > > > In my experience the output of git log is a total mess so cannot > > > > replace ChangeLogs. But we can well decide to drop ChangeLog for > > > > the testsuite. > > > > > > Well, glibc has moved to extracting them from git, building > > > policies and scripts around that. I'm pretty sure other > > > significant projecs are also extracting their ChangeLogs from git. > > > > > > We could do the same, selecting some magic date as the cutover > > > point after which future ChangeLogs are extracted from GIT. In > > > fact, that's precisely what I'd like to see us do. > > > > We don't have a tool that can do it, not even get the boilerplate > > right. Yes, mklog helps, but it very often gets stuff wrong. Not to > > mention that the text what actually changed can't be generated very > > easily. > > I don't know if it counts as a significant project, but GNU make has > been doing this for years. > > What I did was take the existing ChangeLogs and rename them to > ChangeLog.1 or whatever, then started with a new ChangeLog generated > from scratch from Git messages. > > I use the gnulib build-aux/gitlog-to-changelog script to do it. It > requires a little bit of discipline to get right; in particular you > have to remember that the Git commit message will be indented 8 spaces > in the ChangeLog, so you have to be careful that your commit messages > wrap at char 70 (or less) in your Git commit. > > If you have Git hooks you could enforce a bit of formatting; for > example any line not indented by space must be <=70 chars long; this > allows people to use long lines for formatted content if they indent it > with a space or two. > > Otherwise, it's the same as writing the ChangeLog and you only have to > do it once. > > Just to note, the above script simply transcribes the commit message > into ChangeLog format. It does NOT try to auto-generate ChangeLog- > style content (files that changed, functions, etc.) from the Git diff > or whatever. > > There are a few special tokens you can add to your Git commit message > that get reformated to special changelog tokens like "(tiny change)" > etc. > > As mentioned previously, it's very important that the commit message be > provided as part of the code review, and it is very much fair game for > review comments. This is common practice, and a good idea because bad > commit messages are always a bummer, ChangeLog or not. > Libgcrypt includes ChangeLog entries in git commit messages: http://git.gnupg.org/cgi-bin/gitweb.cgi?p=libgcrypt.git In each patch, commit log starts with ChangeLog entries without leading TABs followed by separator line with -- and then commit message. They have a script to extract ChangeLog for release. -- H.J.
