Separate preprocess and compile: hack or feature?
Hi, In the build system I am working on we are looking at always performing the preprocessing and then C/C++ compilation as two separate gcc/g++ invocations. The main reason is support for distributed compilation but see here[1] for other reasons. I realize that tools like ccache/distcc have been relying on this for a while (though see the 'direct' mode in ccache and 'pump' in distcc). However, some compilers apparently do not support this (for example, VC; see the above link for details). So I wonder, in the context of GCC, if this is just a hack that happens to work "for now" or if this is a feature that is expected to continue to work? Also, has anyone seen/heard of any real-world issues with compiling preprocessed source code? [1] https://www.reddit.com/r/cpp/comments/6abi99/rfc_issues_with_separate_preprocess_and_compile/ Thanks, Boris
Re: Separate preprocess and compile: hack or feature?
On 11 May 2017 at 10:13, Boris Kolpackov wrote: > Hi, > > In the build system I am working on we are looking at always performing > the preprocessing and then C/C++ compilation as two separate gcc/g++ > invocations. The main reason is support for distributed compilation but > see here[1] for other reasons. > > I realize that tools like ccache/distcc have been relying on this for > a while (though see the 'direct' mode in ccache and 'pump' in distcc). > However, some compilers apparently do not support this (for example, > VC; see the above link for details). Here's a nickel kid, get yourself a better compiler ;-) http://dilbert.com/strip/1995-06-24 > So I wonder, in the context of GCC, if this is just a hack that happens > to work "for now" or if this is a feature that is expected to continue > to work? It's absolutely a supported feature. Why else do you think the manual would discuss so many options for preprocessing, and options for compiling preprocessed code? https://gcc.gnu.org/onlinedocs/gcc/Overall-Options.html seems quite explicit. > Also, has anyone seen/heard of any real-world issues with compiling > preprocessed source code? There are very occasional bugs where compiling the original source code behaves slightly differently, e.g. regarding what is considered a system header, but it usually only affects warnings or errors, not code generation. And nothing that would mean separate preprocessing and compilation shouldn't be relied on.
Re: [RFC] GCC 8 Project proposal: Extensions supporting C Metaprogramming, pseudo-templates
On 10 May 2017 at 23:14, Daniel Santos wrote: > Well my primary goal is programming with values that are constant in the > compiler. There is no language in any C specification (that I'm aware of) > for a "compile-time constant", but the concept is very important. So just > because some expression is a compile-time constant doesn't mean we morph > into a "constant expression" (as per the spec), even with > __attribute__((const)). The C standard says "An implementation may accept other forms of constant expressions." That means rather than inventing some "constprop" you could just extend GCC to treat more expressions involving constants as constant-expressions. There are enhancement requests for this in Bugzilla, because some compilers are already more catholic in what they accept in constant expressions (notably, Clang).
'make check' questions
I see from https://gcc.gnu.org/install/test.html that it's possible to run tests in parallel. I get the impression from gcc/Makefile that the check concerned has to be set up in the Makefile (in my build tree, configured with --target=x86_64-apple-darwin16 --enable-languages=c,c++,ada,fortran,objc,obj-c++ , I see both lang_checks and lang_checks_parallelized set empty). So, is it necessary for check-ada or check-acats to cope with being run in parallel (i.e., will they ever see GCC_RUNTEST_PARALLELIZE_DIR set?) Also in https://gcc.gnu.org/install/test.html, under what circumstances would a test report ERROR (the testsuite detected an error) or WARNING (the testsuite detected a possible problem)? For example, if a particular test that should compile & run has a build error, is that a FAIL or an ERROR?
Re: Separate preprocess and compile: hack or feature?
On Thu, May 11, 2017 at 11:06:13AM +0100, Jonathan Wakely wrote: > It's absolutely a supported feature. Why else do you think the manual > would discuss so many options for preprocessing, and options for > compiling preprocessed code? > https://gcc.gnu.org/onlinedocs/gcc/Overall-Options.html seems quite > explicit. > > > Also, has anyone seen/heard of any real-world issues with compiling > > preprocessed source code? > > There are very occasional bugs where compiling the original source > code behaves slightly differently, e.g. regarding what is considered a > system header, but it usually only affects warnings or errors, not > code generation. And nothing that would mean separate preprocessing > and compilation shouldn't be relied on. Especially in recent GCC versions the amount of differences for warnings and errors keeps dramatically increasing, with separate preprocessing simply too much information is lost (macro contexts, lint style comments, exact locations, system header issues, ...). So it is far better to not use -E, but instead -E -fdirectives-only for the preprocessing step, which will get you also single file with all the includes in it, but macros, comments etc. are still in there. Tried to explain that to the ccache people, but they aren't listening. Jakub
Re: [RFC] GCC 8 Project proposal: Extensions supporting C Metaprogramming, pseudo-templates
On Thu, May 11, 2017 at 11:12:24AM +0100, Jonathan Wakely wrote: > On 10 May 2017 at 23:14, Daniel Santos wrote: > > Well my primary goal is programming with values that are constant in the > > compiler. There is no language in any C specification (that I'm aware of) > > for a "compile-time constant", but the concept is very important. So just > > because some expression is a compile-time constant doesn't mean we morph > > into a "constant expression" (as per the spec), even with > > __attribute__((const)). > > The C standard says "An implementation may accept other forms of > constant expressions." That means rather than inventing some > "constprop" you could just extend GCC to treat more expressions > involving constants as constant-expressions. > > There are enhancement requests for this in Bugzilla, because some > compilers are already more catholic in what they accept in constant > expressions (notably, Clang). Right, at least: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=69960 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66618 Marek
Re: 'make check' questions
On 11/05/17 11:43, Simon Wright wrote: I see from https://gcc.gnu.org/install/test.html that it's possible to run tests in parallel. I get the impression from gcc/Makefile that the check concerned has to be set up in the Makefile (in my build tree, configured with --target=x86_64-apple-darwin16 --enable-languages=c,c++,ada,fortran,objc,obj-c++ , I see both lang_checks and lang_checks_parallelized set empty). So, is it necessary for check-ada or check-acats to cope with being run in parallel (i.e., will they ever see GCC_RUNTEST_PARALLELIZE_DIR set?) I don't usually build Ada, but testing with "make -j check" works for me where is the parallelism I want Also in https://gcc.gnu.org/install/test.html, under what circumstances would a test report ERROR (the testsuite detected an error) or WARNING (the testsuite detected a possible problem)? For example, if a particular test that should compile & run has a build error, is that a FAIL or an ERROR? ERROR results are usually problems with the testsuite infrastructure, like misformed DejaGNU directives. They don't usually appear in a clean test run. If a test fails to build due to a compiler problem i.e. an ICE or other bug it will be a FAIL. If the test harness has a problem with the testsuite directives syntax I think it will be reported as ERROR. I usually get WARNINGs if a runtime tests times out. It can happen when testing against simulators or if the test was miscompiled into an infinite loop. Kyrill
Re: 'make check' questions
On Thu, May 11, 2017 at 11:50:21AM +0100, Kyrill Tkachov wrote: > > On 11/05/17 11:43, Simon Wright wrote: > > I see from https://gcc.gnu.org/install/test.html that it's possible to run > > tests in parallel. I get the impression from gcc/Makefile that the check > > concerned has to be set up in the Makefile (in my build tree, configured > > with --target=x86_64-apple-darwin16 > > --enable-languages=c,c++,ada,fortran,objc,obj-c++ , I see both lang_checks > > and lang_checks_parallelized set empty). So, is it necessary for check-ada > > or check-acats to cope with being run in parallel (i.e., will they ever see > > GCC_RUNTEST_PARALLELIZE_DIR set?) > > I don't usually build Ada, but testing with "make -j check" works for me > where is the parallelism I want I have export MAKEFLAGS='-j4' in my environment. Marek
Re: Separate preprocess and compile: hack or feature?
On Thu, May 11, 2017 at 12:43 PM, Jakub Jelinek wrote: > On Thu, May 11, 2017 at 11:06:13AM +0100, Jonathan Wakely wrote: >> It's absolutely a supported feature. Why else do you think the manual >> would discuss so many options for preprocessing, and options for >> compiling preprocessed code? >> https://gcc.gnu.org/onlinedocs/gcc/Overall-Options.html seems quite >> explicit. >> >> > Also, has anyone seen/heard of any real-world issues with compiling >> > preprocessed source code? >> >> There are very occasional bugs where compiling the original source >> code behaves slightly differently, e.g. regarding what is considered a >> system header, but it usually only affects warnings or errors, not >> code generation. And nothing that would mean separate preprocessing >> and compilation shouldn't be relied on. > > Especially in recent GCC versions the amount of differences for warnings and > errors keeps dramatically increasing, with separate preprocessing simply > too much information is lost (macro contexts, lint style comments, exact > locations, system header issues, ...). > > So it is far better to not use -E, but instead -E -fdirectives-only > for the preprocessing step, which will get you also single file with all the > includes in it, but macros, comments etc. are still in there. > > Tried to explain that to the ccache people, but they aren't listening. For the very same reason -g3 debuginfo is also not what you expect (all macros are gone). Richard. > Jakub
Re: Separate preprocess and compile: hack or feature?
Hi Jakub, Jakub Jelinek writes: > Especially in recent GCC versions the amount of differences for warnings and > errors keeps dramatically increasing, with separate preprocessing simply > too much information is lost (macro contexts, lint style comments, exact > locations, system header issues, ...). > > So it is far better to not use -E, but instead -E -fdirectives-only > for the preprocessing step, which will get you also single file with all the > includes in it, but macros, comments etc. are still in there. Uh, thanks, that's very helpful. One use case for removing comments is detecting and ignoring comments-only changes (via a checksum). But that's relatively minor. We could probably re-process the result ourselves and strip the comments for that (will have to recognize comments for the C++ module import detection anyway). Also, preserving comments helps with some VC issues (e.g., a/**/b). > Tried to explain that to the ccache people, but they aren't listening. I am, on the hand, all ears ;-). Thanks, Boris
Re: 'make check' questions
On 11 May 2017, at 11:50, Kyrill Tkachov wrote: > On 11/05/17 11:43, Simon Wright wrote: >> I see from https://gcc.gnu.org/install/test.html that it's possible to run >> tests in parallel. I get the impression from gcc/Makefile that the check >> concerned has to be set up in the Makefile (in my build tree, configured >> with --target=x86_64-apple-darwin16 >> --enable-languages=c,c++,ada,fortran,objc,obj-c++ , I see both lang_checks >> and lang_checks_parallelized set empty). So, is it necessary for check-ada >> or check-acats to cope with being run in parallel (i.e., will they ever see >> GCC_RUNTEST_PARALLELIZE_DIR set?) > > I don't usually build Ada, but testing with "make -j check" works for me > where is the parallelism I want So it does. Will have to review the results to see how it's done in the Ada test scripts - deeply unclear to me at the moment. >> Also in https://gcc.gnu.org/install/test.html, under what circumstances >> would a test report ERROR (the testsuite detected an error) or WARNING (the >> testsuite detected a possible problem)? For example, if a particular test >> that should compile & run has a build error, is that a FAIL or an ERROR? > > ERROR results are usually problems with the testsuite infrastructure, like > misformed DejaGNU directives. They don't usually appear in a clean test run. > If a test fails to build due to a compiler problem i.e. an ICE or other bug > it will be a FAIL. If the test harness has a problem with the testsuite > directives > syntax I think it will be reported as ERROR. > > I usually get WARNINGs if a runtime tests times out. It can happen when > testing against simulators or if the test was miscompiled into an infinite > loop. OK, I see, thanks.
Re: Separate preprocess and compile: hack or feature?
Hi Boris In the build system I am working on we are looking at always performing the preprocessing and then C/C++ compilation as two separate gcc/g++ invocations. The main reason is support for distributed compilation but see here[1] for other reasons. How c++ modules fit into a build system is currently an open question. Richard Smith & I have talked about it, but with no firm conclusion. However, I think that breaking out the preprocessor is not the right answer. We were leaning more to some kind of hook in the compiler to answer 'I can't find module FOO, where is it?' nathan -- Nathan Sidwell
Re: [RFC] GCC 8 Project proposal: Extensions supporting C Metaprogramming, pseudo-templates
On Thu, 11 May 2017, Jonathan Wakely wrote: > On 10 May 2017 at 23:14, Daniel Santos wrote: > > Well my primary goal is programming with values that are constant in the > > compiler. There is no language in any C specification (that I'm aware of) > > for a "compile-time constant", but the concept is very important. So just > > because some expression is a compile-time constant doesn't mean we morph > > into a "constant expression" (as per the spec), even with > > __attribute__((const)). > > The C standard says "An implementation may accept other forms of > constant expressions." That means rather than inventing some > "constprop" you could just extend GCC to treat more expressions > involving constants as constant-expressions. Note that while "other forms" might be accepted in initializers, they would still not be integer constant expressions (see DR#312). -- Joseph S. Myers jos...@codesourcery.com
Re: Separate preprocess and compile: hack or feature?
Hi Nathan, Nathan Sidwell writes: > How c++ modules fit into a build system is currently an open question. > Richard Smith & I have talked about it, but with no firm conclusion. > However, I think that breaking out the preprocessor is not the right > answer. Handling modules is one of the least important motivations for my desire to "break out" the preprocessor. Distributed compilation is probably the main one. But I am quickly realizing that this is probably not going to be reliable enough. But for completeness, let me describe how all the pieces would have fitted together. I think it is quite elegant, even if I have to say so myself ;-). Let's say the build system realizes (for example, based on filesystem mtimes) that hello.o may be out-of-date. This is what it does: 1. In a single pass preprocess hello.cxx and extract header dependencies (-E -MD options). Note that if you do -M you are essentially running the preprocessor, so might as well save the result. 2. If any of the extracted headers are auto-generated and are missing or out-of-date, regenerate them. 3. Pre-parse the preprocessed output and extract module dependency information. If any are missing/out-of-date, compile them. What we need to detect here are (1) module imports and (2) module implementation units. All these things are "top level" and it won't be hard to recognize. 4. If nothing above has indicated that hello.o is indeed out of date, hash the preprocessed output to detect and ignore comment-only changes. 5. If we indeed need to update hello.o, pass the preprocessed output either to the compiler or ship it to a remote host for compilation. Boris
gcc-7-20170511 is now available
Snapshot gcc-7-20170511 is now available on ftp://gcc.gnu.org/pub/gcc/snapshots/7-20170511/ and on various mirrors, see http://gcc.gnu.org/mirrors.html for details. This snapshot has been generated from the GCC 7 SVN branch with the following options: svn://gcc.gnu.org/svn/gcc/branches/gcc-7-branch revision 247933 You'll find: gcc-7-20170511.tar.bz2 Complete GCC SHA256=7de3863eba9cb41ef0fb3d6e5e6f1b3578125a88b5d383a521e26520c498efb5 SHA1=bae49f64788ebec566ce364aaf04ff20a4af6abd Diffs from 7-20170504 are available in the diffs/ subdirectory. When a particular snapshot is ready for public consumption the LATEST-7 link is updated and a message is sent to the gcc list. Please do not use a snapshot before it has been announced that way.
