Re: [modules] Preprocessing requires compiled header unit modules
Iain Sandoe writes: > The standard has the concept of an “importable header” which is > implementation-defined. But it must contain all the C++ library headers: https://eel.is/c++draft/headers#4 > We could choose that only headers that are self-contained (i.e. unaffected > by external defines) are “importable” (thus the remaining headers would > not be eligible for include-translation). That would mean that we could > rely on processing any import by processing the header it is created from? > Perhaps that is too great a restriction and we need to be more clever. It will also be hard to determine whether the header (or any header that it includes) satisfies this condition. You would probably want it to be "meaningfully self-contained" (since pretty much every header is not self-contained with regards to include guards) which I think will be hard to automate.
Re: [modules] Preprocessing requires compiled header unit modules
Ben Boeckel writes: > If we need to know and have dependencies prepared before we can figure > out the dependencies for a TU, modules are unsolvable (without an active > build executor). If C++ implementations are really going to require > that, then [...] the following tools are all unsuitable for C++ with > header units without major overhauls (alphabetical): > > - autoconf/automake > - cmake > - gn > - gyp > - make (not GNU make, though even that requires some active > involvement via the socket communications) > - meson > - ninja A couple of points: 1. Firstly, this only applies to header units, not named modules. 2. I am not sure what you mean by "active build executor" (but it does sound ominous, I will grant you that ;-)). 3. I agree some build systems may require "major overhauls" to support header units via the module mapper. I would like this not to be the case, but so far nobody has implemented an alternative (that I am aware of) that is correct and scalable and I personally have doubts such a thing is achievable. > > Even if we manage to do this, there are some implications I > > am not sure we will like: the isolated macros will contain > > inclusion guards, which means we will keep re-scanning the > > same files potentially many many time. Here is an example, > > assume each header-unitN.hpp includes or imports : > > Note that scanning each module TU only happens once. Header units might > just get *read* in the course of scanning other units. > > And headers are read multiple times already over the lifetime of the > build, so we're not making things worse here. I am not sure I follow. Say we have 10 TUs each include or import 10 headers each of which includes . If we use include, then when scanning each of these 10 TUs we have to scan once (since all the subsequent includes are suppressed by include guards). So total of 10x1=10 scans of for the entire build. Now if instead of include we use import (which, during the scan, is treated as include with macro isolation), we are looking at 10 scans of for each TU (because the include guards are ignored). So total of 10x10=100 scans of for the build. What am I missing?
Re: [modules] Preprocessing requires compiled header unit modules
On Mon, Apr 25, 2022 at 11:42:14 +0200, Boris Kolpackov wrote: > 1. Firstly, this only applies to header units, not named modules. Correct. > 2. I am not sure what you mean by "active build executor" (but it >does sound ominous, I will grant you that ;-)). One that does more than schedule builds, but is "actively" participating in the build itself. Ninja's `dyndep` is about as close one can be to the edge IMO. `make` is potentially active because of its behavior around `include` of output files, but this is a niche feature. > 3. I agree some build systems may require "major overhauls" to >support header units via the module mapper. I would like this >not to be the case, but so far nobody has implemented an >alternative (that I am aware of) that is correct and scalable >and I personally have doubts such a thing is achievable. One the compilers can scan headers, CMake should be able to do it. MSVC is the closest right now (though the name it wants for any given header usage site is currently an issue). > Ben Boeckel writes: > > Note that scanning each module TU only happens once. Header units might > > just get *read* in the course of scanning other units. > > > > And headers are read multiple times already over the lifetime of the > > build, so we're not making things worse here. > > I am not sure I follow. Say we have 10 TUs each include or import > 10 headers each of which includes . If we use include, > then when scanning each of these 10 TUs we have to scan > once (since all the subsequent includes are suppressed by include > guards). So total of 10x1=10 scans of for the entire > build. > > Now if instead of include we use import (which, during the scan, is > treated as include with macro isolation), we are looking at 10 scans > of for each TU (because the include guards are ignored). > So total of 10x10=100 scans of for the build. > > What am I missing? Ew, indeed. I suppose a sufficiently smart compiler could just encode headers as a "diff" to the preprocessor and symbol state and apply it upon revisiting, but that seems like a high bar (re-reading if it detects a difference in preprocessor state that "matters"). --Ben
Re: [committed] exec-stack warning for test which wants executable stacks
On 4/24/22 19:42, Jeff Law via Gcc wrote: > About a week ago many targets started failing pr94157_0.c test like this > (bfin-elf, but many other targets are also affected): > >> spawn -ignore SIGHUP /home/jlaw/test/obj/bfin-elf/obj/gcc/gcc/xgcc >> -B/home/jlaw/test/obj/bfin-elf/obj/gcc/gcc/ c_lto_pr94157_0.o >> -fdiagnostics-plain-output -dumpbase -O0 -fipa-vrp -flto -Wa,--noexecstack >> -Wa,--noexecstack -Wa,--execstack -Wa,--execstack -Wa,--execstack >> -Wa,--execstack -Wa,--execstack -Wa,--execstack -Wa,--execstack >> -Wa,--execstack -Wa,--execstack -Wa,--execstack -Wa,--execstack >> -Wa,--execstack -Wa,--execstack -Wa,--execstack -msim -Wl,-wrap,exit >> -Wl,-wrap,_exit -Wl,-wrap,main -Wl,-wrap,abort -Wl,gcc_tg.o -o >> gcc-dg-lto-pr94157-01.exe^M >> /home/jlaw/test/obj/bfin-elf/installed/bfin-elf/bin/ld: warning: >> /tmp/ccfJUEvZ.ltrans0.ltrans.o: requires executable stack (because the >> .note.GNU-stack section is executable)^M >> FAIL: gcc.dg/lto/pr94157 c_lto_pr94157_0.o-c_lto_pr94157_0.o link, -O0 >> -fipa-vrp -flto -Wa,--noexecstack -Wa,--noexecstack -Wa,--execstack >> -Wa,--execstack -Wa,--execstack -Wa,--execstack -Wa,--execstack >> -Wa,--execstack -Wa,--execstack -Wa,--execstack -Wa,--execstack >> -Wa,--execstack -Wa,--execstack -Wa,--execstack -Wa,--execstack >> -Wa,--execstack > > This is due to a new binutils warning. This patch just suppresses the > warning for the one test where we explicitly wanted an executable stack. > > I'm guessing the repeated -Wa,--noexecstack options in this test are supposed > to trigger a buffer overflow or something similar, so I left those alone and > just appended to the argument list. Yes. > > I used -z execstack rather than --no-warn-execstack as the former is > recognized by older versions of ld, but the latter is a new option. Thanks for it. Martin > > The other approach would have been to prune the warning, but this seemed > better since we'd like most tests to fail if somehow their stacks were > executable. > > > Committed to the trunk. > > Jeff >
Re: [committed] exec-stack warning for test which wants executable stacks
On 4/25/2022 6:56 AM, Martin Liška wrote: I used -z execstack rather than --no-warn-execstack as the former is recognized by older versions of ld, but the latter is a new option. Thanks for it. Unfortunately, I should have looked at the other failures that have popped up over the last week. Essentially all the nested function tests are failing on some targets due to the same linker warning. Either pruning or adding the option to all those tests is going to be necessary ;( jeff
Re: [committed] exec-stack warning for test which wants executable stacks
Hi Jeff, I used -z execstack rather than --no-warn-execstack as the former is recognized by older versions of ld, but the latter is a new option. Thanks for it. Unfortunately, I should have looked at the other failures that have popped up over the last week. Essentially all the nested function tests are failing on some targets due to the same linker warning. Either pruning or adding the option to all those tests is going to be necessary ;( Sorry for causing you so much hassle. If it helps, Alan has already added some pruning code to binutils/testsuite/lib/binutils-common.exp in the prune_extra_warnings proc... Cheers Nick
Re: [committed] exec-stack warning for test which wants executable stacks
On 4/25/2022 8:42 AM, Nick Clifton wrote: Hi Jeff, I used -z execstack rather than --no-warn-execstack as the former is recognized by older versions of ld, but the latter is a new option. Thanks for it. Unfortunately, I should have looked at the other failures that have popped up over the last week. Essentially all the nested function tests are failing on some targets due to the same linker warning. Either pruning or adding the option to all those tests is going to be necessary ;( Sorry for causing you so much hassle. No worries. It's a very reasonable warning. And it's only if you're testing with the latest binutils that it causes problems -- which is one of the goals of the tester -- to find this stuff when it happens, not 6-12 months later. If it helps, Alan has already added some pruning code to binutils/testsuite/lib/binutils-common.exp in the prune_extra_warnings proc... Yea. I saw that. I ultimately decided against pruning as it would be useful to know if a test suddenly needs execstack. That would generally be a sign of something gone horribly wrong. I just (locally) added the magic to the rest of the affected tests and I'm testing it now. jeff
Re: [committed] exec-stack warning for test which wants executable stacks
Hi Jeff, Just FYI - I am also looking at adding in another warning. This time for when the linker creates a PT_LOAD segment which has all of the RWX flags set. At the moment my testing seems to show that it only causes problems when a custom linker script is used that defines its own program headers and does not provide separate headers for code and data. Cheers Nick
Re: [committed] exec-stack warning for test which wants executable stacks
On 4/25/2022 9:26 AM, Nick Clifton wrote: Hi Jeff, Just FYI - I am also looking at adding in another warning. This time for when the linker creates a PT_LOAD segment which has all of the RWX flags set. At the moment my testing seems to show that it only causes problems when a custom linker script is used that defines its own program headers and does not provide separate headers for code and data. That also sounds quite reasonable. My biggest worry would be the embedded targets which have their own linker scripts -- but like the executable stack warning, I think you should go for it and we'll deal with the fallout. jeff
