Hello, This is an update of the patch set that I initially posted to http://gcc.gnu.org/ml/gcc-patches/2010-12/msg00858.html.
The main goals achieved by this set are the following: - Decrease the overall memory consumption. On the tests I have done on a reasonably big C++ program compiled with the previous patch set, I have noticed a global memory consumption peak increase of 13%, for the whole compilation. With this new patch set, the global memory consumption peak is down to 0.6%. Details of the figures are appended at the end of this message. To do this, I have quite heavily modified the patch "Generate virtual locations for tokens" and I have added the patch "Reduce memory waste due to non-power-of-2 allocs". The changes of these patches are explained in their respective preambles. - Change the diagnostics output as suggested by Paolo and Gaby in the sub-thread http://gcc.gnu.org/ml/gcc-patches/2010-12/msg00987.html. Besides, I have hopefully addressed most of the comments that were made in the initial discussion thread. Gaby suggested that I drop the -ftrack-macro-expansion flag that I have added to enable this feature. Now that the memory consumption has been reduce to something that is more acceptable, I guess this goal is closer now. However, I have kept the flag for now as I believe there is value in keeping it to ease testing and performance comparison measures on real life examples. When I am comfortable with those aspects, I'll happily remove that flag, e.g, after a period of enabling it by default. Please find below a summary of the patches of this set as well as the details of the memory consumption gain I measured. Linemap infrastructure for virtual locations Generate virtual locations for tokens Emit macro expansion related diagnostics Support -fdebug-cpp option Add line map statistics to -fmem-report output Kill pedantic warnings on system headers macros Reduce memory waste due to non-power-of-2 allocs gcc/Makefile.in | 2 +- gcc/ada/gcc-interface/trans.c | 10 +- gcc/c-decl.c | 17 +- gcc/c-family/c-lex.c | 10 +- gcc/c-family/c-opts.c | 17 + gcc/c-family/c-pch.c | 2 +- gcc/c-family/c-ppoutput.c | 98 ++- gcc/c-family/c.opt | 12 + gcc/c-parser.c | 12 +- gcc/c-tree.h | 2 +- gcc/cp/error.c | 5 +- gcc/diagnostic.c | 25 +- gcc/diagnostic.h | 2 +- gcc/doc/cppopts.texi | 29 + gcc/doc/invoke.texi | 6 +- gcc/fortran/cpp.c | 22 +- gcc/input.c | 105 ++- gcc/input.h | 22 +- gcc/java/jcf-parse.c | 2 +- gcc/testsuite/g++.dg/cpp0x/initlist15.C | 1 + gcc/testsuite/g++.old-deja/g++.robertl/eb43.C | 4 + gcc/testsuite/g++.old-deja/g++.robertl/eb79.C | 4 + gcc/testsuite/gcc.dg/cpp/macro-exp-tracking-1.c | 21 + gcc/testsuite/gcc.dg/cpp/macro-exp-tracking-2.c | 21 + gcc/testsuite/gcc.dg/cpp/macro-exp-tracking-3.c | 14 + gcc/testsuite/gcc.dg/cpp/macro-exp-tracking-4.c | 14 + gcc/testsuite/gcc.dg/cpp/pragma-diagnostic-1.c | 32 + gcc/testsuite/gcc.dg/cpp/pragma-diagnostic-2.c | 34 + gcc/testsuite/gcc.dg/cpp/syshdr3.c | 16 + gcc/testsuite/gcc.dg/cpp/syshdr3.h | 7 + gcc/testsuite/gcc.dg/nofixed-point-2.c | 6 +- gcc/testsuite/gcc.target/i386/sse-vect-types.c | 6 + gcc/toplev.c | 4 + gcc/tree-diagnostic.c | 181 +++- gcc/tree-diagnostic.h | 3 +- libcpp/directives-only.c | 7 +- libcpp/directives.c | 21 +- libcpp/errors.c | 21 +- libcpp/expr.c | 176 ++-- libcpp/files.c | 24 +- libcpp/include/cpp-id-data.h | 6 + libcpp/include/cpplib.h | 15 +- libcpp/include/line-map.h | 827 +++++++++++++-- libcpp/init.c | 5 +- libcpp/internal.h | 60 +- libcpp/lex.c | 107 ++- libcpp/line-map.c | 831 ++++++++++++-- libcpp/macro.c | 1350 ++++++++++++++++++++--- libcpp/traditional.c | 7 +- 49 files changed, 3656 insertions(+), 569 deletions(-) create mode 100644 gcc/testsuite/gcc.dg/cpp/macro-exp-tracking-1.c create mode 100644 gcc/testsuite/gcc.dg/cpp/macro-exp-tracking-2.c create mode 100644 gcc/testsuite/gcc.dg/cpp/macro-exp-tracking-3.c create mode 100644 gcc/testsuite/gcc.dg/cpp/macro-exp-tracking-4.c create mode 100644 gcc/testsuite/gcc.dg/cpp/pragma-diagnostic-1.c create mode 100644 gcc/testsuite/gcc.dg/cpp/pragma-diagnostic-2.c create mode 100644 gcc/testsuite/gcc.dg/cpp/syshdr3.c create mode 100644 gcc/testsuite/gcc.dg/cpp/syshdr3.h Below are the details of the memory consumption I have measured with both the previous and current versions of this patch set. [ For abbreviations, there is a legend at the end of the document ] Memory peak, and GC memory UAEOC increase incurred by macro tokens location tracking, before the work: |pre-processing only |full compilation| ------------------------|-----------------------|----------------| % memory peak increase | 19.9%' | 8% | % GC memory UAEOC | 283% | 9.82% | Memory peak increase incurred by macro tokens location tracking, after the work: |pre-processing only |full compilation| ------------------------|-----------------------|----------------| % memory peak increase | 19.2%' | 0.6% | % GC memory UAEOC | 250.5% | 4.9% | Number of expanded macros: 96524 Average number of tokens per macro expansion: 21 |C0-p-n |C0-p-t |C0-f-n |C0-f-t |C1-p-n |C1-p-t |C1-f-n |C1-f-t | ---------------------------------------------------------------------------------------- maps allocated |310K |21M |310K |21M |255K |20M |255K |20M | maps used |213K |20M |213K |20M |213K |20M |213K |20M | macro maps used |0 |19M |0 |19M |0 |19M |0 |19M | map allocation waste |374K |10.7M |374 |10.7M |64 |4.35M |64 |4.35M | GC memory used at EOC |9.9M |38M |285M |313M |9.7M |34M |285M |299M | G.M.U.A.O.E overhead |140K |396K |4M |4.3M |140K |386K |4M |4M | total GC mem allocation |11M |47M |617M |653.7M |10M |36M |616.5M |642.6M | total GC mem overhead |1M |11M |54M |64.31M |919K |5.2M |53.5M |58M | peak memory usage |166M |199M |658M |711MB |158,3M |188.7M |658M |662,4M | Legend: ======= C0-p-n: Code base 0 (before this work), for pre-processing only, without macro tokens location tracking activated. C0-p-t: Code base 0 (before this work), for pre-processing only, with macro tokens location tracking activated. C0-f-n: Code base 0 (before this work), for full compilation, without macro tokens location tracking activated. C0-f-t: Code base 0 (before this work), for full compilation, with macro tokens location tracking activated. C1-p-n: Code base 1 (after this work), for pre-processing only, without macro tokens location tracking activated. C1-p-t: Code base 1 (after this work), for pre-processing only, with macro tokens location tracking activated. C1-f-n: Code base 1 (after this work), for full compilation, without macro tokens location tracking activated. C1-f-t: Code base 1 (after this work), for full compilation, with macro tokens location tracking activated. EOC : End of compilation. G.M.U.A.O.E : GC memory still used at end of compilation. UAEOC : Used at end of compilation
