On 5/20/21 2:54 PM, Richard Biener wrote:
On Thu, May 20, 2021 at 2:34 PM Martin Liška <mli...@suse.cz> wrote:
Hello.
I've got a patch candidate that leverages partial linking for a couple of
selected object files.
I'm sending make all-host- jX results for my machine:
before: 3m18s (user 32m52s)
https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg
after: 2m57m (user 35m)
https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg
One can utilize it with:
make -j16 all-host PARTIAL_LTO=1
@Segher, Andrew: Can you please measure time improvement for your slow
bootstrap?
One can also tweak --param=lto-partitions=16 param value.
Thoughts?
You're LTO linking multiple objects here - that's almost as if you
were doing this
for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the
libbackend.a rule do a similar partial link trick.
Yeah, apart from that one can't likely do partial linking for an archive:
$ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r
libbackend.a
collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core
dumped
compilation terminated.
while ld.bfd immediately finishes.
That gets you half of a LTO bootstrap then.
So why did you go from applying this per-file to multiple files? Does $(LINKER)
have a proper rule to pick up a jobserver?
When upstreaming in any form you probably have to gate it on bootstrap-lto
being not active.
Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE),
right?
One more thing I face is broken dependency:
$ make clean && make -j32 PARTIAL_LTO=1
g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions
-fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings
-Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute
-Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros
-Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I.
-I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/.
-I/home/marxin/Programming/gcc/gcc/../include
-I/home/marxin/Programming/gcc/gcc/../libcpp/include
-I/home/marxin/Programming/gcc/gcc/../libcody
-I/home/marxin/Programming/gcc/gcc/../libdecnumber
-I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber
-I/home/marxin/Programming/gcc/gcc/../libbacktrace -o gimple-match-lto.o -MT
gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c
-flto
g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions
-fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings
-Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute
-Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros
-Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I.
-I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/.
-I/home/marxin/Programming/gcc/gcc/../include
-I/home/marxin/Programming/gcc/gcc/../libcpp/include
-I/home/marxin/Programming/gcc/gcc/../libcody
-I/home/marxin/Programming/gcc/gcc/../libdecnumber
-I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber
-I/home/marxin/Programming/gcc/gcc/../libbacktrace -o generic-match-lto.o -MT
generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c
-flto
In file included from ./tm.h:26,
from /home/marxin/Programming/gcc/gcc/backend.h:28,
from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23,
from generic-match.c:4:
/home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error:
insn-attr-common.h: No such file or directory
2286 | #include "insn-attr-common.h"
| ^~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:2678: generic-match-lto.o] Error 1
make: *** Waiting for unfinished jobs....
In file included from ./tm.h:26,
from /home/marxin/Programming/gcc/gcc/backend.h:28,
from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23,
from gimple-match.c:4:
/home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error:
insn-attr-common.h: No such file or directory
2286 | #include "insn-attr-common.h"
| ^~~~~~~~~~~~~~~~~~~~
I explicitly added:
gimple-match.o: gimple-match.c $(generated_files)
generic-match.o: generic-match.c $(generated_files)
But it's not obeyed.
Martin
Richard.
Thanks,
Martin
>From b209c7aea76ceb8eadcf5843f30299fc8041a579 Mon Sep 17 00:00:00 2001
From: Martin Liska <mli...@suse.cz>
Date: Thu, 20 May 2021 14:29:35 +0200
Subject: [PATCH] Try LTO partial linking.
---
gcc/Makefile.in | 33 +++++++++++++++++++++++++++++----
1 file changed, 29 insertions(+), 4 deletions(-)
diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index 1164554e6d6..decfea7412b 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -220,7 +220,9 @@ libgcov-util.o-warn = -Wno-error
libgcov-driver-tool.o-warn = -Wno-error
libgcov-merge-tool.o-warn = -Wno-error
gimple-match.o-warn = -Wno-unused
+gimple-match-lto.o-warn = -Wno-unused
generic-match.o-warn = -Wno-unused
+generic-match-lto.o-warn = -Wno-unused
dfp.o-warn = -Wno-strict-aliasing
# All warnings have to be shut off in stage1 if the compiler used then
@@ -1282,12 +1284,10 @@ ANALYZER_OBJS = \
# will build them sooner, because they are large and otherwise tend to be
# the last objects to finish building.
OBJS = \
- gimple-match.o \
- generic-match.o \
+ common-base.a \
insn-attrtab.o \
insn-automata.o \
insn-dfatab.o \
- insn-emit.o \
insn-extract.o \
insn-latencytab.o \
insn-modes.o \
@@ -1295,7 +1295,6 @@ OBJS = \
insn-output.o \
insn-peep.o \
insn-preds.o \
- insn-recog.o \
insn-enums.o \
ggc-page.o \
adjust-alignment.o \
@@ -2627,6 +2626,32 @@ s-match: build/genmatch$(build_exeext) $(srcdir)/match.pd cfn-operators.pd
generic-match.c
$(STAMP) s-match
+gimple-match.o: gimple-match.c $(generated_files)
+generic-match.o: generic-match.c $(generated_files)
+
+ifdef PARTIAL_LTO
+LTO_LINKER_FLAGS = -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r
+LTO_FLAGS = -flto
+
+gimple-match-lto.o: gimple-match.c
+ $(COMPILE) $< $(LTO_FLAGS)
+generic-match-lto.o: generic-match.c
+ $(COMPILE) $< $(LTO_FLAGS)
+insn-recog-lto.o: insn-recog.c
+ $(COMPILE) $< $(LTO_FLAGS)
+insn-emit-lto.o: insn-emit.c
+ $(COMPILE) $< $(LTO_FLAGS)
+
+common-base.a: gimple-match-lto.o generic-match-lto.o insn-recog-lto.o insn-emit-lto.o
+ -rm -rf $@
+ $(LINKER) $^ $(LTO_LINKER_FLAGS) -o common-base.o
+ $(AR) $(AR_FLAGS)T $@ common-base.o
+else
+common-base.a: gimple-match.o generic-match.o insn-recog.o insn-emit.o
+ -rm -rf $@
+ $(AR) $(AR_FLAGS)T $@ $^
+endif
+
GTFILES = $(CPPLIB_H) $(srcdir)/input.h $(srcdir)/coretypes.h \
$(host_xm_file_list) \
$(tm_file_list) $(HASHTAB_H) $(SPLAY_TREE_H) $(srcdir)/bitmap.h \
--
2.31.1
