Hi -
Having stewed over it for a bit to make things more concrete, I'm
thinking about the overall profile-crowdsource-LTO dataflow like this:
profiling --> profile database --> consensus profile --> linker
and name the project "profiledb" because why not.
= Profiling
Profile your own workload. If preferred, compile your binaries with
profiling instrumentation, or run in unmodified sample-collection
mode. LLVM autofdo project native output may not need conversion;
others may. We will also provide some conversion tools (e.g. from
gprof) and self-contained baby profilers (e.g. from elfutils) to
generate such format data directly with lightweight prereqs. Heck,
maybe a tool for producing llvm format output from the polyglot
gprof2dot's .dot format output.
Our main emphasis would be static+dynamic *call graphs*, because
those are likely relatively stable across runs & builds than
e.g. line-by-line hit-count profiles.
= Profile database
A new public git repo, say "profiledb.git", would welcome
submissions of LLVM sampled-profile-text format data files. These
would be collected in distinct subdirectories and/or branches,
containing the profile data and a bit of json metadata about the
binary being instrumented. The intent is that multiple profiles of
approximately the same binary can coexist, and be located near each
other with a naming convention. For example:
file naming convention in "submitted" branch:
$package-version/$binary-basename/UUID/foo.profdata # "llvm sampled text
format"
$package-version/$binary-basename/UUID/metadata.json # submitter or
reviewer weight etc.
example:
gcc-15/cc1plus/47db9098-68a4-11f0-a48a-047c161bdfe3/gmon.profdata
gcc-15/cc1plus/47db9098-68a4-11f0-a48a-047c161bdfe3/metadata.json
{ "architecture":"x86_64",
"buildid":"deadbeef",
"packaging":
{"type":"rpm","name":"coreutils","version":"9.6-5.fc42","archit\
ecture":"x86_64","osCpe":"cpe:/o:fedoraproject:fedora:42"},
"weight":4.0 }
= Profile consensus
Periodically, a tool would traverse the submitted data, and use
llvm-profdata or equivalent to merge all the submitted profiles for
a given binary into a single consensus one. These consensus
profiles would live in a designated branch of the profiledb. (Other
scripts could periodically age out old data.) For example:
branch "consensus":
$package-version/$binary-basename/consensus.profdata
example:
gcc-15/cc1plus/consensus.profdata
= Linker
profiledb consensus-branch trees would constitute directly
linker-consumable data. We would teach gnu binutils-ld to accept
these files for sediment-like function-section reordering. We would
package snapshots of the consensus branch in distributable noarch
tarballs, for installation as build-prereqs for distro builds.
(Think tzdata.) Developers can of course maintain their own local
or shared profiledb if they like. The linkers would find the
appropriate call-graph profile to consult:
C*FLAGS += -ffunction-sections # possibly made a distro default
cc1plus_LDFLAGS += --profiledb=/path/to/profiledb --profile=gcc-15/cc1plus
or, if profiledb is in Standard place and linker can infer binary-name
from -o OUTPUT:
LDFLAGS += --profile=gcc-15
Does this sound generally sensible?
- FChE
