I have a large project whose build process consists of running some independent sub-build steps within several hundred directories. As a preparatory step, we copy each of these directories into a temporary build location:
# collect a list of all source files sources := $(shell find src -mindepth 2 -type f) define build_component # drop a flag that indicates a successful build build/.$(1): build/$(1) [actual build process goes here] touch $$@ # the build directory depends upon each of the individual files within it build/$(1): $$(patsubst src/%,build/%,$(filter src/$(1)/%,$(sources)) # every file in each build subdirectory depends on its corresponding file # in the src directory, but they can all be copied into place in one step $$(patsubst src/%,build/%,$(filter src/$(1)/%,$(sources))) &: $$(filter src/$(1)/%,$(sources)) rm -rf build/$(1) cp -a src/$(1) build/$(1) .SECONDARY: build/.$(1) build/$(1) .SECONDARY: $$(patsubst src/%,build/%,$(filter src/$(1)/%,$(sources)) ended # generate the build steps for each component $(foreach component,$(components),$(eval $(call build_component,$(component)))) We jump through all these hoops for some important reasons. The build process often moves and/or removes files inside of the directories it’s working with. The above rules ensures that if the build is interrupted or fails a some point, if any files are missing for some component of the build, we’ll blow away that directory and start from scratch. It also ensures that each component is rebuilt if any of its individual sources changes. This is working well, but it doesn’t perform particularly well thanks to hundreds of independent calls to cp that could theoretically be merged into one. This is where I’m running into problems. I rewrote the patsubst rule and its corresponding entry in .SECONDARY: # every file in the build directory depends on its corresponding file in # thee src directory, but they can all be copied into place in one step $$(patsubst src/%,build/%,$(sources)) &: $(sources) rm -rf build cp -a src build .SECONDARY: $$(patsubst src/%,build/%,$(sources)) When run in one thread, this works as expected. It cuts down significantly on the overall build speed. When run in parallel, though, it unfortunately causes a massive slowdown. It seems that make isn’t efficiently sharing analysis info amongst parallel workers. It does function as expected, but debug output shows that tens of millions of Pruning file steps are generated before any work is performed. I think the fundamental issue is that this rule doesn’t quite express what I want. The above rule says, roughly, that “each file in the build directory depends upon *all* the files in the source directory”, and so a freshness check for each individual file in the build fans out to a freshness check on every file in the source. This needs to be done over and over and over. Is it possible to have our cake and eat it too? Is there a way to write a rule that expresses something closer to “each file in the build directory depends upon its corresponding file in the source directory” but still uses grouped targets? Note: The above rules were simplified for clarity from what’s in our actual build system. I may have typo’d something in the process. I apologize preemptively if there winds up being any confusion caused by this.
