On 08/27/2014 07:55 AM, Gregory Szorc wrote: > On 8/27/2014 7:39 AM, Benjamin Smedberg wrote: >> On 8/27/14, 10:22 AM, Gregory Szorc wrote: >>> Also, for the JS-centric use case you mentioned, if we invested in those >>> parts of the build system, we could probably get full tree builds with >>> no binary changes to under 10s. e.g. |mach build nonbinaries|. >> Sure. But since we don't have that yet, let's make sure that thing we >> did have keeps working? Is returning to the prior behavior by making >> `mach build dir` recursive a complex fix? > The context that may be missing here is that the build system no longer > does directory traversal like it once did. At moz.build evaluation time, > we essentially build a DAG of all the binaries and construct directory > traversal rules from that: actual directory structure conceptually has > very little to do with things. In reality, the order on the filesystem > matches the derived dependencies because that's how things always > worked. But this won't always hold! Viewing the build as an iteration of > directories in their filesystem layout is antiquated and this practice > needs to further go away to make the build faster. > > Again, I want to provide something. But making it purely derived on > directory layout is artificially constricting to making the build > faster. It ties us to the old world and prevents us from moving forward.
Maybe it's just me, but I'm having a lot of trouble following this thread. Can someone spell out exactly what use cases we're talking about here? Because I've heard several. Enumerating some of them: 1. I touched a file or files. Compile everything within a directory hierarchy that uses that file. Do not link. 2. I touched a file or files. Compile everything that uses that file. Do not link. 3. I touched a file or files. Compile and link everything. 4. Frankenbuild: I touched a file or files. Compile everything within a directory hierarchy that uses that file, and link the compiled files into libraries and binaries. I accept the possibility that I may have object files linked together that use different versions of some of my source files. That is an intentional tradeoff for build speed. 5. I touched a file or files, but they're JS or something that does not trigger compilation. Rebuild the minimum necessary to rerun with the updated files. That's just a sampling. There are other possibilities. Maybe it can be broken down into (a) What changed? Does it require recompilation? (b) Where do we look for stuff to rebuild from? Only stuff in a single directory, only stuff within a hierarchy, only stuff within specific directories or hierarchies, or anything anywhere in the tree that would be affected by the changes? (c) Where are the target files? Are we regenerating anything that needs updating in the tree (even if possibly from a subset of its changed source files)? Or just stuff leading to libxul.so (or whatever)? (d) How should the changes be handled? Compile for syntax only? Linked (potentially into frankenbuilds if the source 'where' (b) is less than the full tree)? But I don't know if that's a useful breakdown. (c) includes intermediate files as well as final outputs. Not all build rules are simple dependencies. Is dumbmake a set of rules for deriving (c)? If you touch directory A, then to get a fully working binary then you need to regenerate things whose build rules live under directory B, C, and D? If you consider the full tree as the "real" view (what |mach build| uses), then is it accurate/useful to consider all of the use cases as full builds using subsets of this view? So you might say "I know that I only changed files within this hierarchy; do a full rebuild but go faster by not even looking for changes anywhere else." (It matters whether you're restricting "root" changes to that hierarchy or changes anywhere -- eg, if you change toolkit/teddybearemulation/*.cpp then it will update libxul.so; should it relink the binary or is that out of scope?) Or you could say "There may be changes all over the place; pretend like no roots were changed except for the ones within this hierarchy" resulting in a (fast) frankenbuild. And then there's the question of the scope that you pull all the other rules from, the ones that aren't simple dependencies of one file on a set of others. I guess I'm stumbling towards some sort of setup where you have the full view, or you pretend you do (possibly using dumbmake-style heuristics to cover up the pretense, or generating module dependencies automagically during an earlier full tree build and then heuristically relying on them not changing). Then you filter it by only considering things with sources and/or targets within specified regions, and construct your (unfortunately still necessary) directory traversal order from that restricted DAG. Does it take too long to load in the full DAG? Could it be accelerated by applying the filters early? Given that I'm not volunteering to help with this, I'll shut up now. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
