On Mon, 28 May 2007, Michel Talon wrote:
Stephen Montgomery-Smith said:
I suggest rewriting "make" so that variables are only evaluated on a
"need to know" basis. ....
or "I have tried to do this."
Of course a lot of people have thinked about it, and quickly realized
that it was not going to work. In the bsd.ports.mk, evaluation of one
variable may be dependent on some conditional, which may itself be
dependant on some other variable, appearing as some target. This
constantly happens in bsd.ports.mk.
If you think about that, you convince yourself that a "reduced make"
needs to understand targets, needs to understand conditionals, and needs
to evaluate not only the variable under consideration, but but possibly
any other. In other words, you need a full blown make.
Actually I have thought quite a lot about this over the last couple of
weeks. You are correct in that the "make" really does have to be a full
blown make. What I am suggesting is something rather different - a rather
sophisticated make that works hard to only that which it really has to do.
And it certainly would require a lot of sophistication, precisely for the
reasons you state.
To gain some performance, a first idea would be to simplify
bsd.ports.mk. I am convinced that a substantial part of the 4000 lines
are historical crap which serve no useful purpose. There are tons of
variables who have probably purely anecdotical interest. There are
targets which could be happily suppressed. Of course, due to the
complexity of bsd.ports.mk, rewriting it is certainly not an easy task.
There is a freebsd port whose aim is to rewrite it, i don't know how
far they are. The NetBSD people have completely rewritten the system, i
have no idea if the new one is faster. The OpenBSD people have also
rewritten bsd.ports.mk, with apparently much faster makefile.
I rewrote a bsd.ports.mk in which ALL targets were removed, and all it did
was evalute variables. (I wrote a perl script that did this
automatically.) The net effect was no speed gain. In other words,
processing unnecessary targets is not the problem.
I really have looked at bsd.ports.mk a lot, and given that it is written
in the context of what make requires, it seems to be rather well and
efficiently written. I did find one speed increase in bsd.gnome.mk, which
I am kind of proud of, but even that only gained about a 5-10% speed
increase in "make index."
A second idea would be to multithread make, since modern machines will
have a lot of cores. At present make -j <something> forks submakes
and waits for their completion. This doesn't help for the problem at
hand. However, multithreading the execution of a single makefile is
certainly not trivial due to the interdependencies.
I don't think multithreaded make will help in this situation.
Anyways, one of the things which cannot be a big factor is reading
bsd.ports.mk from hard disk. It is certainly cached in memory when you
run make index. On the other hand it is so big that it probably doesn't
fit in cache, or probably only fits in caches of machines generously
endowed. I have remarked that the difference of execution speed of make
index between machines of similar speed but very different cache size
(i am thinking Pentium 4 versus Core 2 Duo) is striking. It is less
than 10 minutes on the big cache machine versus 30 minutes on the small
cache one. If the cache effect is indeed dominant, then reducing the
size of bsd.ports.mk by all possible means would be very beneficial.
My suspicion is that the timne taken to read bsd.ports.mk from the hard
disk is a rather small part of what takes it so long. Compare doing a
"make" with a simple "cat" and the time differences are very great. I
think make spends far more time processing the file than reading it.
By the way an alternative system would be to use something other than
make to do the job. This is what the Gentoo people are doing, using
first a python based system, and now a C++ one (paludis). Here also i
don't have any idea if it is faster.
It would be nice to find a solution that doesn't involve redesigning the
whole ports process from scratch.
--
Michel TALON
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
