Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-16 Thread Tom Lane
Peter Eisentraut writes: > On mån, 2010-11-15 at 23:34 -0500, Tom Lane wrote: >> It's clear to me that we are very far from having a handle on what >> it'll really take to run parallel builds safely, and I am therefore >> now of the opinion that we ought to revert the patch. > We don't have to r

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-16 Thread Peter Eisentraut
On mån, 2010-11-15 at 23:34 -0500, Tom Lane wrote: > It's clear to me that we are very far from having a handle on what > it'll really take to run parallel builds safely, and I am therefore > now of the opinion that we ought to revert the patch. Hypothetical > gains in parallelism are useless if w

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-15 Thread Tom Lane
I tried another experiment, which was "make -j100 all" on my relatively new Linux box (2 dual-core CPUs). It blew up real good, as per attached stderr output, which shows evidence of more missing dependencies as well as some additional cases of concurrent attempts to build the same target. It's c

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-15 Thread Tom Lane
Robert Haas writes: > Very odd, but this completely blew up the first time I tried it. > In file included from path.c:34: > pg_config_paths.h:2:11: error: missing terminating " character FWIW, I didn't replicate that, but I did get this during one attempt with -j4: /usr/bin/ranlib: archive memb

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-15 Thread Robert Haas
On Mon, Nov 15, 2010 at 4:10 PM, Peter Eisentraut wrote: >> ld: file not found: ../../../../../../src/backend/postgres >> collect2: ld returned 1 exit status >> make[3]: *** [ascii_and_mic.so] Error 1 >> make[2]: *** [all-ascii_and_mic-recurse] Error 2 >> make[1]: *** [all-backend/utils/mb/convers

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-15 Thread Peter Eisentraut
On mån, 2010-11-15 at 11:13 +0100, Bernd Helmle wrote: > > --On 14. November 2010 11:08:13 -0500 Robert Haas > wrote: > > > +1. The current master branch fails to build on my (rather new) Mac with > > make -j2. I could upgrade my toolchain but it seems like more trouble > > than it's worth, n

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-15 Thread Bernd Helmle
--On 14. November 2010 11:08:13 -0500 Robert Haas wrote: +1. The current master branch fails to build on my (rather new) Mac with make -j2. I could upgrade my toolchain but it seems like more trouble than it's worth, not to mention a possible obstacle to new users and developers. The sa

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-14 Thread Robert Haas
On Sun, Nov 14, 2010 at 12:13 PM, Tom Lane wrote: > I wrote: >> I still think it's worth looking into whether the bug can be dodged >> by shortening the eval calls. > > In fact, that does seem to work; I'll commit a patch after testing a > bit more. > > We still need someone to add the missing bui

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-14 Thread Tom Lane
I wrote: > I still think it's worth looking into whether the bug can be dodged > by shortening the eval calls. In fact, that does seem to work; I'll commit a patch after testing a bit more. We still need someone to add the missing build dependencies so that make -j is trustworthy again.

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-14 Thread Tom Lane
Robert Haas writes: > On Nov 14, 2010, at 10:44 AM, Tom Lane wrote: >> I still think it's worth looking into whether the bug can be dodged >> by shortening the eval calls. But if not, I think I'd vote for >> reverting. Maybe we could revisit this in a couple of years. > +1. The current master

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-14 Thread Robert Haas
On Nov 14, 2010, at 10:44 AM, Tom Lane wrote: > I still think it's worth looking into whether the bug can be dodged > by shortening the eval calls. But if not, I think I'd vote for > reverting. Maybe we could revisit this in a couple of years. +1. The current master branch fails to build on my

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-14 Thread Andrew Dunstan
On 11/14/2010 10:44 AM, Tom Lane wrote: And on the fourth hand, what we're buying here is pretty marginal for developers and of no interest whatever for users. I still think it's worth looking into whether the bug can be dodged by shortening the eval calls. But if not, I think I'd vote for

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-14 Thread Tom Lane
Dave Page writes: > On Sat, Nov 13, 2010 at 8:13 PM, Peter Eisentraut wrote: >> Well, it looks like $(eval) is pretty broken in 3.80, so either we >> require 3.81 or we abandon this line of thought. > 3.81 might be a problem for Solaris - unless I pay for a support > contract from Oracle, I'm no

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-14 Thread Dave Page
On Sat, Nov 13, 2010 at 8:13 PM, Peter Eisentraut wrote: > On lör, 2010-11-13 at 20:07 +0200, Peter Eisentraut wrote: >> On lör, 2010-11-13 at 12:20 -0500, Tom Lane wrote: >> > Peter Eisentraut writes: >> > > On lör, 2010-11-13 at 11:12 -0500, Tom Lane wrote: >> > >> It looks like all the unhappy

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Tom Lane
Peter Eisentraut writes: > Well, it looks like $(eval) is pretty broken in 3.80, so either we > require 3.81 or we abandon this line of thought. [ emerges from some grubbing about in the gmake sources... ] It looks to me like the bug in 3.80 is only triggered when "eval" expands to a long enough

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Peter Eisentraut
On lör, 2010-11-13 at 20:07 +0200, Peter Eisentraut wrote: > On lör, 2010-11-13 at 12:20 -0500, Tom Lane wrote: > > Peter Eisentraut writes: > > > On lör, 2010-11-13 at 11:12 -0500, Tom Lane wrote: > > >> It looks like all the unhappy critters are getting the same "virtual > > >> memory exhausted"

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Peter Eisentraut
On lör, 2010-11-13 at 12:20 -0500, Tom Lane wrote: > Peter Eisentraut writes: > > On lör, 2010-11-13 at 11:12 -0500, Tom Lane wrote: > >> It looks like all the unhappy critters are getting the same "virtual > >> memory exhausted" error. I wonder whether they are all using make > >> 3.80 ... > >

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Erik Rijkers
On Sat, November 13, 2010 18:15, Peter Eisentraut wrote: > On lör, 2010-11-13 at 11:12 -0500, Tom Lane wrote: >> It looks like all the unhappy critters are getting the same "virtual >> memory exhausted" error. I wonder whether they are all using make >> 3.80 ... > > It turns out that there is an

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Tom Lane
Peter Eisentraut writes: > On lör, 2010-11-13 at 11:12 -0500, Tom Lane wrote: >> It looks like all the unhappy critters are getting the same "virtual >> memory exhausted" error. I wonder whether they are all using make >> 3.80 ... > It turns out that there is an unrelated bug in 3.80 that some

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Peter Eisentraut
On lör, 2010-11-13 at 11:23 -0500, Tom Lane wrote: > Consulting stdout shows that indeed it's launched this series of jobs: > > make -C backend/utils/mb/conversion_procs all > make -C ascii_and_mic all > gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith > -Wdeclaration-after-statement -Wendif-lab

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Peter Eisentraut
On lör, 2010-11-13 at 11:12 -0500, Tom Lane wrote: > It looks like all the unhappy critters are getting the same "virtual > memory exhausted" error. I wonder whether they are all using make > 3.80 ... It turns out that there is an unrelated bug in 3.80 that some Linux distributions have patched a

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Peter Eisentraut
On lör, 2010-11-13 at 11:06 -0500, Andrew Dunstan wrote: > But I don't feel we need to squeeze every last pip out of > the build system. Probably not on the buildfarm, but when you are developing, saving 20 seconds or 2 minutes per cycle can lead to hours saved. -- Sent via pgsql-hackers maili

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Dave Page
On Sat, Nov 13, 2010 at 4:12 PM, Tom Lane wrote: > It looks like all the unhappy critters are getting the same "virtual > memory exhausted" error.  I wonder whether they are all using make 3.80 Both my Sparc and Intel Solaris critters have 3.80. -- Dave Page Blog: http://pgsnake.blogspot.com

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Andrew Dunstan
On 11/13/2010 11:12 AM, Tom Lane wrote: It looks like all the unhappy critters are getting the same "virtual memory exhausted" error. I wonder whether they are all using make 3.80 ... Maybe we need to put back make version logging. Interestingly, narwhal, the mingw machine that has reported

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Tom Lane
BTW, there's another problem here: "make -j2" on my Mac blows up with this on stderr: ld: file not found: ../../../../../../src/backend/postgres collect2: ld returned 1 exit status make[3]: *** [ascii_and_mic.so] Error 1 make[2]: *** [all-ascii_and_mic-recursive] Error 2 make[1]: *** [all-backend/

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Tom Lane
Andrew Dunstan writes: > I'm curious to know how much all this buys us. It *would* be nice if "make -k" worked better. I frequently run into the fact that (with the pre-existing setup) a compile error in the backend prevented make from proceeding with builds of interfaces/, bin/, etc, meaning th

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Andrew Dunstan
On 11/12/2010 11:25 PM, Tom Lane wrote: Andrew Dunstan writes: On 11/12/2010 03:16 PM, Peter Eisentraut wrote: Improved parallel make support Looks like this patch has pretty comprehensively broken the MSVC build system. I'll see what I can recover from the wreckage. There are also at leas

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-12 Thread Tom Lane
Andrew Dunstan writes: > On 11/12/2010 03:16 PM, Peter Eisentraut wrote: >> Improved parallel make support > Looks like this patch has pretty comprehensively broken the MSVC build > system. I'll see what I can recover from the wreckage. There are also at least three non-Windows buildfarm member