Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-10-05 Thread David Boyce
On Mon, Sep 30, 2013 at 3:12 PM, Frank Heckenbach wrote: > I tested the new version and found no new issues, so as far as I'm > concerned this feature can now be considered finished. Thanks for > the initial patch, David, and the integration into GNU make, Paul! > And special thanks to Frank and

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-09-30 Thread Frank Heckenbach
I tested the new version and found no new issues, so as far as I'm concerned this feature can now be considered finished. Thanks for the initial patch, David, and the integration into GNU make, Paul! ___ Bug-make mailing list Bug-make@gnu.org https://lis

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-09-29 Thread Paul Smith
As always thanks for your thorough testing Frank. On Tue, 2013-09-24 at 20:41 +0200, Frank Heckenbach wrote: > Paul Smith wrote: > > > On Thu, 2013-09-19 at 14:47 +0200, Frank Heckenbach wrote: > > > Paul Smith wrote: > > > > > > > I didn't fix it this way. Instead I used the existing MAKE_REST

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-09-25 Thread Frank Heckenbach
Edward Welbourne wrote: > >> All but one of those I've looked into is manifestly less powerful than > >> make (and I'm still waiting for my former colleagues to get the > >> exception released to open source). > > > I have no illusions that there aren't very serious issues with make and > > I hav

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-09-24 Thread Frank Heckenbach
Paul Smith wrote: > On Thu, 2013-09-19 at 14:47 +0200, Frank Heckenbach wrote: > > Paul Smith wrote: > > > > > I didn't fix it this way. Instead I used the existing MAKE_RESTART > > > environment variable to communicate from the current make to the > > > restarted make that the enter message was

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-09-21 Thread Edward Welbourne
> No, that wouldn't work. It's not the individual command (between > simicolons) that's too long, the problem is that make can't invoke the > shell itself because the command line + environment is too large. The > only way to work around this limitation is to avoid invoking a single > command tha

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-09-21 Thread Paul Smith
On Thu, 2013-09-19 at 14:47 +0200, Frank Heckenbach wrote: > Paul Smith wrote: > > > *sigh* If it weren't for the enter/leave messaging, the output-sync > > feature would have been quite straightforward! :-/ :-). > > I'm afraid so. But I think we're almost there now. I've pushed what I hope are

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-09-21 Thread Paul Smith
On Sat, 2013-09-21 at 07:28 +, Edward Welbourne wrote: > > I've never understood why someone would use $(shell ...) in a recipe... > > I mean, the recipe will be run in the shell!! > > I remember we once had a library where the command-line to the archiver > was too long (about a quarter megab

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-09-21 Thread Edward Welbourne
> I've never understood why someone would use $(shell ...) in a recipe... > I mean, the recipe will be run in the shell!! I remember we once had a library where the command-line to the archiver was too long (about a quarter megabyte, IIRC). We worked round this by having a temporary scratch dir,

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-09-20 Thread Frank Heckenbach
Paul Smith wrote: > On Thu, 2013-09-19 at 14:47 +0200, Frank Heckenbach wrote: > > > Hm. This is pretty contrived. I have a hard time imagining a real > > > makefile wanting to do this for a good reason. However, it does > > seem > > > that the solution may be simple enough. > > > > I also dou

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-09-19 Thread Paul Smith
On Thu, 2013-09-19 at 14:47 +0200, Frank Heckenbach wrote: > > Hm. This is pretty contrived. I have a hard time imagining a real > > makefile wanting to do this for a good reason. However, it does > seem > > that the solution may be simple enough. > > I also doubt someone would do it intentiona

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-09-19 Thread Frank Heckenbach
Paul Smith wrote: > *sigh* If it weren't for the enter/leave messaging, the output-sync > feature would have been quite straightforward! :-/ :-). I'm afraid so. But I think we're almost there now. > On Mon, 2013-09-16 at 12:18 +0200, Frank Heckenbach wrote: > > > 1. > > > > Twice "Entering",

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-09-18 Thread Paul Smith
Hi Frank, thanks for the thorough testing! This feature will be much better for all your efforts. *sigh* If it weren't for the enter/leave messaging, the output-sync feature would have been quite straightforward! :-/ :-). On Mon, 2013-09-16 at 12:18 +0200, Frank Heckenbach wrote: > 1. > > %

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-09-16 Thread Frank Heckenbach
(I'm afraid I hadn't seen your message from Sep 12, only your followup on Sep 14, so I'm a bit late ...) Paul Smith wrote: > > One of the issues you were running into with your makefile is that > > output from functions like $(info ...) that were expanded inside the > > recipe were simply being p

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-09-16 Thread Eli Zaretskii
> From: Paul Smith > Cc: e...@gnu.org, david.s.bo...@gmail.com, bug-make@gnu.org > Date: Thu, 12 Sep 2013 04:19:13 -0400 > > I've reworked the output methods for make a good bit. Hopefully the > code is somewhat simpler and more understandable. Sorry for the larger > changes, but it seemed nece

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-09-13 Thread Paul Smith
On Thu, 2013-09-12 at 04:19 -0400, Paul Smith wrote: > I think there may still be some change needed for directory tracking > for the -Orecurse mode. If we're collecting output for an entire > recursive make invocation we don't need enter/leave notifications > around each individual target or line

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-09-12 Thread Paul Smith
On Wed, 2013-07-17 at 17:07 +0200, Frank Heckenbach wrote: > May I remind you of my mail of May 26: > http://lists.gnu.org/archive/html/bug-make/2013-05/msg00137.html > > Besides a little cleanup patch, I show in this mail that > "--trace=dir" is not completely sufficient as currently implemented

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-07-17 Thread Frank Heckenbach
May I remind you of my mail of May 26: http://lists.gnu.org/archive/html/bug-make/2013-05/msg00137.html Besides a little cleanup patch, I show in this mail that "--trace=dir" is not completely sufficient as currently implemented (including test case). I realize that you don't like the full tracin

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-05-25 Thread Frank Heckenbach
I've tested the current output-sync implementation in my use case, using "--output-sync=line --trace=dir". I definitely need "--trace=dir" since I want to refer my compiler errors automatically to their directories. Without this option, this often goes wrong in parallel builds. (Which just means,

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-05-04 Thread Paul Smith
On Sun, 2013-05-05 at 04:37 +0200, Frank Heckenbach wrote: > > COMMANDS_RECURSE _does_ mean to recurse. The reason for the '+' > > prerequisite is to tell make that this line, even though it may not look > > like it, will run a recursive make. > > OK, let me just say that the meaning of "recursiv

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-05-04 Thread Frank Heckenbach
Paul Smith wrote: > The first one I've seen but hadn't had time to debug. I'll look at your > patch. I left the truncate where it was rather than doing it after the > sync_output() because I was hoping to avoid truncating a file that we'll > never use again anyway, but I guess it isn't a big dea

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-05-04 Thread Paul Smith
On Sat, 2013-05-04 at 08:57 +0200, Frank Heckenbach wrote: > I shouldn't have written that. :-( Shortly afterwards, I found a bug > or perhaps two: > > foo: > @echo foo > +@echo bar > > (a) > % make -Ojob > foo > bar > foo > > (b) > % make -Otarget > bar > foo > > As you see, (a

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-05-04 Thread Eli Zaretskii
> Date: Sat, 04 May 2013 04:42:32 +0200 > Cc: e...@gnu.org, david.s.bo...@gmail.com, bug-make@gnu.org > From: Frank Heckenbach > > : /* This is needed to avoid the "label at end of compound statement" > : diagnostics on Posix platforms. */ > > I don't think that's a POSIX thing (which is

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-05-03 Thread Frank Heckenbach
: I've tested your changes and so far "-O job" works for my use cases. I shouldn't have written that. :-( Shortly afterwards, I found a bug or perhaps two: foo: @echo foo +@echo bar (a) % make -Ojob foo bar foo (b) % make -Otarget bar foo As you see, (a) "-Ojob" writes "foo" tw

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-05-03 Thread Frank Heckenbach
Paul Smith wrote: > I went a different way: I modified the child_error() function so that if > there was an output-sync file, the error message would be written to the > end of that file instead of printed directly. This way when the output > is shown to the user she sees the entire thing, includ

Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)

2013-05-03 Thread Frank Heckenbach
Eli Zaretskii wrote: > > From: Paul Smith > > Cc: bug-make@gnu.org > > Date: Sun, 28 Apr 2013 22:03:39 -0400 > > > > Now that we seem to have a workable solution for output synchronization > > for both POSIX and Windows systems, I wonder if we shouldn't consider > > enabling it as the default mo

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-05-03 Thread Frank Heckenbach
Paul Smith wrote: > On Sun, 2013-04-28 at 20:00 +0300, Eli Zaretskii wrote: > > > I've pushed a change to add a new argument to the -O/--output-sync > > > option, "job", to write output after each line of the recipe. > > > > What is its purpose? To avoid mixing in the same screen line > > charac

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-05-03 Thread Frank Heckenbach
David Boyce wrote: > On Sun, Apr 28, 2013 at 1:34 AM, Paul Smith wrote: > > I'm not excited about that term ("job"); it's kind of accurate, but in > > the documentation for example we're really mushy about exactly what a > > "job" is, vs. a "recipe" or a "command line" etc. I'd like to pick some

Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)

2013-04-29 Thread Edward Welbourne
Eli: >> cc fred.c -c -o fred.o >> cc bob.c -c -o bob.o >> error on line 20 -X >> error on line 30 - >> error on line 330 - >> makefile:342: recipe for target 'fred.o' failed >> makefile:350: recipe for target 'bob.o' failed > You need to look in both anyway. That is true of the very s

Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)

2013-04-29 Thread Tim Murphy
One doesn't have to suffer the problems and learn the option exists afterwards. In the end I can understand why a new feature might not be default to start with - until a lot of people have used it and are sure that it works everywhere. Cheers, Tim On 29 April 2013 20:21, Eli Zaretskii wrote:

Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)

2013-04-29 Thread Eli Zaretskii
> Date: Mon, 29 Apr 2013 19:33:10 +0100 > From: Tim Murphy > Cc: Eli Zaretskii , "bug-make@gnu.org" > > Come now - the broken excuse is an excuse. There's plenty of crap free > software out there and some poor bastard trying to build it who can't > change the source because the people who own it

Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)

2013-04-29 Thread Eli Zaretskii
> Date: Mon, 29 Apr 2013 18:30:37 +0100 > From: Tim Murphy > Cc: "bug-make@gnu.org" > > cc fred.c -c -o fred.o > cc bob.c -c -o bob.o > error on line 20 -X > error on line 30 - > error on line 330 - > makefile:342: recipe for target 'fred.o' failed > makefile:350: recipe for target '

Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)

2013-04-29 Thread Tim Murphy
Come now - the broken excuse is an excuse. There's plenty of crap free software out there and some poor bastard trying to build it who can't change the source because the people who own it think it should be make's problem. :-) Cheers, Tim On 29 April 2013 19:00, Philip Guenther wrote: > On

Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)

2013-04-29 Thread Philip Guenther
On Mon, Apr 29, 2013 at 10:30 AM, Tim Murphy wrote: > cc fred.c -c -o fred.o > cc bob.c -c -o bob.o > error on line 20 -X > error on line 30 - > error on line 330 - > makefile:342: recipe for target 'fred.o' failed > makefile:350: recipe for target 'bob.o' failed > > ? "Doctor, my ham

Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)

2013-04-29 Thread Tim Murphy
cc fred.c -c -o fred.o cc bob.c -c -o bob.o error on line 20 -X error on line 30 - error on line 330 - makefile:342: recipe for target 'fred.o' failed makefile:350: recipe for target 'bob.o' failed ? Regards, Tim On 29 April 2013 18:25, Eli Zaretskii wrote: > > Date: Mon, 29 Apr

Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)

2013-04-29 Thread Eli Zaretskii
> Date: Mon, 29 Apr 2013 16:40:03 +0100 > From: Tim Murphy > Cc: "bug-make@gnu.org" > > cc fred.c -c -o fred.o > cc bob.c -c -o bob.o > error on line 20 -X > > Which one? Make will actually tell you which one, something like: makefile:342: recipe for target 'oo/i386/acl-errno-valid.o' f

Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)

2013-04-29 Thread Tim Murphy
On 29 April 2013 16:19, Eli Zaretskii wrote: > > Date: Mon, 29 Apr 2013 09:58:50 +0100 > > From: Tim Murphy > > Cc: "bug-make@gnu.org" > > > > try interpreting error messages from compiler/tool X when they're 10 > > lines from the file that they refer to and don't include the > > filename in th

Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)

2013-04-29 Thread Eli Zaretskii
> Date: Mon, 29 Apr 2013 09:58:50 +0100 > From: Tim Murphy > Cc: "bug-make@gnu.org" > > try interpreting error messages from compiler/tool X when they're 10 > lines from the file that they refer to and don't include the > filename in the error message. That's unrelated: interpreting such output

Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)

2013-04-29 Thread Tim Murphy
Let me add my voice as a user. If you are one of the lucky people whose builds consist mostly of 1 line of output per rule then you will rarely have any trouble in a good build but try interpreting error messages from compiler/tool X when they're 10 lines from the file that they refer to and don't

Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)

2013-04-28 Thread Eli Zaretskii
> From: Paul Smith > Cc: bug-make@gnu.org > Date: Sun, 28 Apr 2013 22:03:39 -0400 > > Now that we seem to have a workable solution for output synchronization > for both POSIX and Windows systems, I wonder if we shouldn't consider > enabling it as the default mode when parallel builds are running.

Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)

2013-04-28 Thread Paul Smith
Now that we seem to have a workable solution for output synchronization for both POSIX and Windows systems, I wonder if we shouldn't consider enabling it as the default mode when parallel builds are running. I understand that this will be a change that could be visible (beyond the collection of ou

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-28 Thread Paul Smith
On Thu, 2013-04-18 at 22:36 +0200, Frank Heckenbach wrote: > % make -Omake # same with -Otarget > m:2: recipe for target 'foo' failed > make: *** [foo] Error 1 > foo:error > > This seems at least strange to me: The conclusion "recipe failed" is > printed before the reason (the messages from the jo

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-28 Thread Paul Smith
On Sun, 2013-04-28 at 20:00 +0300, Eli Zaretskii wrote: > > I've pushed a change to add a new argument to the -O/--output-sync > > option, "job", to write output after each line of the recipe. > > What is its purpose? To avoid mixing in the same screen line > characters from several parallel sub-

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-28 Thread Eli Zaretskii
> Date: Sun, 28 Apr 2013 10:22:40 -0400 > From: David Boyce > Cc: Frank Heckenbach , Eli Zaretskii , > bug-make > > So I'd argue for: > > -O line (new) > -O job (current -O target) > -O make Agree about "line" (assuming I understood what it means), but disagree about renaming "target": it is

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-28 Thread Eli Zaretskii
> From: Paul Smith > Cc: e...@gnu.org, david.s.bo...@gmail.com, bug-make@gnu.org > Date: Sun, 28 Apr 2013 01:34:44 -0400 > > On Thu, 2013-04-18 at 22:36 +0200, Frank Heckenbach wrote: > > > > This is useful (to me) because at any time, I know what's running. > > > > ("[Start]" messages minus "[En

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-28 Thread David Boyce
On Sun, Apr 28, 2013 at 1:34 AM, Paul Smith wrote: > I'm not excited about that term ("job"); it's kind of accurate, but in > the documentation for example we're really mushy about exactly what a > "job" is, vs. a "recipe" or a "command line" etc. I'd like to pick some > terms for this, define th

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-27 Thread Paul Smith
On Thu, 2013-04-18 at 22:36 +0200, Frank Heckenbach wrote: > > > This is useful (to me) because at any time, I know what's running. > > > ("[Start]" messages minus "[End]" messages.) > > > > Thanks, this is the reason I was looking for; that use-case wasn't clear > > to me based on the previous em

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-27 Thread Paul Smith
On Sat, 2013-04-27 at 14:39 +0300, Eli Zaretskii wrote: > The changes needed to make -O work on MS-Windows are now committed to > the master repository, see commits da7df54 and 049f8e8. Please review > and comment. Thanks Eli!! I'll take a look over the next few days. _

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-27 Thread Paul Smith
On Sat, 2013-04-27 at 13:09 +0300, Eli Zaretskii wrote: > Is this intended behavior that these two messages: > > mkfsync:6: recipe for target 'two' failed > gnumake[1]: [two] Error 1 (ignored) > > are separated and wrapped by separate "Entering...Leaving" blocks, > instead of being produced t

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-27 Thread Eli Zaretskii
> Date: Sat, 27 Apr 2013 13:09:02 +0300 > From: Eli Zaretskii > Cc: f.heckenb...@fh-soft.de, bug-make@gnu.org > > > The basic feature can be tested trivially like this: > > > > all: one two > > > > one two: > > @echo start $@ > > @sleep 1 > > @echo stop $@ > >

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-27 Thread Eli Zaretskii
> From: Paul Smith > Cc: Frank Heckenbach , bug-make@gnu.org > Date: Sun, 21 Apr 2013 19:30:05 -0400 > > On Fri, 2013-04-19 at 14:09 +0300, Eli Zaretskii wrote: > > > Date: Fri, 19 Apr 2013 11:54:05 +0200 > > > Cc: bo...@kolpackov.net, bug-make@gnu.org > > > From: Frank Heckenbach > > > > > > >

RE: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-26 Thread Martin Dorey
> They surely store the uncompressed size somewhere NTFS certainly stores that. It's the field referred to as "real" at eg http://ftp.kolibrios.org/users/Asper/docs/NTFS/ntfsdoc.html#id4793056. ("allocated" is what it says on the tin. "initialized" would be smaller than "real" if it's compre

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-26 Thread Frank Heckenbach
Eli Zaretskii wrote: > You underestimate me ;-) > > What I have is actually this: > [...] > > and I wrote 'fcntl' emulation for Windows that uses a mutex. Indeed, I had not expected this. :-) > That said, I'm not wedded to the above approach, and if people like a > completely disjoint code for

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-26 Thread Eli Zaretskii
> Date: Thu, 25 Apr 2013 02:16:33 +0200 > Cc: e...@gnu.org, bug-make@gnu.org > From: Frank Heckenbach > > > > On Windows, you said fstat was very expensive, didn't you? Or is > > > lseek even worse? > > > > I think anything that potentially moves the file pointer can be > > sometimes expensive a

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-25 Thread Eli Zaretskii
> Date: Wed, 24 Apr 2013 21:19:45 +0300 > From: Eli Zaretskii > Cc: bug-make@gnu.org > > > Date: Wed, 24 Apr 2013 19:14:01 +0200 > > Cc: bug-make@gnu.org > > From: Frank Heckenbach > > > > > Testing clearly shows it doesn't: GetFileInformationByHandle simply > > > fails for handles open on cons

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-25 Thread Eli Zaretskii
> Date: Thu, 25 Apr 2013 05:14:41 +0200 > Cc: psm...@gnu.org, bug-make@gnu.org > From: Frank Heckenbach > > My suggestion was under the assumption that we use different > declarations because of different types anyway -- which I still > recommend, though Paul might have to decide this one. Yes,

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Frank Heckenbach
Eli Zaretskii wrote: > > Date: Thu, 25 Apr 2013 02:16:33 +0200 > > Cc: e...@gnu.org, bug-make@gnu.org > > From: Frank Heckenbach > > > > > > I can't follow you here. On POSIX, we don't need to pass a fd > > > > because it's always stdout/stderr. Or do mean something else? > > > > > > I meant th

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Eli Zaretskii
> Date: Thu, 25 Apr 2013 02:16:33 +0200 > Cc: e...@gnu.org, bug-make@gnu.org > From: Frank Heckenbach > > > > I can't follow you here. On POSIX, we don't need to pass a fd > > > because it's always stdout/stderr. Or do mean something else? > > > > I meant the file descriptor passed to fcntl to p

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Eli Zaretskii
> Date: Wed, 24 Apr 2013 22:55:59 +0100 > From: Tim Murphy > Cc: "bug-make@gnu.org" > > trying to pass kernel object handles around (seems a bit nasty to > me)? Why is that nasty? The handle is returned by a documented interface, and communicating it to another process is an officially documen

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Frank Heckenbach
Eli Zaretskii wrote: > > From: Paul Smith > > Cc: Frank Heckenbach , bug-make@gnu.org > > Date: Wed, 24 Apr 2013 16:03:56 -0400 > > > > > Or maybe we should abandon this optimization and take the lock > > > regardless. How bad can that be? > > > > Well, we want to know if the file has any cont

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Eli Zaretskii
> From: Paul Smith > Cc: Frank Heckenbach , bug-make@gnu.org > Date: Wed, 24 Apr 2013 16:03:56 -0400 > > > Or maybe we should abandon this optimization and take the lock > > regardless. How bad can that be? > > Well, we want to know if the file has any content anyway: for example we > don't wan

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Frank Heckenbach
Paul Smith wrote: > On Wed, 2013-04-24 at 20:46 +0200, Frank Heckenbach wrote: > > That's true about SEEK_CUR which was there originally. I actually > > changed it to SEEK_END, which does move the position to the end. > > Oh right. My head cold is keeping me foggy. What was the reason to > chan

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Paul Smith
On Wed, 2013-04-24 at 22:55 +0100, Tim Murphy wrote: > why not use a named semaphore wherever possible (windows and linux) > and lock a file where not instead of trying to pass kernel object > handles around (seems a bit nasty to me)? Hi Tim; I think you're late to the party :-). Let me summarize

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Tim Murphy
why not use a named semaphore wherever possible (windows and linux) and lock a file where not instead of trying to pass kernel object handles around (seems a bit nasty to me)? On 24 April 2013 21:19, Paul Smith wrote: > On Wed, 2013-04-24 at 22:39 +0300, Eli Zaretskii wrote: > > > Nothing is ac

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Paul Smith
On Wed, 2013-04-24 at 22:39 +0300, Eli Zaretskii wrote: > > Nothing is actually read by lseek (and even if it were, it would > > only need to look at the first and last part of the file, not read > > all the content, if that was the worry). > > Are you sure? How can lseek "jump" to the last byte

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Paul Smith
On Wed, 2013-04-24 at 22:25 +0300, Eli Zaretskii wrote: > > From: Paul Smith > > Cc: e...@gnu.org, bug-make@gnu.org > > Date: Wed, 24 Apr 2013 15:07:21 -0400 > > > > I'm not so sure fstat() is that cheap. struct stat contains a lot of > > information. Although I guess since we are only ever tal

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Eli Zaretskii
> Date: Wed, 24 Apr 2013 20:46:14 +0200 > Cc: p...@mad-scientist.net, bug-make@gnu.org > From: Frank Heckenbach > > > > I don't see why it would terminate prematurely > > > > It was long ago, but I presume I thought about the ^Z character that > > some programs write or interpret to signal end o

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Eli Zaretskii
> From: Paul Smith > Cc: e...@gnu.org, bug-make@gnu.org > Date: Wed, 24 Apr 2013 15:07:21 -0400 > > I'm not so sure fstat() is that cheap. struct stat contains a lot of > information. Although I guess since we are only ever talking about temp > files, not NFS files or something, it's probably n

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Paul Smith
I'm fully prepared to accept the blame for not doing the best job getting buy-in etc. on this effort. Can we leave the discussion on the process behind? I'd prefer that, unless there are real constructive comments on how to do better next time rather than rehashing what was done wrong. I think w

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Frank Heckenbach
Eli Zaretskii wrote: > > : Btw, there will be other side effects, at least on non-Posix > > : platforms, due to the fact that stuff that was supposed to go to the > > : screen is redirected to a file instead. Some programs sense that and > > : behave differently, e.g. with binary non-printable ch

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Paul Smith
On Wed, 2013-04-24 at 21:17 +0300, Eli Zaretskii wrote: > There's one issue that perhaps needs discussing. A mutex is > identified by a handle, which on Windows is actually a pointer to an > opaque object (maintained by the kernel). As such, using just 'int' > for sync_handle is not wide enough,

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Eli Zaretskii
> Date: Wed, 24 Apr 2013 10:34:20 -0700 > From: David Boyce > Cc: Frank Heckenbach , > "bug-make@gnu.org" > > On Wed, Apr 24, 2013 at 10:26 AM, Tim Murphy wrote: > > > I would suggest pretending that one has semaphores first with some nice > > little abstraction and then implementing th

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Eli Zaretskii
> Date: Wed, 24 Apr 2013 19:14:01 +0200 > Cc: bug-make@gnu.org > From: Frank Heckenbach > > > Testing clearly shows it doesn't: GetFileInformationByHandle simply > > fails for handles open on console devices and the null device. And we > > will be comparing handles for console devices in the maj

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Eli Zaretskii
> Date: Wed, 24 Apr 2013 18:50:15 +0200 > Cc: bug-make@gnu.org > From: Frank Heckenbach > > > That's one way. Another one is to discuss the design more thoroughly > > before the patches are accepted. > > I think it was discussed quite extensively. Also in retrospect, I > don't see how the desig

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread David Boyce
On Wed, Apr 24, 2013 at 10:26 AM, Tim Murphy wrote: > I would suggest pretending that one has semaphores first with some nice > little abstraction and then implementing them in the best way the platform > has to offer. > How about we introduce functions called acquire_semaphore() and release_se

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Tim Murphy
Some time ago when solving the same problem in a different way we used semaphores on Windows and Linux. Compatibility might make it less interesting but I would suggest pretending that one has semaphores first with some nice little abstraction and then implementing them in the best way the platform

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Frank Heckenbach
Eli Zaretskii wrote: > > I know about this method, but I'm not sure it does what we want here. > > The number used by this method is not guaranteed to be unique on some > > versions of Windows. More importantly, it only works for disk files, > > I don't know whether it reports a meaningful value

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Eli Zaretskii
> Date: Thu, 18 Apr 2013 21:57:56 +0300 > From: Eli Zaretskii > Cc: bug-make@gnu.org, bo...@kolpackov.net > > > Date: Thu, 18 Apr 2013 19:09:06 +0200 > > From: Frank Heckenbach > > > > > . calculation of combined_output in start_job_command will need to be > > >reimplemented for Windows, s

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Eli Zaretskii
> Date: Wed, 24 Apr 2013 05:56:49 +0300 > From: Eli Zaretskii > Cc: f.heckenb...@fh-soft.de, bug-make@gnu.org > > I will see if locking works on console handles; if not, I will have > to introduce a command-line argument for passing the name or the > handle of a mutex to a sub-Make. As I suspect

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-24 Thread Frank Heckenbach
Eli Zaretskii wrote: > That's one way. Another one is to discuss the design more thoroughly > before the patches are accepted. I think it was discussed quite extensively. Also in retrospect, I don't see how the design could have been much different (see below). > I tried to turn the discussion

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-23 Thread Eli Zaretskii
> From: Paul Smith > Cc: Frank Heckenbach , bug-make@gnu.org > Date: Tue, 23 Apr 2013 16:54:33 -0400 > > Without knowing what kind of resource Windows can take locks on, we > can't really know how to help with that. The only resources that don't need their names passed to sub-Make are the standa

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-23 Thread Frank Heckenbach
Eli Zaretskii wrote: > > Date: Tue, 23 Apr 2013 21:41:22 +0200 > > From: Frank Heckenbach > > > > Sure, it's excluding much more than necessary, but since > > the critical section is very short, this shouldn't hurt much. (In > > other words, if make jobs produce such huge output that copying it

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-23 Thread Paul Smith
On Tue, 2013-04-23 at 23:16 +0300, Eli Zaretskii wrote: > > All it requires is inheriting the redirected stdout/stderr to child > > processes. This was already possible under Dos (with the exception > > that since there was no fork, you had to redirect in the parent > > process, call the child, the

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-23 Thread Eli Zaretskii
> Date: Tue, 23 Apr 2013 21:41:22 +0200 > From: Frank Heckenbach > > Yes, as I wrote in another mail, even a completely global semaphore > should do. Not healthy, IMHO. Some snafu could inadvertently and completely silently stop an unrelated build somewhere on the same system. > Sure, it's exc

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-23 Thread Frank Heckenbach
David Boyce wrote: > The first thing is get the word "lock" out of your mind because we aren't > really locking anything. Yes, that API is in use but it's only to create a > semaphore or baton. Nobody is ever prevented from doing anything. It just > happens that on Unix the most portable (i.e. old

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-23 Thread Frank Heckenbach
(TL;DR: Probably irrelevant.) Paul Smith wrote: > I'm not sure if the lock locks the FD (so that if you dup'd the FD but > it still pointed to the same output, you could take exclusive locks on > both), or if it locks the underlying resource. If the former I guess > it's possible to break the sy

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-23 Thread David Boyce
On Tue, Apr 23, 2013 at 12:04 PM, Paul Smith wrote: > > I'm not sure if the lock locks the FD (so that if you dup'd the FD but > it still pointed to the same output, you could take exclusive locks on > both), or if it locks the underlying resource. I'm pretty sure it's the underlying resource th

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-23 Thread Eli Zaretskii
> From: Paul Smith > Cc: David Boyce , f.heckenb...@fh-soft.de, > bug-make@gnu.org > Date: Tue, 23 Apr 2013 15:04:39 -0400 > > When thinking about this, remember that it's not enough to consider how > a single make invocation will work. If you run with a single make > instance under -j, t

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-23 Thread Paul Smith
On Tue, 2013-04-23 at 21:40 +0300, Eli Zaretskii wrote: > > Date: Tue, 23 Apr 2013 11:29:35 -0700 > > From: David Boyce > > Cc: Frank Heckenbach , bug-make > > > > Since you asked basic questions I'm going to start this at a basic level. > > Apologies if it covers some stuff you already know or

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-23 Thread Eli Zaretskii
> Date: Tue, 23 Apr 2013 11:29:35 -0700 > From: David Boyce > Cc: Frank Heckenbach , bug-make > > Since you asked basic questions I'm going to start this at a basic level. > Apologies if it covers some stuff you already know or if I misinterpreted > the questions. Note that I haven't actually lo

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-23 Thread David Boyce
Since you asked basic questions I'm going to start this at a basic level. Apologies if it covers some stuff you already know or if I misinterpreted the questions. Note that I haven't actually looked at the patch that went in so this is generally wrt the original. The first thing is get the word "l

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-23 Thread Eli Zaretskii
> Date: Fri, 19 Apr 2013 11:54:05 +0200 > Cc: bo...@kolpackov.net, bug-make@gnu.org > From: Frank Heckenbach > > Eli Zaretskii wrote: > > > Initial investigation indicates that tmpfile should do the job just > > fine: the file is deleted only when the last descriptor for it is > > closed. That

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-21 Thread Paul Smith
On Fri, 2013-04-19 at 12:36 +0300, Eli Zaretskii wrote: > Also, where is the best place to put the emulated Posix functions? > Some new file in w32/compat/? I'd like to see it there. I'm thinking I want to move the new stuff out of job.c even for POSIX systems. The ifdefs are really getting to

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-21 Thread Paul Smith
On Fri, 2013-04-19 at 14:09 +0300, Eli Zaretskii wrote: > > Date: Fri, 19 Apr 2013 11:54:05 +0200 > > Cc: bo...@kolpackov.net, bug-make@gnu.org > > From: Frank Heckenbach > > > > > Is there a simple enough Makefile somewhere that could be used to test > > > this feature, once implemented? > > >

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-19 Thread Eli Zaretskii
> Date: Fri, 19 Apr 2013 11:54:05 +0200 > Cc: bo...@kolpackov.net, bug-make@gnu.org > From: Frank Heckenbach > > > Is there a simple enough Makefile somewhere that could be used to test > > this feature, once implemented? > > We have a test in the test suite (output-sync). Can you use that? I h

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-19 Thread Frank Heckenbach
Eli Zaretskii wrote: > Initial investigation indicates that tmpfile should do the job just > fine: the file is deleted only when the last descriptor for it is > closed. That includes any duplicated descriptors. Great. > As for fcntl, F_SETLKW, and F_GETFD, they will need to be emulated. > In pa

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-19 Thread Eli Zaretskii
> Date: Fri, 19 Apr 2013 00:06:52 +0300 > From: Eli Zaretskii > Cc: bo...@kolpackov.net, bug-make@gnu.org > > > > > Indeed, as you suggested earlier, it might be useful to use the main > > > > part of open_tmpfile() (i.e. without the fdopen()), though we'd have > > > > to manually remove the file

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-18 Thread Eli Zaretskii
> Date: Thu, 18 Apr 2013 22:05:33 +0200 > Cc: bug-make@gnu.org, david.s.bo...@gmail.com, psm...@gnu.org, > bo...@kolpackov.net > From: Frank Heckenbach > > Eli Zaretskii wrote: > > > > Date: Thu, 18 Apr 2013 20:39:44 +0200 > > > Cc: psm...@gnu.org, e...@gnu.org, bo...@kolpackov.net > > > F

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-18 Thread Frank Heckenbach
Paul Smith wrote: > On Thu, 2013-04-18 at 20:36 +0200, Frank Heckenbach wrote: > > And with my progress mechanism, that's exactly what I want. In my > > case it'd look like this: > > > > [Start] Compiling foo.c > > [Start] Compiling bar.c > > # time passes > > foo.c: some error > > # time passes

Re: [bug #33138] .PARLLELSYNC enhancement with patch

2013-04-18 Thread Frank Heckenbach
Eli Zaretskii wrote: > > Date: Thu, 18 Apr 2013 20:39:44 +0200 > > Cc: psm...@gnu.org, e...@gnu.org, bo...@kolpackov.net > > From: Frank Heckenbach > > > > Indeed, as you suggested earlier, it might be useful to use the main > > part of open_tmpfile() (i.e. without the fdopen()), though we'd hav

  1   2   >