Re: Evading the command line length limit (on Linux)

[Paul Smith]

On Wed, 2009-07-29 at 22:51 +0200, Ralf Wildenhues wrote:
> 1) One possibility would be to drop -c and pipe the command line to sh
> on standard input.  Just like parallel make, this would prevent using
> the standard input of the make process within the child command process.
> 
> 2) Another possibility would be a hack like the following: split the
> long line into N substrings, each below the argument limit, and invoke
> 
>   sh -c 'eval "${1}${2}...${10}...${N}"' sh quoted-substring1 ...
> 
> This changes the following user-observable semantics:
> 
> - the positional parameters have different settings; this can be
>   fixed with a 'set x; shift;' at the start of the eval'ed string,
> - LINENO is not reliable in eval with some shells.
> - ${10} and up are not portable to pre-Posix shells like Solaris sh and
>   the heirloom shell under Linux
>   (See autoconf.info 'Limitations of Builtins' for more details on these
>   issues.)
> 
> I don't think these cause big problems in practice, more so if we go
> this way only with command line lengths that would otherwise have failed
> anyway, and only on Linux systems.

Hrm.  I have to say I don't think I like option #2.  It has "special
hack" written all over it.

Not only will it only work on Linux but the ${10} thing ties it to bash,
which means even for Linux-based for systems like Ubuntu where /bin/sh !
= bash it's not so great.

Remember that we actually have this problem in spades on other systems,
like Windows, where the command line length if far SHORTER than Linux.
I'd prefer to find some solution which solves the problem for everyone,
rather than adding some pretty obscure special-case situations.

I'd prefer to use something closer to #1, which is more portable (I
think).  However, it does break POSIX and could break some makefiles
that rely on stdin being available.  Maybe the thing to do is require
the user to specify which rules would be treated in this special way.
That makes it less automatic, which is too bad, but the length of the
command line is pretty static so the person writing the makefile should
know when it's needed.

Alternatively if we can find a way to do it automatically in a portable
way that would be OK too, since the alternative would be to fail anyway.
For Windows and other systems we can probably just choose a hardcoded
number; for UNIX systems it's more complicated.

There have been various ideas floating around about allowing make to
create/use @-files, which a lot of utilities are supporting now.  Maybe
that is a solution here as well?

-- 
---
 Paul D. Smith   Find some GNU make tips at:
 http://www.gnu.org  http://make.mad-scientist.us
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist


___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

[bug #27143] Crash in a pair of mutually recursive macros

[Timothy N Murphy]

URL:
  

 Summary: Crash in a pair of mutually recursive macros
 Project: make
Submitted by: tnmurphy
Submitted on: Thu 30 Jul 2009 16:42:11 GMT
Severity: 3 - Normal
  Item Group: Bug
  Status: None
 Privacy: Public
 Assigned to: None
 Open/Closed: Open
 Discussion Lock: Any
   Component Version: 3.81
Operating System: Any
   Fixed Release: None

___

Details:

This crash happens on win32 with mingw make 3.81 for us and on Redhat 4 (32)
with custom built gnu make 3.81 (32-bit), also CentOS 5.3 (64) using gnu make
3.81 (32).

It appears to succeed on Fedora 11 with make 3.81 (64) but fail with a custom
built 32-bit make (linked to compat-glibc).

Sorry for the confusion there.  

My example is a bit big at the moment.

I used valgrind to see what was happening:



==24453== 
==24453== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 14 from 1)
==24453== malloc/free: in use at exit: 44,167 bytes in 1,216 blocks.
==24453== malloc/free: 2,785 allocs, 1,569 frees, 93,301 bytes allocated.
==24453== For counts of detected errors, rerun with: -v
==24453== searching for pointers to 1,216 not-freed blocks.
==24453== checked 147,452 bytes.
==24453== 
==24453== LEAK SUMMARY:
==24453==definitely lost: 0 bytes in 0 blocks.
==24453==  possibly lost: 0 bytes in 0 blocks.
==24453==still reachable: 44,167 bytes in 1,216 blocks.
==24453== suppressed: 0 bytes in 0 blocks.
==24453== Rerun with --leak-check=full to see details of leaked memory.
[tmur...@head t]cd ~/tmp/
[tmur...@head tmp]valgrind make -f t.mk
==24462== Memcheck, a memory error detector.
==24462== Copyright (C) 2002-2008, and GNU GPL'd, by Julian Seward et al.
==24462== Using LibVEX rev 1884, a library for dynamic binary translation.
==24462== Copyright (C) 2004-2008, and GNU GPL'd, by OpenWorks LLP.
==24462== Using valgrind-3.4.1, a dynamic binary instrumentation framework.
==24462== Copyright (C) 2000-2008, and GNU GPL'd, by Julian Seward et al.
==24462== For more details, rerun with: -v
==24462== 
==24462== Warning: client switching stacks?  SP change: 0xbefbba80 -->
0xbd5fbc50
==24462==  to suppress, use: --max-stackframe=27000368 or greater
==24462== 
==24462== Process terminating with default action of signal 11 (SIGSEGV)
==24462==  Access not within mapped region at address 0xBD5FBC48
==24462==at 0x804D4FC: expand_argument (expand.c:439)
==24462==  If you believe this happened as a result of a stack overflow in
your
==24462==  program's main thread (unlikely but possible), you can try to
increase
==24462==  the size of the main thread stack using the --main-stacksize=
flag.
==24462==  The main thread stack size used in this run was -1.
==24462== 
==24462== Process terminating with default action of signal 11 (SIGSEGV)
==24462==  Access not within mapped region at address 0xBD5FBC44
==24462==at 0x40005A3: _vgnU_freeres (vg_preloaded.c:56)
==24462==  If you believe this happened as a result of a stack overflow in
your
==24462==  program's main thread (unlikely but possible), you can try to
increase
==24462==  the size of the main thread stack using the --main-stacksize=
flag.
==24462==  The main thread stack size used in this run was -1.
==24462== 
==24462== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 14 from 1)
==24462== malloc/free: in use at exit: 137,065,676 bytes in 699 blocks.
==24462== malloc/free: 2,031,760 allocs, 2,031,061 frees, 2,078,692,395 bytes
allocated.
==24462== For counts of detected errors, rerun with: -v
==24462== searching for pointers to 699 not-freed blocks.
==24462== checked 137,170,860 bytes.
==24462== 
==24462== LEAK SUMMARY:
==24462==definitely lost: 0 bytes in 0 blocks.
==24462==  possibly lost: 0 bytes in 0 blocks.
==24462==still reachable: 137,065,676 bytes in 699 blocks.
==24462== suppressed: 0 bytes in 0 blocks.
==24462== Rerun with --leak-check=full to see details of leaked memory.


The makefile contains:

# Crash in a recursing macro 


# AllSuffixSubst
## Converting a list of source files to a list of object files without
altering
## their relative order. Also deals with multiple file types mapping to the
same
## object type e.g. .CPP and .cpp and .c++ all map to .o

# $1 - the list of suffixes to replace
# $2 - the suffix to replace them with
# $3 - the list of strings to perform the replacement on
define allsuffixsubst_internal
$(if $1,$$(patsubst %$(firstword $1),%$2,$(call
allsuffixsubst_internal,$(wordlist 2,$(words $1),$1),$2,$3)),$3)
endef
define allsuffixsubst
$(eval allsuffixsubst_RESULT:=$(call
allsuffixsubst_internal,$1,$2,$3))$(allsuffixsubst_RESULT)
endef

x10:=1 2 3 4 5 6 7 8 9 0
x100:=$(foreach i,$(x10),$(x10))
x1:=$(foreach i

[bug #23960] Crash when MAKEFILE_LIST becomes too long.

[Timothy N Murphy]

Additional Item Attachment, bug #23960 (project make):

File name: variable_assign_crash.mk   Size:6 KB


___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

Re: Evading the command line length limit (on Linux)

[Ralf Wildenhues]

Hi Paul,

* Paul Smith wrote on Thu, Jul 30, 2009 at 04:19:53PM CEST:
> On Wed, 2009-07-29 at 22:51 +0200, Ralf Wildenhues wrote:
> > 
> > 2) Another possibility would be a hack like the following: split the
> > long line into N substrings, each below the argument limit, and invoke
> > 
> >   sh -c 'eval "${1}${2}...${10}...${N}"' sh quoted-substring1 ...
[...]
> > - ${10} and up are not portable to pre-Posix shells like Solaris sh and
> >   the heirloom shell under Linux

> > I don't think these cause big problems in practice, more so if we go
> > this way only with command line lengths that would otherwise have failed
> > anyway, and only on Linux systems.
> 
> Hrm.  I have to say I don't think I like option #2.  It has "special
> hack" written all over it.

Yep.

> Not only will it only work on Linux but the ${10} thing ties it to bash,
> which means even for Linux-based for systems like Ubuntu where /bin/sh !
> = bash it's not so great.

That is not true.  All remotely Posixy shells that were written after
1990 accept ${10}.  Heirloom sh is special in that, after the SYSV
sources were released under a liberal license, somebody dug out the code
that Steve Bourne wrote and made it compilable under GNU/Linux.  That's
the only Linux shell that doesn't grok ${10}.  All of dash, pdksh,
ksh93, zsh, bash, posh, have no problem with ${10}.  And nobody right in
their mind would use heirloom sh as /bin/sh on GNU/Linux.

Of course, other (proprietary) unices is a different matter altogether,
but that's not what I'm attacking here at all.

> Remember that we actually have this problem in spades on other systems,
> like Windows, where the command line length if far SHORTER than Linux.

I know.  I think what made me try this patch was the little kid in me
that wants to have one developer system without arbitrary limits, but
not as esoteric as plan 9 or as experimental as the Hurd.

Let's face it, there are packages out there that only ever care about
GNU/Linux any more.  GNU Automake is all about portable makefiles and
working around all kinds of limitations (and we'll probably add
some measure for this issue, too), but GNU make is, at least somewhat,
also about unleashing all the power of your system to you.

> I'd prefer to find some solution which solves the problem for everyone,
> rather than adding some pretty obscure special-case situations.

Except of course that the code already effectively has at least two
special-case situations for EMX, w32.

> I'd prefer to use something closer to #1, which is more portable (I
> think).  However, it does break POSIX and could break some makefiles
> that rely on stdin being available.  Maybe the thing to do is require
> the user to specify which rules would be treated in this special way.
> That makes it less automatic, which is too bad, but the length of the
> command line is pretty static so the person writing the makefile should
> know when it's needed.

Well, I guess my point is that, if you require special treatment from
your users anyway, then they can easily use a technique to avoid hitting
this limit.  GNU make provides the means to do so, see for example
.
However, this of course makes your complete makefile unportable to other
makes (as opposed to the original bug report, where with my patch, only
'make dist' would not work with other makes).

> Alternatively if we can find a way to do it automatically in a portable
> way that would be OK too, since the alternative would be to fail anyway.

Yes.  I too would prefer such a solution.

> For Windows and other systems we can probably just choose a hardcoded
> number; for UNIX systems it's more complicated.

For Windows you can likely use @-file support from command.com(?).

> There have been various ideas floating around about allowing make to
> create/use @-files, which a lot of utilities are supporting now.  Maybe
> that is a solution here as well?

Well if the command you're invoking needs this support, then I'd say
this is even less of an automatic solution than anything else.  I guess
we should add @-file support to bash, but even then your argument above
(regarding other shells) applies.

Oh well.  Even if I sound a bit defensive, I won't mind you rejecting
this patch, seeing that you have good reasons, but I do think that it
has some merit.  I'm open to exploring other solutions.

Cheers, and thanks,
Ralf


___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

Re: Evading the command line length limit (on Linux)

[Eli Zaretskii]

> From: Paul Smith 
> Date: Thu, 30 Jul 2009 10:19:53 -0400
> Cc: Xan Lopez , bug-make@gnu.org
> 
> Remember that we actually have this problem in spades on other systems,
> like Windows, where the command line length if far SHORTER than Linux.

Perhaps you think about the 127-character limitation in DOS.

The way Make invokes programs on Windows allow command lines up to
32KB.  (If the command being invoked is the stock Windows shell
cmd.exe, it has a smaller built-in limitation: 8KB.)  Even the DOS
port of Make supports up to 16KB.  That might still be ``far SHORTER
than Linux'' (I assume you mean Bash on Linux), but maybe it's enough
to stop talking about problems with long command lines.


___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make