Re: sort(1) memory usage

2008-02-04 Thread Dag-Erling Smørgrav
Joerg Sonnenberger <[EMAIL PROTECTED]> writes: > On Mon, Feb 04, 2008 at 04:04:35PM +0100, Dag-Erling Sm?rgrav wrote: > > Last I checked, it also (rather surprisingly) lacked -u (unique), > > which is required by POSIX. > That must have been before the import into src/usr.bin/sort in 2000. Oops -

Re: sort(1) memory usage

2008-02-04 Thread Joerg Sonnenberger
On Mon, Feb 04, 2008 at 04:04:35PM +0100, Dag-Erling Sm?rgrav wrote: > Last I checked, it also (rather surprisingly) lacked -u (unique), > which is required by POSIX. That must have been before the import into src/usr.bin/sort in 2000. Joerg ___ freebsd

Re: sort(1) memory usage

2008-02-04 Thread David Schultz
On Sun, Feb 03, 2008, Dag-Erling Smørgrav wrote: > Dag-Erling Smørgrav <[EMAIL PROTECTED]> writes: > > Erik Trulsson <[EMAIL PROTECTED]> writes: > > > Yep, it seems that GNU sort allocates a quite large buffer by default when > > > the size of the input is unknown (such as when it reads input from

Re: sort(1) memory usage

2008-02-04 Thread Dag-Erling Smørgrav
David Schultz <[EMAIL PROTECTED]> writes: > We had been using a BSD-licensed sort(1), but ache@ changed it back to > GNU sort several years ago. Anyone know why? If I had to guess I'd say > i18n [...] That is my recollection as well. We would do well to take a look at the code (and CVS logs) from

Re: sort(1) memory usage

2008-02-04 Thread Dag-Erling Smørgrav
Jeremy Chadwick <[EMAIL PROTECTED]> writes: > As you said: the code shows that when no files are specified (e.g. read > off a pipe), sort will make some assumptions regarding the initial > buffer size to read data into. The buffer size allocated in that case > is fairly large, rather than basing

Re: sort(1) memory usage

2008-02-04 Thread Jeremy Chadwick
On Sun, Feb 03, 2008 at 04:31:34PM +0100, Dag-Erling Smørgrav wrote: > Dag-Erling Smørgrav <[EMAIL PROTECTED]> writes: > > Erik Trulsson <[EMAIL PROTECTED]> writes: > > > Yep, it seems that GNU sort allocates a quite large buffer by default when > > > the size of the input is unknown (such as when

Re: sort(1) memory usage

2008-02-03 Thread Ed Schouten
* Dag-Erling Smørgrav <[EMAIL PROTECTED]> wrote: > Count this as a vote for ditching GNU sort in favor of a BSD-licensed > implementation (from {Net,Open}BSD for instance). I just looked at the OpenBSD implementation and I can see it already misses one option that some people will miss, namely num

Re: sort(1) memory usage

2008-02-03 Thread Erik Trulsson
On Sun, Feb 03, 2008 at 04:31:34PM +0100, Dag-Erling Smørgrav wrote: > Dag-Erling Smørgrav <[EMAIL PROTECTED]> writes: > > Erik Trulsson <[EMAIL PROTECTED]> writes: > > > Yep, it seems that GNU sort allocates a quite large buffer by default when > > > the size of the input is unknown (such as when

Re: sort(1) memory usage

2008-02-03 Thread Wayne Sierke
On Sun, 2008-02-03 at 14:13 +0100, Ed Schouten wrote: > * Dag-Erling Smørgrav <[EMAIL PROTECTED]> wrote: > > I've been trying to figure out why some periodic scripts consume so much > > memory. I've narrowed it down to sort(1). > > > > At first, I thought the scripts were using it inefficiently,

Re: sort(1) memory usage

2008-02-03 Thread Dag-Erling Smørgrav
Dag-Erling Smørgrav <[EMAIL PROTECTED]> writes: > Erik Trulsson <[EMAIL PROTECTED]> writes: > > Yep, it seems that GNU sort allocates a quite large buffer by default when > > the size of the input is unknown (such as when it reads input from stdin.) > > A quick check in the source code indicates th

Re: sort(1) memory usage

2008-02-03 Thread Erik Trulsson
On Sun, Feb 03, 2008 at 02:13:22PM +0100, Ed Schouten wrote: > * Dag-Erling Smørgrav <[EMAIL PROTECTED]> wrote: > > I've been trying to figure out why some periodic scripts consume so much > > memory. I've narrowed it down to sort(1). > > > > At first, I thought the scripts were using it ineffici

Re: sort(1) memory usage

2008-02-03 Thread Dag-Erling Smørgrav
Erik Trulsson <[EMAIL PROTECTED]> writes: > Yep, it seems that GNU sort allocates a quite large buffer by default when > the size of the input is unknown (such as when it reads input from stdin.) > A quick check in the source code indicates that it tries to size this buffer > according to how much

Re: sort(1) memory usage

2008-02-03 Thread Ed Schouten
* Dag-Erling Smørgrav <[EMAIL PROTECTED]> wrote: > I've been trying to figure out why some periodic scripts consume so much > memory. I've narrowed it down to sort(1). > > At first, I thought the scripts were using it inefficiently, feeding it > more data than was really needed. Then I discovere