Re: [dev] [sbase] sponge v2

2013-07-05 Thread Roberto E. Vargas Caballero
> Files created by “tmpfile” are automatically deleted after they are > closed. Also, how do you delete a file you don’t know the name of anyway? I didn't know it, thanks for your reply.

Re: [dev] [sbase] sponge v2

2013-07-05 Thread Jakob Kramer
On 07/05/2013 04:48 PM, Roberto E. Vargas Caballero wrote: >> Sponge's "killer feature" is that it doesn't open the output file until >> after the input is finished. Using it in a pipeline removes this, >> because it's something else instead of sponge writing to the file. > > > There is something

Re: [dev] [sbase] sponge v2

2013-07-05 Thread Roberto E. Vargas Caballero
> Sponge's "killer feature" is that it doesn't open the output file until > after the input is finished. Using it in a pipeline removes this, > because it's something else instead of sponge writing to the file. There is something I don't understand, why the temporal file is not deleted after writ

Re: [dev] [sbase] sponge v2

2013-07-05 Thread random832
On Tue, Jul 2, 2013, at 13:42, Calvin Morrison wrote: > doesn't sponge soak up into memory, not into a file? Sponge's "killer feature" is that it doesn't open the output file until after the input is finished. Using it in a pipeline removes this, because it's something else instead of sponge writi

Re: [dev] [sbase] sponge v2

2013-07-03 Thread hiro
just give it up, this thread sucks. On 7/3/13, Bjartur Thorlacius wrote: > On 07/02/2013 07:54 PM, Calvin Morrison wrote: >>> >If on any system other than linux, I would consider loading into ram, >>> >but because of memory overcommit, malloc never fails, the whole system >>> >crawls to a halt, a

Re: [dev] [sbase] sponge v2

2013-07-03 Thread Bjartur Thorlacius
On 07/02/2013 07:54 PM, Calvin Morrison wrote: >If on any system other than linux, I would consider loading into ram, >but because of memory overcommit, malloc never fails, the whole system >crawls to a halt, and the oom killer takes 20 minutes to put >everything back together. No thanks. Okay s

Re: [dev] [sbase] sponge v2

2013-07-03 Thread Calvin Morrison
On 2 July 2013 20:41, Galos, David wrote: >> and also depends on having a non-read only filesystem >> to write too. > You understand that in order to change the target file, the > filesystem needs to be writable, right? What if I had extremely limited filespace though, or only write permissions t

Re: [dev] [sbase] sponge v2

2013-07-03 Thread Calvin Morrison
How? On 3 July 2013 08:13, hiro <23h...@gmail.com> wrote: > won't this increase cache misses for everything else? > > On 7/2/13, Calvin Morrison wrote: >> On 2 July 2013 15:50, Galos, David wrote: I'm failing to see the problem with loading everything into memory. >>> >>> If the next step i

Re: [dev] [sbase] sponge v2

2013-07-03 Thread hiro
won't this increase cache misses for everything else? On 7/2/13, Calvin Morrison wrote: > On 2 July 2013 15:50, Galos, David wrote: >>> I'm failing to see the problem with loading everything into memory. >> >> If the next step is "write that to a temp file" there is a big >> problem. There's not

Re: [dev] [sbase] sponge v2

2013-07-02 Thread Roberto E. Vargas Caballero
> > Maybe a better solution could be using mmap. > Perhaps. I still wouldn't know how much to map, so I'd need > mremap, and from the manpage: >This call is Linux-specific, and should not be used >in programs intended to be portable. > If you write a portable patch making use of mma

Re: [dev] [sbase] sponge v2

2013-07-02 Thread Galos, David
> and also depends on having a non-read only filesystem > to write too. You understand that in order to change the target file, the filesystem needs to be writable, right? > > It's reasonably fast > I want faster. It's so fast that you never even realized that the moreutils version uses a tempfile

Re: [dev] [sbase] sponge v2

2013-07-02 Thread Roberto E. Vargas Caballero
> > If on any system other than linux, I would consider loading into ram, > > but because of memory overcommit, malloc never fails, the whole system > > crawls to a halt, and the oom killer takes 20 minutes to put > > everything back together. No thanks. > > Okay so you are discussing a problem wit

Re: [dev] [sbase] sponge v2

2013-07-02 Thread Calvin Morrison
On 2 July 2013 15:50, Galos, David wrote: >> I'm failing to see the problem with loading everything into memory. > > If the next step is "write that to a temp file" there is a big > problem. There's nothing wrong with the present tempfile approach. Yes there is. It wastes read/writes on my hard d

Re: [dev] [sbase] sponge v2

2013-07-02 Thread Galos, David
> I'm failing to see the problem with loading everything into memory. If the next step is "write that to a temp file" there is a big problem. There's nothing wrong with the present tempfile approach. It's reasonably fast, it's completely portable (so no crazy getrlimit or /proc reading). Strace sh

Re: [dev] [sbase] sponge v2

2013-07-02 Thread Calvin Morrison
I'm failing to see the problem with loading everything into memory. On 2 July 2013 15:39, Galos, David wrote: >> It also sucks. avoiding a temporary file is the whole point to sponge. >> otherwise: >> >> cat file | grep file > file_temp; mv file_temp file; > > Actually, it would be > tmp =

Re: [dev] [sbase] sponge v2

2013-07-02 Thread Galos, David
> It also sucks. avoiding a temporary file is the whole point to sponge. > otherwise: > > cat file | grep file > file_temp; mv file_temp file; Actually, it would be tmp = $(mktemp) grep 'foo' file > $tmp cat $tmp > file Not only that, but tmpfile(3) is safer with respect to race co

Re: [dev] [sbase] sponge v2

2013-07-02 Thread Calvin Morrison
It also sucks. avoiding a temporary file is the whole point to sponge. otherwise: cat file | grep file > file_temp; mv file_temp file; On 2 July 2013 14:19, Galos, David wrote: >> doesn't sponge soak up into memory, not into a file? > > Soaking up into a file allows sponge to work on large files

Re: [dev] [sbase] sponge v2

2013-07-02 Thread Galos, David
> doesn't sponge soak up into memory, not into a file? Soaking up into a file allows sponge to work on large files. It also greatly simplifies the actual sponge code.

Re: [dev] [sbase] sponge v2

2013-07-02 Thread Calvin Morrison
doesn't sponge soak up into memory, not into a file? Calvin On 2 July 2013 13:29, Galos, David wrote: >> In my opinion it is okay to have sponge called >> without arguments to write to stdout. > > I disagree. The only use case I could think of was > something like this: > >sed 's/foo/bar/' f

Re: [dev] [sbase] sponge v2

2013-07-02 Thread Galos, David
> In my opinion it is okay to have sponge called > without arguments to write to stdout. I disagree. The only use case I could think of was something like this: sed 's/foo/bar/' file | sponge | tee file | frobnicate However, that would fail to work as expected, so I have removed the feature.

[dev] [sbase] sponge v2

2013-06-26 Thread Jakob Kramer
Hello! I have rewritten sponge(1) so stdout is not such of a special case anymore. Now it also writes to symbolic links' destinations instead and supports other kinds of files. Also output to stdout works as it should now (soaks it up first). In my opinion it is okay to have sponge called witho