Re: Unexpectedly low read/write performance of open-pipe

Just wow. I was aware for a while that Guile's I/O "story" wasn't stellar. So I was watching this thread, as my awe grew, mail by mail. Thanks Rob, Mark and all the others. This was really impressive. You rock. Cheers -- tomás signature.asc Description: Digital signature

Re: Unexpectedly low read/write performance of open-pipe

Rob Browning writes: >> For something similar to the original python code and roughly similar to >> open-pipe OPEN_BOTH (see code below) I see: >> mb/s: 1713.26754296 > Here for "guile -s io-perf.scm > /dev/null" I see: > > 2.2.4: 1.39 mb/s >100% CPU in top > your first p

Re: Unexpectedly low read/write performance of open-pipe

Mark H Weaver writes: > I've attached four patches. Only the first two should be needed for > what I have described above. > > The last two patches will only be of interest if you need suspendable > I/O. They add suspendable I/O support for custom binary ports, and in > particular for these OPE

Re: Unexpectedly low read/write performance of open-pipe

Hi Rob, Rob Browning writes: > For a first quick test of your patch using the original program I was > working on, I see about ~1.4MiB/s without the patch, and about 150MiB/s > with it, measured by pv. > > (If the patch holds up, it'd be nice to have in 2.2, but I suppose that > might not be app

Re: Unexpectedly low read/write performance of open-pipe

> Earlier, I wrote: >> I'm not sure off-hand what would be required to re-implement custom >> ports in suspendable Scheme code. > > I finally dug into this code, and was delighted to find that Andy Wingo > has already laid the groundwork to avoid going through C code in our > custom port handlers,

Re: Unexpectedly low read/write performance of open-pipe

Hi again, Earlier, I wrote: > I'm not sure off-hand what would be required to re-implement custom > ports in suspendable Scheme code. I finally dug into this code, and was delighted to find that Andy Wingo has already laid the groundwork to avoid going through C code in our custom port handlers,

Re: Unexpectedly low read/write performance of open-pipe

On Tue, 09 Apr 2019 14:24:09 -0400 Mark H Weaver wrote: > Hi Chris, > Chris Vine writes: > > On Tue, 09 Apr 2019 04:35:38 -0400 > > Mark H Weaver wrote: > >> > >> I think it's probably fine for 2.2, although a more careful check should > >> be made for differences in behavior between the old an

Re: Unexpectedly low read/write performance of open-pipe

Chris Vine writes: > On Tue, 09 Apr 2019 04:35:38 -0400 > Mark H Weaver wrote: >> >> I think it's probably fine for 2.2, although a more careful check should >> be made for differences in behavior between the old and new >> implementations, and tests should be added. I'll try to get to it soon.

Re: Unexpectedly low read/write performance of open-pipe

Hi Chris, Chris Vine writes: > On Tue, 09 Apr 2019 04:35:38 -0400 > Mark H Weaver wrote: >> >> I think it's probably fine for 2.2, although a more careful check should >> be made for differences in behavior between the old and new >> implementations, and tests should be added. I'll try to get

Re: Unexpectedly low read/write performance of open-pipe

On Tue, 09 Apr 2019 04:35:38 -0400 Mark H Weaver wrote: > Hi Rob, > > Rob Browning writes: > > > Mark H Weaver writes: > > > >> See below for a draft reimplementation of the OPEN_BOTH mode of > >> open-pipe* based on R6RS custom binary input/output. On my machine it > >> increases the speed o

Re: Unexpectedly low read/write performance of open-pipe

Hi Rob, Rob Browning writes: > Mark H Weaver writes: > >> See below for a draft reimplementation of the OPEN_BOTH mode of >> open-pipe* based on R6RS custom binary input/output. On my machine it >> increases the speed of your test by a factor of ~1k. > > Hah, I was about to report that I'd tes

Re: Unexpectedly low read/write performance of open-pipe

Mark H Weaver writes: > See below for a draft reimplementation of the OPEN_BOTH mode of > open-pipe* based on R6RS custom binary input/output. On my machine it > increases the speed of your test by a factor of ~1k. Hah, I was about to report that I'd tested something along similar lines (though

Re: Unexpectedly low read/write performance of open-pipe

Hi Rob, Rob Browning writes: > Rob Browning writes: > >> I haven't tried to track it down yet, but if the only underlying way to >> get a fixed block of data out of an OPEN_BOTH port is read-char, then >> that might explain much of the difference. > > And this, in popen.scm was why I started wo

Re: Unexpectedly low read/write performance of open-pipe

Rob Browning writes: > I haven't tried to track it down yet, but if the only underlying way to > get a fixed block of data out of an OPEN_BOTH port is read-char, then > that might explain much of the difference. And this, in popen.scm was why I started wondering about that: (call-with-values

Re: Unexpectedly low read/write performance of open-pipe

Rob Browning writes: > For what it's worth, in an earlier round of testing I also hacked up > open-pipe to let me access the underlying ports and set their buffers to > 65k. That doubled the transfer rate, but of course, it's still fairly > slow. I also ran statprof on the OPEN_BOTH case and sa

Re: Unexpectedly low read/write performance of open-pipe

Rob Browning writes: > While evaluating guile as a possibility to replace some python code, > assuming I'm not just doing something wrong, I noticed that open-pipe > appears to transfer data *much* more slowly than python when OPEN_BOTH is > specified as opposed to OPEN_READ Oh, and I should hav

Unexpectedly low read/write performance of open-pipe

While evaluating guile as a possibility to replace some python code, assuming I'm not just doing something wrong, I noticed that open-pipe appears to transfer data *much* more slowly than python when OPEN_BOTH is specified as opposed to OPEN_READ: discarding dev-zero as file: 12500.00 mb/s d