I would just copy and paste the stdlib pipe.go and change the ctor to take a size for the channel. I’m pretty sure that is all that is needed to fix the problem.
> On Nov 21, 2019, at 11:18 PM, Marcin Romaszewicz <marc...@gmail.com> wrote: > > > I am in fact replacing an io.Pipe implementation, because I need to buffer > some data. io.Pipe doesn't buffer, it just matches up read and writes with > each other. > > What I'm effectively doing is producing chunks of data at a certain rate from > one server, and forwarding them to another. Due to the latency of issuing the > read from the server, I need to do some pre-fetch into a buffer, but I don't > want to prefetch to much. > > With the pipe implementation, my reader had many stalls waiting for the > server to produce the next chunk. The server can produce data a lot quicker > than the reader can read, however, setup time is high. It's a complicated > mess :) > >> On Thu, Nov 21, 2019 at 8:37 PM Robert Engels <reng...@ix.netcom.com> wrote: >> The OP specifically requested io.Reader/Writer interfaces. >> >> A pipe is what he wants. Not a ring buffer. (A pipe essentially has a ring >> buffer in the implementation though). >> >> > On Nov 21, 2019, at 10:20 PM, Dan Kortschak <d...@kortschak.io> wrote: >> > >> > There is this: https://godoc.org/bitbucket.org/ausocean/utils/ring >> > >> > It has been used in production fairly extensively. >> > >> >> On Thu, 2019-11-21 at 19:47 -0800, Marcin Romaszewicz wrote: >> >> Hi All, >> >> >> >> Before I reinvent the wheel, and because this wheel is particularly >> >> tricky >> >> to get right, I was wondering if anyone was aware of a a library >> >> providing >> >> something like this >> >> >> >> - conforms to io.Reader >> >> - conforms to io.Writer >> >> - Contains a buffer of fixed size, say, 64MB. If you try to write >> >> when the >> >> buffer is too full, write blocks. When you try to read from an empty >> >> one, >> >> read blocks. >> >> >> >> This describes the behavior of make(chan byte, 64 * MB), however, it >> >> doesn't seem to be efficient to do this with a channel. Say I'm >> >> transferring a few hundred GB via this mechanism. A chan of byte >> >> would need >> >> a few hundred billion byte writes, and a few hundred billion reads. >> >> Doesn't >> >> sound like it could be efficient at all. You can't implement this >> >> with a >> >> channel of []byte, because you'd violate your buffering limit if you >> >> pass >> >> too large of a byte array, so you'd have to chop things up into >> >> blocks, but >> >> maybe that's simpler than a full fledged blocking ring buffer. >> >> >> >> Anyhow, I've implemented such things in various OS level plumbing, so >> >> I >> >> know I can do it in Go much more easily, just hoping to avoid it :) >> >> >> >> Thanks for any advice, >> >> -- Marcin >> >> >> > >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "golang-nuts" group. >> > To unsubscribe from this group and stop receiving emails from it, send an >> > email to golang-nuts+unsubscr...@googlegroups.com. >> > To view this discussion on the web visit >> > https://groups.google.com/d/msgid/golang-nuts/152a954eaa3eea02514e71bb142904480241ad6c.camel%40kortschak.io. >> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/C1587EA0-9752-4AE5-A232-BB5DCCC2343E%40ix.netcom.com.
