I am in fact replacing an io.Pipe implementation, because I need to buffer some data. io.Pipe doesn't buffer, it just matches up read and writes with each other.
What I'm effectively doing is producing chunks of data at a certain rate from one server, and forwarding them to another. Due to the latency of issuing the read from the server, I need to do some pre-fetch into a buffer, but I don't want to prefetch to much. With the pipe implementation, my reader had many stalls waiting for the server to produce the next chunk. The server can produce data a lot quicker than the reader can read, however, setup time is high. It's a complicated mess :) On Thu, Nov 21, 2019 at 8:37 PM Robert Engels <reng...@ix.netcom.com> wrote: > The OP specifically requested io.Reader/Writer interfaces. > > A pipe is what he wants. Not a ring buffer. (A pipe essentially has a ring > buffer in the implementation though). > > > On Nov 21, 2019, at 10:20 PM, Dan Kortschak <d...@kortschak.io> wrote: > > > > There is this: https://godoc.org/bitbucket.org/ausocean/utils/ring > > > > It has been used in production fairly extensively. > > > >> On Thu, 2019-11-21 at 19:47 -0800, Marcin Romaszewicz wrote: > >> Hi All, > >> > >> Before I reinvent the wheel, and because this wheel is particularly > >> tricky > >> to get right, I was wondering if anyone was aware of a a library > >> providing > >> something like this > >> > >> - conforms to io.Reader > >> - conforms to io.Writer > >> - Contains a buffer of fixed size, say, 64MB. If you try to write > >> when the > >> buffer is too full, write blocks. When you try to read from an empty > >> one, > >> read blocks. > >> > >> This describes the behavior of make(chan byte, 64 * MB), however, it > >> doesn't seem to be efficient to do this with a channel. Say I'm > >> transferring a few hundred GB via this mechanism. A chan of byte > >> would need > >> a few hundred billion byte writes, and a few hundred billion reads. > >> Doesn't > >> sound like it could be efficient at all. You can't implement this > >> with a > >> channel of []byte, because you'd violate your buffering limit if you > >> pass > >> too large of a byte array, so you'd have to chop things up into > >> blocks, but > >> maybe that's simpler than a full fledged blocking ring buffer. > >> > >> Anyhow, I've implemented such things in various OS level plumbing, so > >> I > >> know I can do it in Go much more easily, just hoping to avoid it :) > >> > >> Thanks for any advice, > >> -- Marcin > >> > > > > -- > > You received this message because you are subscribed to the Google > Groups "golang-nuts" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to golang-nuts+unsubscr...@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/152a954eaa3eea02514e71bb142904480241ad6c.camel%40kortschak.io > . > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CA%2Bv29Ls8P76hjKFRXMQ4-GrhSFY82zGVQ3xnKO6%3DGLZLSLuWxA%40mail.gmail.com.
