To give a little bit more background on what I was doing - I have an API endpoint which is used to download large amounts of data that is backed by S3 in AWS. Using an io.Pipe to simply proxy from S3 tops out at around 30MB/sec due to single threaded S3 performance and setup time. I wrote an adapter around it which downloads increasingly more S3 chunks in parallel, until my producer rate starts to exceed the consumer rate - now I'm getting upwards of a 1GB/sec on really large files. The only tricky bit is sequencing chunks for feeding into the io.Pipe, but there's few of them, so an array with a mutex around it works just fine. I didn't have to write the ring buffer in the end.
Working in Go is a joy at times like this - having a multi threaded chunked fetcher hiding behind a single threaded HTTP GET all the while not buffering too much data is a fiendishly hard thing to do in other languages due to the nature of their threading libraries and the sparse library support for asynchronous primitives. I've done stuff like this in the past in the IRIX kernel (don't laugh), or in C/C++ and pthreads, and each of those would have been days of work to get right instead of half a day of writing simple code to glue together available library functions. -- Marcin On Thu, Nov 21, 2019 at 9:48 PM Robert Engels <reng...@ix.netcom.com> wrote: > I would just copy and paste the stdlib pipe.go and change the ctor to take > a size for the channel. I’m pretty sure that is all that is needed to fix > the problem. > > On Nov 21, 2019, at 11:18 PM, Marcin Romaszewicz <marc...@gmail.com> > wrote: > > > I am in fact replacing an io.Pipe implementation, because I need to buffer > some data. io.Pipe doesn't buffer, it just matches up read and writes with > each other. > > What I'm effectively doing is producing chunks of data at a certain rate > from one server, and forwarding them to another. Due to the latency of > issuing the read from the server, I need to do some pre-fetch into a > buffer, but I don't want to prefetch to much. > > With the pipe implementation, my reader had many stalls waiting for the > server to produce the next chunk. The server can produce data a lot > quicker than the reader can read, however, setup time is high. It's a > complicated mess :) > > On Thu, Nov 21, 2019 at 8:37 PM Robert Engels <reng...@ix.netcom.com> > wrote: > >> The OP specifically requested io.Reader/Writer interfaces. >> >> A pipe is what he wants. Not a ring buffer. (A pipe essentially has a >> ring buffer in the implementation though). >> >> > On Nov 21, 2019, at 10:20 PM, Dan Kortschak <d...@kortschak.io> wrote: >> > >> > There is this: https://godoc.org/bitbucket.org/ausocean/utils/ring >> > >> > It has been used in production fairly extensively. >> > >> >> On Thu, 2019-11-21 at 19:47 -0800, Marcin Romaszewicz wrote: >> >> Hi All, >> >> >> >> Before I reinvent the wheel, and because this wheel is particularly >> >> tricky >> >> to get right, I was wondering if anyone was aware of a a library >> >> providing >> >> something like this >> >> >> >> - conforms to io.Reader >> >> - conforms to io.Writer >> >> - Contains a buffer of fixed size, say, 64MB. If you try to write >> >> when the >> >> buffer is too full, write blocks. When you try to read from an empty >> >> one, >> >> read blocks. >> >> >> >> This describes the behavior of make(chan byte, 64 * MB), however, it >> >> doesn't seem to be efficient to do this with a channel. Say I'm >> >> transferring a few hundred GB via this mechanism. A chan of byte >> >> would need >> >> a few hundred billion byte writes, and a few hundred billion reads. >> >> Doesn't >> >> sound like it could be efficient at all. You can't implement this >> >> with a >> >> channel of []byte, because you'd violate your buffering limit if you >> >> pass >> >> too large of a byte array, so you'd have to chop things up into >> >> blocks, but >> >> maybe that's simpler than a full fledged blocking ring buffer. >> >> >> >> Anyhow, I've implemented such things in various OS level plumbing, so >> >> I >> >> know I can do it in Go much more easily, just hoping to avoid it :) >> >> >> >> Thanks for any advice, >> >> -- Marcin >> >> >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "golang-nuts" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to golang-nuts+unsubscr...@googlegroups.com. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/152a954eaa3eea02514e71bb142904480241ad6c.camel%40kortschak.io >> . >> >> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CA%2Bv29LtMLyffk5gqGgd%2BTWi56mZCgbY4ibe2Wt6gXQtCEF6okw%40mail.gmail.com.
