You could always use os.Pipe (which unlike io.Pipe is a standard posix pipe), and wrap one end or the other with bufio.Reader/Writer. Or if all you want to do is buffer the response from http.Get you can just wrap the http response reader with bufio.Reader. Or am I missing something here about this use case?
On Saturday, November 23, 2019 at 8:51:51 AM UTC-8, Marcin Romaszewicz wrote: > > To give a little bit more background on what I was doing - I have an API > endpoint which is used to download large amounts of data that is backed by > S3 in AWS. Using an io.Pipe to simply proxy from S3 tops out at around > 30MB/sec due to single threaded S3 performance and setup time. I wrote an > adapter around it which downloads increasingly more S3 chunks in parallel, > until my producer rate starts to exceed the consumer rate - now I'm getting > upwards of a 1GB/sec on really large files. The only tricky bit is > sequencing chunks for feeding into the io.Pipe, but there's few of them, so > an array with a mutex around it works just fine. I didn't have to write the > ring buffer in the end. > > Working in Go is a joy at times like this - having a multi threaded > chunked fetcher hiding behind a single threaded HTTP GET all the while not > buffering too much data is a fiendishly hard thing to do in other languages > due to the nature of their threading libraries and the sparse library > support for asynchronous primitives. I've done stuff like this in the past > in the IRIX kernel (don't laugh), or in C/C++ and pthreads, and each of > those would have been days of work to get right instead of half a day of > writing simple code to glue together available library functions. > > -- Marcin > > On Thu, Nov 21, 2019 at 9:48 PM Robert Engels <ren...@ix.netcom.com > <javascript:>> wrote: > >> I would just copy and paste the stdlib pipe.go and change the ctor to >> take a size for the channel. I’m pretty sure that is all that is needed to >> fix the problem. >> >> On Nov 21, 2019, at 11:18 PM, Marcin Romaszewicz <mar...@gmail.com >> <javascript:>> wrote: >> >> >> I am in fact replacing an io.Pipe implementation, because I need to >> buffer some data. io.Pipe doesn't buffer, it just matches up read and >> writes with each other. >> >> What I'm effectively doing is producing chunks of data at a certain rate >> from one server, and forwarding them to another. Due to the latency of >> issuing the read from the server, I need to do some pre-fetch into a >> buffer, but I don't want to prefetch to much. >> >> With the pipe implementation, my reader had many stalls waiting for the >> server to produce the next chunk. The server can produce data a lot >> quicker than the reader can read, however, setup time is high. It's a >> complicated mess :) >> >> On Thu, Nov 21, 2019 at 8:37 PM Robert Engels <ren...@ix.netcom.com >> <javascript:>> wrote: >> >>> The OP specifically requested io.Reader/Writer interfaces. >>> >>> A pipe is what he wants. Not a ring buffer. (A pipe essentially has a >>> ring buffer in the implementation though). >>> >>> > On Nov 21, 2019, at 10:20 PM, Dan Kortschak <d...@kortschak.io >>> <javascript:>> wrote: >>> > >>> > There is this: https://godoc.org/bitbucket.org/ausocean/utils/ring >>> > >>> > It has been used in production fairly extensively. >>> > >>> >> On Thu, 2019-11-21 at 19:47 -0800, Marcin Romaszewicz wrote: >>> >> Hi All, >>> >> >>> >> Before I reinvent the wheel, and because this wheel is particularly >>> >> tricky >>> >> to get right, I was wondering if anyone was aware of a a library >>> >> providing >>> >> something like this >>> >> >>> >> - conforms to io.Reader >>> >> - conforms to io.Writer >>> >> - Contains a buffer of fixed size, say, 64MB. If you try to write >>> >> when the >>> >> buffer is too full, write blocks. When you try to read from an empty >>> >> one, >>> >> read blocks. >>> >> >>> >> This describes the behavior of make(chan byte, 64 * MB), however, it >>> >> doesn't seem to be efficient to do this with a channel. Say I'm >>> >> transferring a few hundred GB via this mechanism. A chan of byte >>> >> would need >>> >> a few hundred billion byte writes, and a few hundred billion reads. >>> >> Doesn't >>> >> sound like it could be efficient at all. You can't implement this >>> >> with a >>> >> channel of []byte, because you'd violate your buffering limit if you >>> >> pass >>> >> too large of a byte array, so you'd have to chop things up into >>> >> blocks, but >>> >> maybe that's simpler than a full fledged blocking ring buffer. >>> >> >>> >> Anyhow, I've implemented such things in various OS level plumbing, so >>> >> I >>> >> know I can do it in Go much more easily, just hoping to avoid it :) >>> >> >>> >> Thanks for any advice, >>> >> -- Marcin >>> >> >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups "golang-nuts" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an email to golan...@googlegroups.com <javascript:>. >>> > To view this discussion on the web visit >>> https://groups.google.com/d/msgid/golang-nuts/152a954eaa3eea02514e71bb142904480241ad6c.camel%40kortschak.io >>> . >>> >>> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/288d479a-95d2-4bc5-914d-0a88f32ede66%40googlegroups.com.
