Since http2 multiplexes streams it will delicately affect latency on other streams. This is why I suggested using multiple transports - one for high throughput transfers and another for lower latency “interactive” sessions.
> On Nov 15, 2021, at 9:23 AM, Kevin Chowski <ke...@chowski.com> wrote: > > These are interesting results, thanks for investigating and sharing results! > > I see that you have mostly been focusing on throughput in your posts, have > you done testing for latency differences too? > >> On Saturday, November 13, 2021 at 6:11:40 PM UTC-7 ren...@ix.netcom.com >> wrote: >> As another data point, I decided to test a few implementations of http2 >> downloads on OSX. >> >> Using a Go server with default frame size (16k): >> >> Go client: 900 MB/s >> Java client: 1300 MB/s >> curl: 1500 MB/s >> >> Using a Java server with default frame size (16k): >> >> Go client: 670 MB/s >> Java client: 720 MB/s >> curl: 800 M/s >> >> Using Go server using 256k client max frame size: >> >> Go client: 2350 MB/s >> Java client: 2800 MB/s >> h2load: 4300 MB/s >> >> Using Java server using 256k client max frame size: >> >> Go client: 2900 MB/s >> Java client: 2800 MB/s >> h2load: 3750 MB/s >> >> For h2load, I needed to create a PR to allow the frame size to be set, see >> https://github.com/nghttp2/nghttp2/pull/1640 >> >> >>> On Nov 10, 2021, at 7:04 PM, robert engels <ren...@ix.netcom.com> wrote: >>> >>> No worries. I updated the issue and the CL. I will comment in the CL with a >>> few more details. >>> >>>> On Nov 10, 2021, at 2:30 PM, Andrey T. <xnow4f...@sneakemail.com> wrote: >>>> >>>> Thank you Robert, >>>> I somehow missed the reference to the ticket in the first message, sorry >>>> about that. >>>> >>>> As for the CL - I think adding link to the github issue, and add a bit of >>>> explanation in a commit message would help. >>>> I added link to your CL to the github issue's discussion, hopefully it >>>> will bring more attention to it. >>>> >>>> A. >>>> >>>>> On Wednesday, November 10, 2021 at 1:22:42 PM UTC-7 ren...@ix.netcom.com >>>>> wrote: >>>>> As reported in the OP, the issue was filed long ago >>>>> https://github.com/golang/go/issues/47840 >>>>> >>>>> My CL https://go-review.googlesource.com/c/net/+/362834 is a viable fix >>>>> (and should of been supported originally). >>>>> >>>>>>> On Nov 10, 2021, at 12:59 PM, Andrey T. <xnow4f...@sneakemail.com> >>>>>>> wrote: >>>>>>> >>>>>> Fellas, >>>>>> I would say the 5x throughput difference is a serious problem.Would you >>>>>> be kind and open an issue on github about it? >>>>>> Also, the PR that you have might benefit from explanation about what you >>>>>> are trying to solve (and probably link to an issue on github), so it >>>>>> would get more attention. >>>>>> >>>>>> Thanks! >>>>>> >>>>>> Andrey >>>>>> >>>>>> >>>>>>> On Tuesday, November 9, 2021 at 4:50:34 PM UTC-7 ren...@ix.netcom.com >>>>>>> wrote: >>>>>>> Well, I figured out a way to do it simply. The CL is here >>>>>>> https://go-review.googlesource.com/c/net/+/362834 >>>>>>> >>>>>>> The frame size will be used for all connections using that transport, >>>>>>> so it is probably better to create a transport specifically for the >>>>>>> high-throughput transfers. >>>>>>> >>>>>>> You can also create perform single shot requests like: >>>>>>> >>>>>>> if useH2C { >>>>>>> rt = &http2.Transport{ >>>>>>> AllowHTTP: true, >>>>>>> DialTLS: func(network, addr string, cfg *tls.Config) (net.Conn, >>>>>>> error) { >>>>>>> return dialer.Dial(network, addr) >>>>>>> }, >>>>>>> MaxFrameSize: 1024*256, >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> var body io.ReadCloser = http.NoBody >>>>>>> >>>>>>> req, err := http.NewRequestWithContext(ctx, "GET", url, body) >>>>>>> if err != nil { >>>>>>> return err >>>>>>> } >>>>>>> >>>>>>> resp, err := rt.RoundTrip(req) >>>>>>> >>>>>>> >>>>>>>> On Nov 9, 2021, at 3:31 PM, Robert Engels <ren...@ix.netcom.com> wrote: >>>>>>>> >>>>>>>> To be clear, I have no plans to submit a Cl to improve this at this >>>>>>>> time. >>>>>>>> >>>>>>>> It would require some api changes to implement properly. >>>>>>>> >>>>>>>>>> On Nov 9, 2021, at 12:19 PM, Kirth Gersen <kirth...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>> Great ! >>>>>>>>> >>>>>>>>> > I made some local mods to the net library, increasing the frame >>>>>>>>> > size to 256k, and the http2 performance went from 8Gbps to 38Gbps. >>>>>>>>> That is already enormous for us. thx for finding this. >>>>>>>>> >>>>>>>>> 4 -> Indeed a lot of WINDOW_UPDATE messages are visible when using >>>>>>>>> GODEBUG=http2debug=1 >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Tuesday, November 9, 2021 at 6:28:16 PM UTC+1 >>>>>>>>>> ren...@ix.netcom.com wrote: >>>>>>>>>> I did a review of the codebase. >>>>>>>>>> >>>>>>>>>> Http2 is a multiplexed protocol with independent streams. The Go >>>>>>>>>> implementation uses a common reader thread/routine to read all of >>>>>>>>>> the connection content, and then demuxes the streams and passes the >>>>>>>>>> data via pipes to the stream readers. This multithreaded nature >>>>>>>>>> requires the use of locks to coordinate. By managing the window >>>>>>>>>> size, the connection reader should never block writing to a steam >>>>>>>>>> buffer - but a stream reader may stall waiting for data to arrive - >>>>>>>>>> get descheduled - only to be quickly rescheduled when reader places >>>>>>>>>> more data in the buffer - which is inefficient. >>>>>>>>>> >>>>>>>>>> Out of the box on my machine, http1 is about 37 Gbps, and http2 is >>>>>>>>>> about 7 Gbps on my system. >>>>>>>>>> >>>>>>>>>> Some things that jump out: >>>>>>>>>> >>>>>>>>>> 1. The chunk size is too small. Using 1MB pushed http1 from 37 Gbs >>>>>>>>>> to 50 Gbps, and http2 to 8 Gbps. >>>>>>>>>> >>>>>>>>>> 2. The default buffer in io.Copy() is too small. Use io.CopyBuffer() >>>>>>>>>> with a larger buffer - I changed to 4MB. This pushed http1 to 55 >>>>>>>>>> Gbs, and http2 to 8.2. Not a big difference but needed for later. >>>>>>>>>> >>>>>>>>>> 3. The http2 receiver frame size of 16k is way too small. There is >>>>>>>>>> overhead on every frame - the most costly is updating the window. >>>>>>>>>> >>>>>>>>>> I made some local mods to the net library, increasing the frame size >>>>>>>>>> to 256k, and the http2 performance went from 8Gbps to 38Gbps. >>>>>>>>>> >>>>>>>>>> 4. I haven’t tracked it down yet, but I don’t think the window size >>>>>>>>>> update code is not working as intended - it seems to be sending >>>>>>>>>> window updates (which are expensive due to locks) far too >>>>>>>>>> frequently. I think this is the area that could use the most >>>>>>>>>> improvement - using some heuristics there is the possibility to >>>>>>>>>> detect the sender rate, and adjust the refresh rate (using high/low >>>>>>>>>> water marks). >>>>>>>>>> >>>>>>>>>> 5. The implementation might need improvements using lock-free >>>>>>>>>> structures, atomic counters, and busy-waits in order to achieve >>>>>>>>>> maximum performance. >>>>>>>>>> >>>>>>>>>> So 38Gbps for http2 vs 55 Gbps for http1. Better but still not >>>>>>>>>> great. Still, with some minor changes, the net package could allow >>>>>>>>>> setting of a large frame size on a per stream basis - which would >>>>>>>>>> enable much higher throughput. The gRPC library allows this. >>>>>>>>>> >>>>>>>>>>>> On Nov 8, 2021, at 10:58 AM, Kirth Gersen <kirth...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>> http/2 implementation seems ~5x slower in bytes per seconds (when >>>>>>>>>>> transfer is cpu capped). >>>>>>>>>>> >>>>>>>>>>> POC: https://github.com/nspeed-app/http2issue >>>>>>>>>>> >>>>>>>>>>> I submitted an issue about this 3 months ago in the Go Github ( >>>>>>>>>>> https://github.com/golang/go/issues/47840 ) but first commenter >>>>>>>>>>> misunderstood it and it got buried (they're probably just swamped >>>>>>>>>>> with too many open issues (5k+...)). >>>>>>>>>>> >>>>>>>>>>> Everything using Golang net/http is impacted, the Caddy web server >>>>>>>>>>> for instance. >>>>>>>>>>> >>>>>>>>>>> I know it probably doesn't matter for most use cases because it's >>>>>>>>>>> only noticeable with high throughput transfers (>1 Gbps). >>>>>>>>>>> Most http benchmarks focus on "requests per second" and not "bits >>>>>>>>>>> per seconds" but this performance matters too sometimes. >>>>>>>>>>> >>>>>>>>>>> If anyone with expertise in profiling Go code and good knowledge of >>>>>>>>>>> the net/http lib internal could take a look. It would be nice to >>>>>>>>>>> optimize it or at least have an explanation. >>>>>>>>>>> >>>>>>>>>>> thx (sorry if wrong group to post this). >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>>>> Groups "golang-nuts" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>>> send an email to golang-nuts...@googlegroups.com. >>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/89926c2f-ec73-43ad-be49-a8bc76a18345n%40googlegroups.com. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "golang-nuts" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to golang-nuts...@googlegroups.com. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/7332f727-6716-4c4d-85c5-a86cacd0c89fn%40googlegroups.com. >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "golang-nuts" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>> an email to golang-nuts...@googlegroups.com. >>>>> >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/golang-nuts/1bfe6aec-abd2-4f63-bf77-bbfa6fd213ban%40googlegroups.com. >>>>> >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "golang-nuts" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to golang-nuts...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/golang-nuts/1b63863b-45af-45d0-a885-8716acc65ac7n%40googlegroups.com. >>> >> > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/3138434b-d480-4473-8b20-2598412a0eden%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/1064ECEA-E5CE-4021-B27B-7861302609F8%40ix.netcom.com.