Since http2 multiplexes streams it will delicately affect latency on other 
streams. This is why I suggested using multiple transports - one for high 
throughput transfers and another for lower latency “interactive” sessions. 

> On Nov 15, 2021, at 9:23 AM, Kevin Chowski <ke...@chowski.com> wrote:
> 
> These are interesting results, thanks for investigating and sharing results!
> 
> I see that you have mostly been focusing on throughput in your posts, have 
> you done testing for latency differences too?
> 
>> On Saturday, November 13, 2021 at 6:11:40 PM UTC-7 ren...@ix.netcom.com 
>> wrote:
>> As another data point, I decided to test a few implementations of http2 
>> downloads on OSX.
>> 
>> Using a Go server with default frame size (16k):
>> 
>> Go client:  900 MB/s
>> Java client: 1300 MB/s
>> curl: 1500 MB/s
>> 
>> Using a Java server with default frame size (16k):
>> 
>> Go client: 670 MB/s
>> Java client: 720 MB/s
>> curl: 800 M/s
>> 
>> Using Go server using 256k client max frame size:
>> 
>> Go client: 2350 MB/s
>> Java client: 2800 MB/s
>> h2load: 4300 MB/s
>> 
>> Using Java server using 256k client max frame size:
>> 
>> Go client: 2900 MB/s
>> Java client: 2800 MB/s
>> h2load: 3750 MB/s
>> 
>> For h2load, I needed to create a PR to allow the frame size to be set, see 
>> https://github.com/nghttp2/nghttp2/pull/1640
>> 
>> 
>>> On Nov 10, 2021, at 7:04 PM, robert engels <ren...@ix.netcom.com> wrote:
>>> 
>>> No worries. I updated the issue and the CL. I will comment in the CL with a 
>>> few more details.
>>> 
>>>> On Nov 10, 2021, at 2:30 PM, Andrey T. <xnow4f...@sneakemail.com> wrote:
>>>> 
>>>> Thank you Robert, 
>>>> I somehow missed the reference to the ticket in the first message, sorry 
>>>> about that.
>>>> 
>>>> As for the CL - I think adding link to the github issue, and add a bit of 
>>>> explanation in a commit message would help.
>>>> I added link to your CL to the github issue's discussion, hopefully it 
>>>> will bring more attention to it. 
>>>> 
>>>> A.
>>>> 
>>>>> On Wednesday, November 10, 2021 at 1:22:42 PM UTC-7 ren...@ix.netcom.com 
>>>>> wrote:
>>>>> As reported in the OP, the issue was filed long ago 
>>>>> https://github.com/golang/go/issues/47840
>>>>> 
>>>>> My CL https://go-review.googlesource.com/c/net/+/362834 is a viable fix 
>>>>> (and should of been supported originally).
>>>>> 
>>>>>>> On Nov 10, 2021, at 12:59 PM, Andrey T. <xnow4f...@sneakemail.com> 
>>>>>>> wrote:
>>>>>>> 
>>>>>> Fellas, 
>>>>>> I would say the 5x throughput difference is a serious problem.Would you 
>>>>>> be kind and open an issue on github about it? 
>>>>>> Also, the PR that you have might benefit from explanation about what you 
>>>>>> are trying to solve (and probably link to an issue on github), so it 
>>>>>> would get more attention. 
>>>>>> 
>>>>>> Thanks!
>>>>>> 
>>>>>> Andrey
>>>>>> 
>>>>>> 
>>>>>>> On Tuesday, November 9, 2021 at 4:50:34 PM UTC-7 ren...@ix.netcom.com 
>>>>>>> wrote:
>>>>>>> Well, I figured out a way to do it simply. The CL is here 
>>>>>>> https://go-review.googlesource.com/c/net/+/362834
>>>>>>> 
>>>>>>> The frame size will be used for all connections using that transport, 
>>>>>>> so it is probably better to create a transport specifically for the 
>>>>>>> high-throughput transfers.
>>>>>>> 
>>>>>>> You can also create perform single shot requests like:
>>>>>>> 
>>>>>>> if useH2C {
>>>>>>>    rt = &http2.Transport{
>>>>>>>       AllowHTTP: true,
>>>>>>>       DialTLS: func(network, addr string, cfg *tls.Config) (net.Conn, 
>>>>>>> error) {
>>>>>>>          return dialer.Dial(network, addr)
>>>>>>>       },
>>>>>>>       MaxFrameSize: 1024*256,
>>>>>>>    }
>>>>>>> }
>>>>>>> 
>>>>>>> var body io.ReadCloser = http.NoBody
>>>>>>> 
>>>>>>> req, err := http.NewRequestWithContext(ctx, "GET", url, body)
>>>>>>> if err != nil {
>>>>>>>    return err
>>>>>>> }
>>>>>>> 
>>>>>>> resp, err := rt.RoundTrip(req)
>>>>>>> 
>>>>>>> 
>>>>>>>> On Nov 9, 2021, at 3:31 PM, Robert Engels <ren...@ix.netcom.com> wrote:
>>>>>>>> 
>>>>>>>> To be clear, I have no plans to submit a Cl to improve this at this 
>>>>>>>> time. 
>>>>>>>> 
>>>>>>>> It would require some api changes to implement properly. 
>>>>>>>> 
>>>>>>>>>> On Nov 9, 2021, at 12:19 PM, Kirth Gersen <kirth...@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>> Great !
>>>>>>>>> 
>>>>>>>>> > I made some local mods to the net library, increasing the frame 
>>>>>>>>> > size to 256k, and the http2 performance went from 8Gbps to 38Gbps.
>>>>>>>>> That is already enormous for us. thx for finding this.
>>>>>>>>> 
>>>>>>>>> 4 -> Indeed  a lot of WINDOW_UPDATE messages are visible when using 
>>>>>>>>> GODEBUG=http2debug=1 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Tuesday, November 9, 2021 at 6:28:16 PM UTC+1 
>>>>>>>>>> ren...@ix.netcom.com wrote:
>>>>>>>>>> I did a review of the codebase.
>>>>>>>>>> 
>>>>>>>>>> Http2 is a multiplexed protocol with independent streams. The Go 
>>>>>>>>>> implementation uses a common reader thread/routine to read all of 
>>>>>>>>>> the connection content, and then demuxes the streams and passes the 
>>>>>>>>>> data via pipes to the stream readers. This multithreaded nature 
>>>>>>>>>> requires the use of locks to coordinate. By managing the window 
>>>>>>>>>> size, the connection reader should never block writing to a steam 
>>>>>>>>>> buffer - but a stream reader may stall waiting for data to arrive - 
>>>>>>>>>> get descheduled - only to be quickly rescheduled when reader places 
>>>>>>>>>> more data in the buffer - which is inefficient.
>>>>>>>>>> 
>>>>>>>>>> Out of the box on my machine, http1 is about 37 Gbps, and http2 is 
>>>>>>>>>> about 7 Gbps on my system.
>>>>>>>>>> 
>>>>>>>>>> Some things that jump out:
>>>>>>>>>> 
>>>>>>>>>> 1. The chunk size is too small. Using 1MB pushed http1 from 37 Gbs 
>>>>>>>>>> to 50 Gbps, and http2 to 8 Gbps.
>>>>>>>>>> 
>>>>>>>>>> 2. The default buffer in io.Copy() is too small. Use io.CopyBuffer() 
>>>>>>>>>> with a larger buffer - I changed to 4MB. This pushed http1 to 55 
>>>>>>>>>> Gbs, and http2 to 8.2. Not a big difference but needed for later.
>>>>>>>>>> 
>>>>>>>>>> 3. The http2 receiver frame size of 16k is way too small. There is 
>>>>>>>>>> overhead on every frame - the most costly is updating the window.
>>>>>>>>>> 
>>>>>>>>>> I made some local mods to the net library, increasing the frame size 
>>>>>>>>>> to 256k, and the http2 performance went from 8Gbps to 38Gbps.
>>>>>>>>>> 
>>>>>>>>>> 4. I haven’t tracked it down yet, but I don’t think the window size 
>>>>>>>>>> update code is not working as intended - it seems to be sending 
>>>>>>>>>> window updates (which are expensive due to locks) far too 
>>>>>>>>>> frequently. I think this is the area that could use the most 
>>>>>>>>>> improvement - using some heuristics there is the possibility to 
>>>>>>>>>> detect the sender rate, and adjust the refresh rate (using high/low 
>>>>>>>>>> water marks).
>>>>>>>>>> 
>>>>>>>>>> 5. The implementation might need improvements using lock-free 
>>>>>>>>>> structures, atomic counters, and busy-waits in order to achieve 
>>>>>>>>>> maximum performance.
>>>>>>>>>> 
>>>>>>>>>> So 38Gbps for http2 vs 55 Gbps for http1. Better but still not 
>>>>>>>>>> great. Still, with some minor changes, the net package could allow 
>>>>>>>>>> setting of a large frame size on a per stream basis - which would 
>>>>>>>>>> enable much higher throughput. The gRPC library allows this.
>>>>>>>>>> 
>>>>>>>>>>>> On Nov 8, 2021, at 10:58 AM, Kirth Gersen <kirth...@gmail.com> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>> http/2 implementation seems ~5x slower in bytes per seconds (when 
>>>>>>>>>>> transfer is cpu capped).
>>>>>>>>>>> 
>>>>>>>>>>> POC: https://github.com/nspeed-app/http2issue
>>>>>>>>>>> 
>>>>>>>>>>> I submitted an issue about this 3 months ago in the Go Github ( 
>>>>>>>>>>> https://github.com/golang/go/issues/47840 ) but first commenter 
>>>>>>>>>>> misunderstood it and it got buried (they're probably just swamped 
>>>>>>>>>>> with too many open issues (5k+...)).
>>>>>>>>>>> 
>>>>>>>>>>> Everything using Golang net/http is impacted, the Caddy web server 
>>>>>>>>>>> for instance.
>>>>>>>>>>> 
>>>>>>>>>>> I know it probably doesn't matter for most use cases because it's 
>>>>>>>>>>> only noticeable with high throughput transfers (>1 Gbps). 
>>>>>>>>>>> Most http benchmarks focus on "requests per second" and not "bits 
>>>>>>>>>>> per seconds" but this performance matters too sometimes.
>>>>>>>>>>> 
>>>>>>>>>>> If anyone with expertise in profiling Go code and good knowledge of 
>>>>>>>>>>> the net/http lib internal could take a look. It would be nice to 
>>>>>>>>>>> optimize it or at least have an explanation.
>>>>>>>>>>> 
>>>>>>>>>>> thx (sorry if wrong  group to post this).
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> -- 
>>>>>>>>>>> You received this message because you are subscribed to the Google 
>>>>>>>>>>> Groups "golang-nuts" group.
>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, 
>>>>>>>>>>> send an email to golang-nuts...@googlegroups.com.
>>>>>>>>>>> To view this discussion on the web visit 
>>>>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/89926c2f-ec73-43ad-be49-a8bc76a18345n%40googlegroups.com.
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -- 
>>>>>>>>> You received this message because you are subscribed to the Google 
>>>>>>>>> Groups "golang-nuts" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, 
>>>>>>>>> send an email to golang-nuts...@googlegroups.com.
>>>>>>>>> To view this discussion on the web visit 
>>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/7332f727-6716-4c4d-85c5-a86cacd0c89fn%40googlegroups.com.
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -- 
>>>>>> You received this message because you are subscribed to the Google 
>>>>>> Groups "golang-nuts" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it, send 
>>>>>> an email to golang-nuts...@googlegroups.com.
>>>>> 
>>>>>> To view this discussion on the web visit 
>>>>>> https://groups.google.com/d/msgid/golang-nuts/1bfe6aec-abd2-4f63-bf77-bbfa6fd213ban%40googlegroups.com.
>>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> You received this message because you are subscribed to the Google Groups 
>>>> "golang-nuts" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>>> email to golang-nuts...@googlegroups.com.
>>>> To view this discussion on the web visit 
>>>> https://groups.google.com/d/msgid/golang-nuts/1b63863b-45af-45d0-a885-8716acc65ac7n%40googlegroups.com.
>>> 
>> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/3138434b-d480-4473-8b20-2598412a0eden%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1064ECEA-E5CE-4021-B27B-7861302609F8%40ix.netcom.com.

Reply via email to