Do you have a self-contained reproducer? E.g., a single Go program that hosts a server (either using the server functionality from package net/http, or hard-coded HTTP responses) and shows that using the client logic in net/http causes this problem? Preferably shared via play.golang.org link. If the problem is in the Go standard library, that is the best avenue for getting the root cause fixed, which usually is better than using workarounds in your application.
On Wednesday, June 22, 2022 at 8:41:54 AM UTC-6 minxie...@gmail.com wrote: > We've managed to identify the issue and found a temporary solution. > > The issue only happens when the stream encoding is "gzip" and the > uncompressed size is greater than 32KBytes. Side by side we've compared the > server stream with plain text and gzip. The gzip uncompression is > automatically engaged if server response stream encoding is detected. If > the uncompressed data is truncated at 32768 bytes. > > > https://go.googlesource.com/go/+/refs/heads/dev.boringcrypto.go1.17/src/net/http/h2_bundle.go#8835. > > (same in 1.19). > > The client could use either "Accept-Encoding: identity" header or disable > the compression in the transport instance to ask for an uncompressed stream. > > Look like a Golang net/http bug to me. > > On Monday, June 20, 2022 at 3:19:26 AM UTC-5 Brian Candler wrote: > >> 1. What's the web server you're connecting to? (Is it written in go, and >> if so, can you share it?) >> 2. What happens if you use io.Read instead of io.ReadAtLeast? >> 3. What happens if you remove the time.Sleep()? (Maybe the target >> webserver has a timeout which requires the transaction to be completed >> within 30 seconds?) >> >> *io.ReadAtLeast* does exactly what it says: read *at least* that number >> of bytes. If the sender has nothing more to send, and does not end the >> stream, then it will wait indefinitely for more data. Note that >> io.ReadAtLeast never returns io.EOF, but it may return io.ErrUnexpectedEOF >> if it's unable to read the number of requested bytes. For source code see >> here >> <https://cs.opensource.google/go/go/+/refs/tags/go1.18.3:src/io/io.go;l=325> >> . >> >> However, even if the sender does drop the connection, it does seem weird >> that you don't get an error. (I don't suppose you are going through a >> badly-configured firewall with an aggressive TCP timeout and which doesn't >> send RST?) >> >> On Monday, 20 June 2022 at 06:44:38 UTC+1 minxie...@gmail.com wrote: >> >>> All, >>> >>> I got puzzled by how to read a response size > 32KBytes from http >>> resp.Body. All I get is 32768 bytes. With either io.Read functions or >>> bufio.Reader. >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> *buffer := make([]byte, 1024)var total_bytes_read int for { len, err := >>> io.ReadAtLeast(res.Body, buffer, 1024) total_bytes_read += len >>> log.Println(len, "bytes recevied, total received bytes: ", >>> total_bytes_read) if err != nil { if err == io.EOF { >>> log.Println("EOF: last chunk received") } else { log.Println(err) >>> } break } time.Sleep(1 * time.Second)} log.Println("Total bytes >>> received:", total_bytes_read)err = res.Body.Close()* >>> >>> What I see is: >>> >>> *{"level":"info","msg":"1024 bytes recevied, total received bytes: >>> 1024","time":"2022-06-19T19:47:48-05:00"} * >>> *{"level":"info","msg":"1024 bytes recevied, total received bytes: >>> 2048","time":"2022-06-19T19:47:49-05:00"} * >>> >>> *{"level":"info","msg":"1024 bytes recevied, total received bytes: >>> 3072","time":"2022-06-19T19:47:50-05:00"}* >>> *...* >>> *{"level":"info","msg":"1024 bytes recevied, total received bytes: >>> 30720","time":"2022-06-19T19:48:17-05:00"}* >>> *{"level":"info","msg":"1024 bytes recevied, total received bytes: >>> 31744","time":"2022-06-19T19:48:18-05:00"} * >>> *{"level":"info","msg":"1024 bytes recevied, total received bytes: >>> 32768","time":"2022-06-19T19:48:19-05:00"}* >>> >>> Then it got stuck and waiting for return from the io.Read. Not sure if >>> what I did is correct. Got the same observation when using more popular >>> bufio.Read as well. >>> >>> Thanks! >>> -Min >>> >>> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/790828ba-3775-4566-9a0a-0e39852e434en%40googlegroups.com.
