I'm reading a lot of gzipped responses with Go client, so I'd say that this is probably problematic http server and not the client.
On Thursday, June 23, 2022 at 2:00:18 AM UTC+2 Kevin Chowski wrote: > Do you have a self-contained reproducer? E.g., a single Go program that > hosts a server (either using the server functionality from package > net/http, or hard-coded HTTP responses) and shows that using the client > logic in net/http causes this problem? Preferably shared via > play.golang.org link. If the problem is in the Go standard library, that > is the best avenue for getting the root cause fixed, which usually is > better than using workarounds in your application. > > On Wednesday, June 22, 2022 at 8:41:54 AM UTC-6 minxie...@gmail.com wrote: > >> We've managed to identify the issue and found a temporary solution. >> >> The issue only happens when the stream encoding is "gzip" and the >> uncompressed size is greater than 32KBytes. Side by side we've compared the >> server stream with plain text and gzip. The gzip uncompression is >> automatically engaged if server response stream encoding is detected. If >> the uncompressed data is truncated at 32768 bytes. >> >> >> https://go.googlesource.com/go/+/refs/heads/dev.boringcrypto.go1.17/src/net/http/h2_bundle.go#8835. >> >> (same in 1.19). >> >> The client could use either "Accept-Encoding: identity" header or disable >> the compression in the transport instance to ask for an uncompressed stream. >> >> Look like a Golang net/http bug to me. >> >> On Monday, June 20, 2022 at 3:19:26 AM UTC-5 Brian Candler wrote: >> >>> 1. What's the web server you're connecting to? (Is it written in go, >>> and if so, can you share it?) >>> 2. What happens if you use io.Read instead of io.ReadAtLeast? >>> 3. What happens if you remove the time.Sleep()? (Maybe the target >>> webserver has a timeout which requires the transaction to be completed >>> within 30 seconds?) >>> >>> *io.ReadAtLeast* does exactly what it says: read *at least* that number >>> of bytes. If the sender has nothing more to send, and does not end the >>> stream, then it will wait indefinitely for more data. Note that >>> io.ReadAtLeast never returns io.EOF, but it may return io.ErrUnexpectedEOF >>> if it's unable to read the number of requested bytes. For source code see >>> here >>> <https://cs.opensource.google/go/go/+/refs/tags/go1.18.3:src/io/io.go;l=325> >>> . >>> >>> However, even if the sender does drop the connection, it does seem weird >>> that you don't get an error. (I don't suppose you are going through a >>> badly-configured firewall with an aggressive TCP timeout and which doesn't >>> send RST?) >>> >>> On Monday, 20 June 2022 at 06:44:38 UTC+1 minxie...@gmail.com wrote: >>> >>>> All, >>>> >>>> I got puzzled by how to read a response size > 32KBytes from http >>>> resp.Body. All I get is 32768 bytes. With either io.Read functions or >>>> bufio.Reader. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> *buffer := make([]byte, 1024)var total_bytes_read int for { len, err >>>> := io.ReadAtLeast(res.Body, buffer, 1024) total_bytes_read += len >>>> log.Println(len, "bytes recevied, total received bytes: ", >>>> total_bytes_read) if err != nil { if err == io.EOF { >>>> log.Println("EOF: last chunk received") } else { log.Println(err) >>>> } break } time.Sleep(1 * time.Second)} log.Println("Total bytes >>>> received:", total_bytes_read)err = res.Body.Close()* >>>> >>>> What I see is: >>>> >>>> *{"level":"info","msg":"1024 bytes recevied, total received bytes: >>>> 1024","time":"2022-06-19T19:47:48-05:00"} * >>>> *{"level":"info","msg":"1024 bytes recevied, total received bytes: >>>> 2048","time":"2022-06-19T19:47:49-05:00"} * >>>> >>>> *{"level":"info","msg":"1024 bytes recevied, total received bytes: >>>> 3072","time":"2022-06-19T19:47:50-05:00"}* >>>> *...* >>>> *{"level":"info","msg":"1024 bytes recevied, total received bytes: >>>> 30720","time":"2022-06-19T19:48:17-05:00"}* >>>> *{"level":"info","msg":"1024 bytes recevied, total received bytes: >>>> 31744","time":"2022-06-19T19:48:18-05:00"} * >>>> *{"level":"info","msg":"1024 bytes recevied, total received bytes: >>>> 32768","time":"2022-06-19T19:48:19-05:00"}* >>>> >>>> Then it got stuck and waiting for return from the io.Read. Not sure if >>>> what I did is correct. Got the same observation when using more popular >>>> bufio.Read as well. >>>> >>>> Thanks! >>>> -Min >>>> >>>> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/b0021441-b29d-4356-b56a-a79f78173c71n%40googlegroups.com.
