On 4/20/2019 3:12 AM, Ray Satiro wrote:
> On 4/19/2019 5:01 PM, Dmitri Trembovetski via curl-library wrote:
>> As with my earlier email to this list, the configuration is
>> self-compiled libcurl with HTTP/2 support (openssl, nghttp2) for my
>> project.
>> curl 7.64.1
>> nghttp2 v1.37.0
>> openssl 1.1.0
>> I've compiled and run the same app on MacOSX, Android (x64), and Linux with
>> the same results.
>>
>> It uses platform zlib for gzip encoding on MacOSX (compatibility
>> version 1.0.0, current version 1.2.11), on linux zlib was 1.2.8
>> (libz.so.1.2.8), and on Android who knows, so it doesn't look like
>> zlib version is the issue.
>>
>> I'm using multi interface.
>>
>> The problem is that if I enable gzip encoding
>> (curl_easy_setopt(handle, CURLOPT_ACCEPT_ENCODING, "gzip")) my GET
>> requests fail with CURLE_WRITE_ERROR.
>> Note that the error is not caused by my WRITE callback's return
>> value, it originates from process_trailer() function in
>> lib/content_encoding.c:
>> static CURLcode process_trailer(struct connectdata *conn, zlib_params *zp)
>> {
>> ...
>> z->next_in += len;
>> if(z->avail_in) {
>> result = CURLE_WRITE_ERROR; <<< here
>> }
>>
>> This is before my WRITE callback is invoked for this batch of data.
>>
>> This happens with HTTP1.1 and HTTP/2 (so unrelated to my earlier
>> issue that I asked about on this list).
>>
>> It doesn't fail on every GET request, looks like only if the request
>> has a few tens of KB of data to read.
>>
>> The interesting thing is that it worked with earlier curl version
>> (7.56.1, I haven't bisected further) in the same exact environments,
>> with the same server, both with HTTP 1.1 and /2 (HTTP/2 didn't work
>> for me on POSTs on 7.56.1 but GETs were fine).
>>
>> Attaching relevant part of the log (taken on MacOSX) with debug
>> tracing enabled and a diff that I applied for my very sophisticated
>> debugging. In the log 0x7f9c79813c00 is the handle which failed.
>>
>> Has something changed between those curl versions? Perhaps some
>> validation was added or something?
> Possibly known bug 1.7 "Deflate error after all content was received"
> [1]. Starting in 7.58.0 there was additional validation added to check
> for unparseable trailer bytes [2]. Later in 7.61.1 it was relaxed a
> little [3].
>
> It's interesting that it fails only on certain body sizes. Given that
> it's not hitting the writefunction we can reconstruct it from the
> trace.log and then try playing it back. At first glance there's some
> null bytes at the end, so that may be something.
>
> [1]: https://github.com/curl/curl/blob/curl-7_64_1/docs/KNOWN_BUGS#L161-L168
> [2]: https://github.com/curl/curl/commit/4acc9d3
> [3]: https://github.com/curl/curl/commit/f8be737
>
The additional data appears to be separately compressed gzip data that
we are treating as an unexpected trailer. I'm not sure whether or not
that's allowed. I opened an issue for it [1], please follow up there if
you have any additional information.
[1]: https://github.com/curl/curl/issues/3796
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
