On 2/9/2021 2:32 PM, Adrián Gimeno Balaguer via curl-library wrote:
First of all, the reason why I have split module parts search paths
instead of using the main installation prefixes is due to a permission
issue on my CI/CD setup, not allowing me to publish directly the
resulting OpenSSL/ZLib installed paths (both modules having a separate
build pipeline for reusability purposes). On the other hand, I’ve
confirmed that all generated dependent libraries are being correctly
used in runtime with your indication, comparing the following
resulting output with the versions on the headers:
libcurl/7.74.0-DEV OpenSSL/1.1.1i zlib/1.2.11
Note the program prevents system libraries from leaking in by having
LD_LIBRARY_PATH set pointing to a ‘bin’ subdirectory where the
aforementioned libraries go. Also, setting CURLOPT_ACCEPT_ENCODING to
“” doesn’t make a difference.
So, recently I ran the following for a trial web service which returns
gzipped data:
CURL* pCURL;
pCURL = curl_easy_init();
curl_easy_setopt(pCURL, CURLOPT_URL, "http://httpbin.org/gzip";);
curl_easy_setopt(pCURL, CURLOPT_ACCEPT_ENCODING, "");
curl_easy_setopt(pCURL, CURLOPT_VERBOSE, 1L);
curl_easy_perform(pCURL);
curl_easy_cleanup(pCURL);
The response body is supposed to contain a demo JSON and I see that
the above sample correctly displays the original data in the standard
output:
{
"gzipped": true,
"headers": {
"Accept": "*/*",
"Accept-Encoding": "deflate, gzip",
"Host": "httpbin.org",
"X-Amzn-Trace-Id": "Root=1-6022d6d8-6e2f080d3d5c1e001d076a2a"
},
"method": "GET",
"origin": "81.202.236.251"
}
* Connection #0 to host httpbin.org left intact
So I believe my libcurl is correct. On the other hand, the equivalent
output from the desired server (which is a REST API managed by a third
party), where I’ll also include the relevant response headers, is as
follows:
HTTP/1.1 200 OK
< Date: Tue, 09 Feb 2021 18:39:21 GMT
< Content-Encoding: gzip
< X-Powered-By: Undertow/1
< Content-Type: application/xml;charset=UTF-8
< Content-Length: 758
< Set-Cookie:
TS0171c831=0147f0636e20bb9c874b5ab4aad7726d9800ce6aafa5b7e8da21a7f47ea8dd54ee8da97abd26a11019badc63800405b90d7d95c801;
Path=/
<
* Connection #0 to host the.domain.com left intact
As you see, the Content-Length is positive, while the body section
appears to be missing. In contrast, when I print the response gathered
through my default program logic, which includes CURLOPT_WRITEDATA,
CURLOPT_WRITEFUNCTION setup and the like, I see an initial byte (only)
instead from gzipped content, the next byte being a null terminator.
I’m then able to decompress the data and get the correct output (an
XML file) manually through the ZLib library (passing in the allocated
response which is wrapped in a std::string and should have the exact
Content-Length size).
If relevant, by any chance, their request also expects a certain
gzipped XML file and my lines involving related headers setup are like
the following:
curl_slist* pHTTPHeaders = curl_slist_append(NULL, “Content-Type:
application/octet-stream”); // Required per specifications (unlike
“application/XML”)
pHTTPHeaders = curl_slist_append(pHTTPHeaders , "Content-Encoding: gzip");
curl_easy_setopt(pCURL, CURLOPT_ACCEPT_ENCODING, "gzip");
Finally, the managers of that service confidently claim that they
don’t send the data doubly compressed, but instead “my system probably
compresses it upon receiving it”. I don’t know how this could make any
sense at all. I haven’t yet tried to analyze the network packets to
determine the right data nature.
Thanks for any response in advance.
Regards,
Adrián
*De: *Ray Satiro via curl-library <mailto:curl-library@cool.haxx.se>
*Enviado: *viernes, 5 de febrero de 2021 21:52
*Para: *curl-library@cool.haxx.se <mailto:curl-library@cool.haxx.se>
*CC: *Ray Satiro <mailto:raysat...@yahoo.com>
*Asunto: *Re: CURLOPT_ACCEPT_ENCODING not working (help)
On 2/4/2021 10:07 AM, Adrián Gimeno Balaguer via curl-library wrote:
I’m using a self built libcurl shared library for embedded use in
a C++ application. In the attempt of requesting it to
automatically decompress response data from a remote server of
interest by using a line like the following in my request setup:
curl_easy_setopt(mpCURL, CURLOPT_ACCEPT_ENCODING, "gzip");
The returned data seems to remain compressed. I enabled the
CURLOPT_HEADER option and can see that the server returns positive
Content-Lengths, with few non-human readable characters in the
body content. To be clear, removing the CURLOPT_ACCEPT_ENCODING
option doesn’t make any difference.
The libcurl library compilation is done in a custom automated
CI/CD in the cloud which also compiles OpenSSL and ZLib as shared
libraries in separate pipelines, with libcurl pipeline pulling
from the master branch (from official libcurl’s repo) and a
compilation script like the following:
cmake -Bbuild -DBUILD_SHARED_LIBS=OFF
-DCMAKE_BUILD_TYPE=MinSizeRel -DCMAKE_POSITION_INDEPENDENT_CODE=ON
-DCURL_DISABLE_COOKIES=ON -DCURL_DISABLE_CRYPTO_AUTH=ON
-DCURL_DISABLE_LDAP=ON -DCURL_DISABLE_PROXY=ON -DENABLE_IPV6=OFF
-DENABLE_UNIX_SOCKETS=OFF -DHTTP_ONLY=ON
-DOPENSSL_CRYPTO_LIBRARY=$(Pipeline.Workspace)/ssl/lib/libcrypto.so.1.1
-DOPENSSL_INCLUDE_DIR=$(Pipeline.Workspace)/ssl/include
-DOPENSSL_SSL_LIBRARY=$(Pipeline.Workspace)/ssl/lib/libssl.so.1.1
-DZLIB_INCLUDE_DIR=$(Pipeline.Workspace)/zlib/include
-DZLIB_LIBRARY=$(Pipeline.Workspace)/zlib/lib/libz.so.1 . && cmake
--build build
I can see the following possible relevant build output lines (not
contiguous) that may indicate ZLib gets correctly integrated:
Found ZLIB: /home/vsts/work/1/zlib/lib/libz.so.1 (found version
"1.2.11")
Enabled features: SSL libz AsynchDNS alt-svc HTTPS-proxy
However, I’ve seen in the following answer:
https://stackoverflow.com/a/29966893
<https://stackoverflow.com/a/29966893>. Looking at my generated
libcurl headers, I can’t find any match for “HAVE_LIBZ” (neither
in my build output).
The proper way to specify OpenSSL and zlib locations is by using
OPENSSL_ROOT_DIR [1] and ZLIB_ROOT variables [2], because we call
cmake's find_package and that's what they use. For example:
-DCMAKE_USE_OPENSSL=ON -DOPENSSL_ROOT_DIR=C:\somewhere -DCURL_ZLIB=ON
-DZLIB_ROOT=C:\somewhere
This assumes you've already installed openssl and zlib to those
locations and they have lib,bin,etc. To set the install location when
configuring zlib via cmake you can use -DCMAKE_INSTALL_PREFIX:PATH=.
To set the install location when configuring openssl via Configure you
can use --prefix= and --openssldir=.
Setting ACCEPT_ENCODING to a specific string is almost never used
correctly. What you should do instead is set it to an empty string ""
and libcurl will only send the encodings it actually supports [3].
printf("%s\n", curl_version()) to make sure your program is actually
using the libcurl you built and not some other one in the path.
[1]:
https://github.com/Kitware/CMake/blob/master/Modules/FindOpenSSL.cmake
<https://github.com/Kitware/CMake/blob/master/Modules/FindOpenSSL.cmake>
[2]:
https://github.com/Kitware/CMake/blob/master/Modules/FindZLIB.cmake
<https://github.com/Kitware/CMake/blob/master/Modules/FindZLIB.cmake>
[3]: https://curl.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html
<https://curl.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html>
Please do not top-post it makes the conversation hard to follow. [1]
It looks like you have solved the first problem and your build is able
to decode compressed content. I reiterate don't set ACCEPT_ENCODING to
specific encodings.
Your second problem seems to be with a specific server that you request
gzip encoding (accept-encoding) and it returns gzip encoding
(content-encoding). Does curl_easy_perform return an error? If not then
curl should have received and decoded the data.
[1]: https://curl.se/mail/etiquette.html#Do_Not_Top_Post
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
