Package: curl
Version: 7.19.7-1
Tags: patch
If a server returns a Content-Length header and then it uses chunked transfer,
curl often fails with with the following error:
curl: (18) transfer closed with outstanding read data remaining
I think the cause is that when it finds a Content-Length header while parsing
the response, it makes k->maxdownload = contentlength; (transfer.c:1216).
Although it sets k->size = -1; (transfer.c:1014), k->maxdownload is still the
contentlength:
/* According to RFC2616 section 4.4, we MUST ignore
Content-Length: headers if we are now receiving data
using chunked Transfer-Encoding.
*/
if(k->chunk)
k->size=-1;
Later, when the reading from the server (readwrite_data), the following can
happen: if the server wants to send the final "0\r\n\r\n" chunk in a separate
packet, libcurl will hit the condition in transfer.c:635:
if((-1 != k->maxdownload) &&
(k->bytecount + nread >= k->maxdownload)) { ... }
where it sets
k->keepon &= ~KEEP_RECV; /* we're done reading */
and considers reading finished. So it will never read the final chunk, hence
the error.
On the other hand, if the server sends the final chunk in a previous packet
with other data, Curl_httpchunk_read() will happily loop through all of it and
correctly register the final state (CHUNK_STOP).
Setting k->maxdownload to -1 when the transfer encoding is chunked seems to
solve the problem (for me at least):
--- a/lib/transfer.c
+++ b/lib/transfer.c
@@ -1010,8 +1010,10 @@ static CURLcode readwrite_http_headers(struct
SessionHandle *data,
Content-Length: headers if we are now receiving data
using chunked Transfer-Encoding.
*/
- if(k->chunk)
+ if(k->chunk) {
k->size=-1;
+ k->maxdownload=-1;
+ }
}
if(-1 != k->size) {
The curl upstream diverged somewhat already from the version used in Debian,
but the problem persists there also (I checked today's cvs snapshot).
diff --git a/lib/transfer.c b/lib/transfer.c
index 1f69706..b1595b5 100644
--- a/lib/transfer.c
+++ b/lib/transfer.c
@@ -1010,8 +1010,10 @@ static CURLcode readwrite_http_headers(struct SessionHandle *data,
Content-Length: headers if we are now receiving data
using chunked Transfer-Encoding.
*/
- if(k->chunk)
+ if(k->chunk) {
k->size=-1;
+ k->maxdownload=-1;
+ }
}
if(-1 != k->size) {
