[libmicrohttpd] MHD cannot write the entire chunked deflate data (deflate/chunked)

[silvioprog]

Hello everyone,

I've tried to compress a chunked data using ZLib & MHD, but the connection
is closed while the data are transferred. Using the curl tool as client, it
reports "curl: (23) Failed writing data" when we try to get a large file
(about 100 kB). The attached MHD example uses the files "README" and "
ChangeLog", available in the MHD sources, but it fails to send the "
ChangeLog" file, because it is about 100 kB.

It is easy to reproduce the problem. Just download the attached file, and:

$ gcc -o httpsrv  main.c -lmicrohttpd -lz
$ ./httpsrv

Now, open another tab on your terminal, and:

$ curl -v --compressed http://localhost:8080

You will get something like:

[snip]
...
Tue Jul 23 11:32:00 MSK 2017
Updated chunked_example.c to provide real illustration of usage of
chunked encoding. -EG

Thu Jul 13 21:41:00 MSK 2017
Restored SIGPIPE suppression in TLS mode.
* Failed writing data
* stopped the pause stream!
* Closing connection 0
curl: (23) Failed writing data
Added new value MHD_FEATURE_AUTOSUPPRESS

Lastly, comment the line 56 (file = fopen(MHD_DIR "ChangeLog", "rb");) and
uncomment the 57 (/*file = fopen(MHD_DIR "README", "rb");*/), rebuild the
test and use curl again, you will get the the entire content of the README
file.

I've used the latest MHD from trunk (default build, by "./configure && make
install-strip"), and the ZLib version is 1.2.11. I'm on Xubuntu 18.04
64-bit.

Could you confirm if it is some bug in MHD?

Thanks in advance!

P.S.: remember to change the line 7 (#define MHD_DIR "~/libmicrohttpd/") to
avoid application crash, all the errors handling was omitted to make the
example clear and small.

--
Silvio Clécio

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <zlib.h>
#include <microhttpd.h>

#define MHD_DIR "~/libmicrohttpd/" /* change it to your MHD directory */

static int compress_buf(void **buf, size_t *size) {
    Bytef *cbuf;
    uLongf csize;
    int ret;
    csize = compressBound(*size);
    cbuf = malloc(csize);
    if (!cbuf)
        return MHD_NO;
    ret = compress2(cbuf, &csize, *buf, *size, Z_BEST_COMPRESSION);
    if ((ret != Z_OK) || (csize >= *size)) {
        free(cbuf);
        return MHD_NO;
    }
    free(*buf);
    *buf = cbuf;
    *size = csize;
    return MHD_YES;
}

ssize_t read_cb(void *cls, uint64_t pos, char *buf, size_t size) {
    size_t zsize;
    void *zbuf = malloc(size);
    zsize = fread(zbuf, 1, size, cls);
    if (zsize == 0) {
        free(zbuf);
        return MHD_CONTENT_READER_END_OF_STREAM;
    }
    compress_buf(&zbuf, &zsize);
    memcpy(buf, zbuf, zsize);
    free(zbuf);
    return zsize;
}

void free_cb(void *cls) {
    fclose(cls);
}

int ahc_echo(void *cls, struct MHD_Connection *con, const char *url, const char *method, const char *version,
             const char *data, size_t *size, void **ptr) {
    struct MHD_Response *res;
    FILE *file;
    int ret;
    if (!*ptr) {
        *ptr = (void *) 1;
        return MHD_YES;
    }
    *ptr = NULL;
    file = fopen(MHD_DIR "ChangeLog", "rb");
    /*file = fopen(MHD_DIR "README", "rb");*/
    res = MHD_create_response_from_callback(MHD_SIZE_UNKNOWN, 1024, &read_cb, file, &free_cb);
    MHD_add_response_header(res, MHD_HTTP_HEADER_CONTENT_ENCODING, "deflate");
    MHD_add_response_header(res, MHD_HTTP_HEADER_CONTENT_TYPE, "text/plain");
    ret = MHD_queue_response(con, MHD_HTTP_OK, res);
    MHD_destroy_response(res);
    return ret;
}

int main() {
    struct MHD_Daemon *d;
    d = MHD_start_daemon(MHD_USE_AUTO | MHD_USE_INTERNAL_POLLING_THREAD, 8080, NULL, NULL, &ahc_echo, NULL,
                         MHD_OPTION_END);
    getchar();
    MHD_stop_daemon(d);
    return 0;
}