On 4/27/21 10:41 AM, Iron Bug at gmail.com wrote: > Greetings again. > > But I take it, the chunked transfer is the only way for sending large files > that cannot/should not fit in buffer in RAM.
For large files you should use MHD_create_response_from_fd_at_offset64(). That will do the right thing. > Of course, I set unspecified size and intentionally use chunked transfer. > Moreover, I use pipelining for transferring many chunked files in one TCP > session and close it when client doesn't need it anymore to avoid dangling > TCP sessions that system may keep open for quite long time which is not > desireable in my case. I optimize my server part for memory print and it > should not read the whole files content into RAM. So chunked encoding is all > right and very useful. Again, the above method will be MUCH better than you trying to manage your memory footprint. > The proper behavior (I specially checked in standard) when a client sends a > request with "Connection: close" header, the server should complete the > response (whether it's chunked or not) to the request and then close the > connection. And everything works fine until the "Connection: close" request > comes: then server misses the "Transfer-Encoding: chunked" header for the > last response. As I take it, it only has to check that connection is not > closed yet, it should be closed at the end of transfer, after processing the > last client's request. So it's not a bug when application forcibly sets > chunked encoding when connection is closed, it's just a hack to fix the real > bug with lost header in chunked transfer that should be ended with closing > connection. MHD does not use chunked transfer unless it is needed or someone asks for it. Here, it is not needed, _even if_ you provide the reply incrementally. > Btw, if I write here, what do you think about HTTP/2? I found > libmicrohttpd-http2 on the net, but it was a bit outdated and a little buggy. > I fixed bugs and merged both http2 and the latest libmicrohttpd versions > together, so now I use the extended version that supports both HTTP/1.x and > HTTP/2. But I wonder if HTTP/2 (or HTTP/3 that is a draft yet but is already > implemented in main browsers) are going to be supported in the library? Properly supporting HTTP/2 and HTTP/3 will take developer time (and thus funding) that we currently do not have. Evgeny and I would like to see it happen eventually -- alas primarily in the context of the microhttpd2.h / improved v2.0-API effort. Which itself is also stalled because we'd really need to find a huge chunk of time to spend on this to make it happen. So contributions are in principle welcome -- alas it is a huge task that I doubt can be done purely with volunteers in a timely fashion. > Sincerely, Yana A. Kireyonok aka Iron Bug > > On Mon, 26 Apr 2021 12:00:39 -0400 > libmicrohttpd-requ...@gnu.org wrote: > >> Send libmicrohttpd mailing list submissions to >> libmicrohttpd@gnu.org >> >> To subscribe or unsubscribe via the World Wide Web, visit >> https://lists.gnu.org/mailman/listinfo/libmicrohttpd >> or, via email, send a message with subject or body 'help' to >> libmicrohttpd-requ...@gnu.org >> >> You can reach the person managing the list at >> libmicrohttpd-ow...@gnu.org >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of libmicrohttpd digest..." >> >> >> Today's Topics: >> >> 1. Problem with HTTP/1.1 chunked packets when connection is >> about to close (Iron Bug at gmail.com) >> 2. Re: Problem with HTTP/1.1 chunked packets when connection is >> about to close (Evgeny Grin) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Sun, 25 Apr 2021 21:45:58 +0500 >> From: "Iron Bug at gmail.com" <death.iron....@gmail.com> >> To: libmicrohttpd@gnu.org >> Subject: [libmicrohttpd] Problem with HTTP/1.1 chunked packets when >> connection is about to close >> Message-ID: <20210425214558.c6cd4b410fc806e70505d...@gmail.com> >> Content-Type: text/plain; charset=US-ASCII >> >> Greetings. >> >> I use libmicrohttpd in my project and I found one bug in a very specific >> situation with chunked messages. >> >> The scheme I use is keep-alive connection for transferring some files >> (chunked content, HTTP/1.1 pipelining). When transfer is over, connection is >> closed by client request. >> >> I create responses with MHD_create_response_from_callback and the HTTP >> header "Transfer-Encoding: chunked" is automatically added to packet >> headers. But when the transfer is over and the last file requested, client >> sends "Connection: close". Theoretically, by HTTP standard, the server >> should process the request and then close the socket. And it does so, it >> sends the chunked data and closes the connection. But in the response to the >> last request it loses the "Transfer-Encoding: chunked" header and the client >> fails to accept the wrong formatted HTTP packet. If I manually add >> "Transfer-Encoding: chunked" to the last HTTP response, it works fine. >> >> I think the problem hides in this code piece: >> >> file src/microhttpd/connection.c, line 1376, in function >> have_chunked_upload: >> ----------------------------------------------------------------- >> if ( (MHD_NO != keepalive_possible (connection)) && >> (MHD_str_equal_caseless_ (MHD_HTTP_VERSION_1_1, >> connection->version) ) ) >> { >> if (NULL == have_encoding) >> { >> must_add_chunked_encoding = true; >> connection->have_chunked_upload = true; >> } >> ... >> ----------------------------------------------------------------- >> >> The connection is marked as non keep-alive at the moment after "Connection: >> close" and the content is still chunked. But the flag is not set and then >> the error occurs. >> >> Sincerely, Yana A. Kireyonok aka Iron Bug >> >> >> >> ------------------------------ >> >> Message: 2 >> Date: Mon, 26 Apr 2021 16:42:20 +0300 >> From: Evgeny Grin <k...@yandex.ru> >> To: libmicrohttpd@gnu.org >> Subject: Re: [libmicrohttpd] Problem with HTTP/1.1 chunked packets >> when connection is about to close >> Message-ID: <67616383-2a38-5ddc-353c-696456da1...@yandex.ru> >> Content-Type: text/plain; charset="utf-8"; Format="flowed" >> >> Привет, Яна! >> >> Actually this behaviour is not a bug. >> Chunked encoding produces binary larger output and makes heavier system >> and network load, so MHD tries to avoid chunked output if possible. >> >> If you provide response size as the first argument for >> MHD_create_response_from_callback() MHD will never use chunked encoding. >> >> If you provide "-1" (MHD_SIZE_UNKNOWN) as the size then MHD will follow >> the next logic: >> * if connection is "keep-alive" then only choice is chunked encoding and >> chunked encoding is used, >> * if connection have to be closed after sending response then end of >> response can be indicated by end of connection so "heavier" chunked >> encoding is not used. >> >> This logic most probably will be re-considered soon as latest HTTP/1.1 >> specification forbids responses without both encoding and content-size >> (and ending connection without specifying content size is unreliable). >> >> However I discovered bug related to chunked encoding (when application >> forcibly sets "Transfer-Encoding: chunked") and "Connection: close". It >> will be fixed in the next MHD release. >> >> -- >> Wishes, >> Evgeny >> >> On 25.04.2021 19:45, Iron Bug at gmail.com wrote: >>> Greetings. >>> >>> I use libmicrohttpd in my project and I found one bug in a very specific >>> situation with chunked messages. >>> >>> The scheme I use is keep-alive connection for transferring some files >>> (chunked content, HTTP/1.1 pipelining). When transfer is over, connection >>> is closed by client request. >>> >>> I create responses with MHD_create_response_from_callback and the HTTP >>> header "Transfer-Encoding: chunked" is automatically added to packet >>> headers. But when the transfer is over and the last file requested, client >>> sends "Connection: close". Theoretically, by HTTP standard, the server >>> should process the request and then close the socket. And it does so, it >>> sends the chunked data and closes the connection. But in the response to >>> the last request it loses the "Transfer-Encoding: chunked" header and the >>> client fails to accept the wrong formatted HTTP packet. If I manually add >>> "Transfer-Encoding: chunked" to the last HTTP response, it works fine. >>> >>> I think the problem hides in this code piece: >>> >>> file src/microhttpd/connection.c, line 1376, in function >>> have_chunked_upload: >>> ----------------------------------------------------------------- >>> if ( (MHD_NO != keepalive_possible (connection)) && >>> (MHD_str_equal_caseless_ (MHD_HTTP_VERSION_1_1, >>> connection->version) ) ) >>> { >>> if (NULL == have_encoding) >>> { >>> must_add_chunked_encoding = true; >>> connection->have_chunked_upload = true; >>> } >>> ... >>> ----------------------------------------------------------------- >>> >>> The connection is marked as non keep-alive at the moment after "Connection: >>> close" and the content is still chunked. But the flag is not set and >> then the error occurs. >>> >>> Sincerely, Yana A. Kireyonok aka Iron Bug >>> >> >> -------------- next part -------------- >> A non-text attachment was scrubbed... >> Name: OpenPGP_signature >> Type: application/pgp-signature >> Size: 840 bytes >> Desc: OpenPGP digital signature >> URL: >> <https://lists.gnu.org/archive/html/libmicrohttpd/attachments/20210426/955bfc0a/attachment.sig> >> >> ------------------------------ >> >> Subject: Digest Footer >> >> _______________________________________________ >> libmicrohttpd mailing list >> libmicrohttpd@gnu.org >> https://lists.gnu.org/mailman/listinfo/libmicrohttpd >> >> >> ------------------------------ >> >> End of libmicrohttpd Digest, Vol 144, Issue 2 >> ********************************************* > >
signature.asc
Description: OpenPGP digital signature
