Re: [libmicrohttpd] MHD_quiesce_daemon() question
Fixed version of you code:
bool bf_httpsrv_shutdown(struct bf_httpsrv *srv) { MHD_socket fd;
uint8_t shutdown_attempts; /* Max 256. */ if (srv && srv->listening) {
fd = MHD_quiesce_daemon(srv->mhd); if (fd != MHD_INVALID_SOCKET) {
srv->listening = false; close(fd); if (! srv->forced_shutdown) {
shutdown_attempts = 1; while (MHD_get_daemon_info(srv->mhd,
MHD_DAEMON_INFO_CURRENT_CONNECTIONS)->num_connections > 0) { if
(shutdown_attempts >= srv->shutdown_attempts) { /* Default
srv->shutdown_attempts is 10. */ _BF_LOG("Forced server shutdown.\n");
break; } shutdown_attempts++; sleep(1); } } } else _BF_LOG("Server
quiesce failed.\n"); MHD_stop_daemon(srv->mhd); return true; } return
false; }
--
Best Wishes,
Evgeny Grin
On 28.03.2017 4:46, silvioprog wrote:
> Thanks for replying Evgeny, you and Christian always saving my day. :-)
> I read and re-read your and Christian answer, and finally I found a
> possible way to never lock the server at shutdown: adding an option to
> exit the server after configurable attempts. It can solve the timeout
> problem too, because I can't ensure that the programmer have
> configured it (anyway my library will assume 15 seconds by default).
> So, the code below applies your tips and implements the attempts approach:
> bool bf_httpsrv_shutdown(struct bf_httpsrv *srv) { MHD_socket fd;
> uint8_t shutdown_attempts; /* Max 256. */ if (srv && srv->listening) {
> fd = MHD_quiesce_daemon(srv->mhd); if (fd != MHD_INVALID_SOCKET)
> close(fd); else _BF_LOG("Server quiesce failed.\n"); if
> (srv->forced_shutdown) { shutdown_attempts = 1; while
> (MHD_get_daemon_info(srv->mhd,
> MHD_DAEMON_INFO_CURRENT_CONNECTIONS)->num_connections > 0) { if
> (shutdown_attempts >= srv->shutdown_attempts) { /* Default
> srv->shutdown_attempts is 10. */ _BF_LOG("Forced server shutdown.\n");
> exit(EINTR); } shutdown_attempts++; sleep(1); } }
> MHD_stop_daemon(srv->mhd); srv->listening = false; return true; }
> return false; }
> Unfortunately if exit was called the MHD_stop_daemon() will be not
> called, probably raising some memory leak, but current I have no idea
> how it could be solved.
> Feel totally free to point improvements about this new version. :-)
> On Sun, Mar 26, 2017 at 5:36 PM, Evgeny Grin > wrote:
>
> On 26.03.2017 8:33, silvioprog wrote: > I found the following
> related message: > >
> https://lists.gnu.org/archive/html/libmicrohttpd/2014-09/msg00012.html
>
> > > I've used a similar logic, but with item X below, because I
> need to wait > the client processing: > > 1) MHD_quiesce_daemon()
> > *X) while (info.num_connections > 0) sleep(0.5s) # pseudo code*
> > 2) MHD_stop_daemon() > 3) close() > > Real implementation: > >
> bool bf_httpsrv_shutdown(struct bf_httpsrv *srv, bool force) { >
>MHD_socket fd; > if (srv && srv->listening) { > fd
> = MHD_quiesce_daemon(srv->mhd); > if (!force) >
> while (MHD_get_daemon_info(srv->mhd,
> MHD_DAEMON_INFO_CURRENT_CONNECTIONS)->num_connections > 0) >
>usleep(1000 * 500); //TODO: allow to use external
> callback > MHD_stop_daemon(srv->mhd); > if (fd !=
> MHD_INVALID_SOCKET) > close(fd); >
> srv->listening = false; > return true; > } >
> return false; > } > > > Calling it with bf_httpsrv_shutdown(srv,
> false) the server stops waiting > for clients processing. > > So,
> what do you think about the logic above? Should it be improved?! >
> > Thanks in advance for any suggestions! If you don't check
> returned value from MHD_quiesce_daemon(), you may later found that
> you didn't quiesced daemon, so move check right after
> MHD_quiesce_daemon() and added error handling. If you didn't set
> connection timeout, connection may live indefinitely. Moreover,
> even with connection timeout, clients may continue processing on
> same HTTP 1.1 connections with new requests indefinitely.
> Furthermore, even with HTTP 1.0 and connection timeout
> hypothetical client may read answer very slow with results in very
> long unpredictable closure of connection. So: yes, you code will
> work but may result in very long (hours, for example) or even
> indefinitely long daemon shutdown. -- Best Wishes, Evgeny Grin
>
>
> --
> Silvio Clécio
Re: [libmicrohttpd] MHD_quiesce_daemon() question
Dude, thanks for fixing it, I'm going to study the changes and apply it
here. :-)
I've got a situation that locks the shutdown even using this fix, but I
need to create a small project simulating the problem, I'm going to do it
now ...
On Tue, Mar 28, 2017 at 12:08 PM, Evgeny Grin wrote:
> Fixed version of you code:
>
> bool bf_httpsrv_shutdown(struct bf_httpsrv *srv) { MHD_socket fd;
> uint8_t shutdown_attempts; /* Max 256. */ if (srv && srv->listening) {
> fd = MHD_quiesce_daemon(srv->mhd); if (fd != MHD_INVALID_SOCKET) {
> srv->listening = false; close(fd); if (! srv->forced_shutdown) {
> shutdown_attempts = 1; while (MHD_get_daemon_info(srv->mhd,
> MHD_DAEMON_INFO_CURRENT_CONNECTIONS)->num_connections > 0) { if
> (shutdown_attempts >= srv->shutdown_attempts) { /* Default
> srv->shutdown_attempts is 10. */ _BF_LOG("Forced server shutdown.\n");
> break; } shutdown_attempts++; sleep(1); } } } else _BF_LOG("Server
> quiesce failed.\n"); MHD_stop_daemon(srv->mhd); return true; } return
> false; }
>
> --
> Best Wishes,
> Evgeny Grin
>
> On 28.03.2017 4:46, silvioprog wrote:
> > Thanks for replying Evgeny, you and Christian always saving my day. :-)
> > I read and re-read your and Christian answer, and finally I found a
> > possible way to never lock the server at shutdown: adding an option to
> > exit the server after configurable attempts. It can solve the timeout
> > problem too, because I can't ensure that the programmer have
> > configured it (anyway my library will assume 15 seconds by default).
> > So, the code below applies your tips and implements the attempts
> approach:
> > bool bf_httpsrv_shutdown(struct bf_httpsrv *srv) { MHD_socket fd;
> > uint8_t shutdown_attempts; /* Max 256. */ if (srv && srv->listening) {
> > fd = MHD_quiesce_daemon(srv->mhd); if (fd != MHD_INVALID_SOCKET)
> > close(fd); else _BF_LOG("Server quiesce failed.\n"); if
> > (srv->forced_shutdown) { shutdown_attempts = 1; while
> > (MHD_get_daemon_info(srv->mhd,
> > MHD_DAEMON_INFO_CURRENT_CONNECTIONS)->num_connections > 0) { if
> > (shutdown_attempts >= srv->shutdown_attempts) { /* Default
> > srv->shutdown_attempts is 10. */ _BF_LOG("Forced server shutdown.\n");
> > exit(EINTR); } shutdown_attempts++; sleep(1); } }
> > MHD_stop_daemon(srv->mhd); srv->listening = false; return true; }
> > return false; }
> > Unfortunately if exit was called the MHD_stop_daemon() will be not
> > called, probably raising some memory leak, but current I have no idea
> > how it could be solved.
> > Feel totally free to point improvements about this new version. :-)
> > On Sun, Mar 26, 2017 at 5:36 PM, Evgeny Grin > > wrote:
> >
> > On 26.03.2017 8:33, silvioprog wrote: > I found the following
> > related message: > >
> > https://lists.gnu.org/archive/html/libmicrohttpd/2014-09/
> msg00012.html
> > 2014-09/msg00012.html>
> > > > I've used a similar logic, but with item X below, because I
> > need to wait > the client processing: > > 1) MHD_quiesce_daemon()
> > > *X) while (info.num_connections > 0) sleep(0.5s) # pseudo code*
> > > 2) MHD_stop_daemon() > 3) close() > > Real implementation: > >
> > bool bf_httpsrv_shutdown(struct bf_httpsrv *srv, bool force) { >
> >MHD_socket fd; > if (srv && srv->listening) { > fd
> > = MHD_quiesce_daemon(srv->mhd); > if (!force) >
> > while (MHD_get_daemon_info(srv->mhd,
> > MHD_DAEMON_INFO_CURRENT_CONNECTIONS)->num_connections > 0) >
> >usleep(1000 * 500); //TODO: allow to use external
> > callback > MHD_stop_daemon(srv->mhd); > if (fd !=
> > MHD_INVALID_SOCKET) > close(fd); >
> > srv->listening = false; > return true; > } >
> > return false; > } > > > Calling it with bf_httpsrv_shutdown(srv,
> > false) the server stops waiting > for clients processing. > > So,
> > what do you think about the logic above? Should it be improved?! >
> > > Thanks in advance for any suggestions! If you don't check
> > returned value from MHD_quiesce_daemon(), you may later found that
> > you didn't quiesced daemon, so move check right after
> > MHD_quiesce_daemon() and added error handling. If you didn't set
> > connection timeout, connection may live indefinitely. Moreover,
> > even with connection timeout, clients may continue processing on
> > same HTTP 1.1 connections with new requests indefinitely.
> > Furthermore, even with HTTP 1.0 and connection timeout
> > hypothetical client may read answer very slow with results in very
> > long unpredictable closure of connection. So: yes, you code will
> > work but may result in very long (hours, for example) or even
> > indefinitely long daemon shutdown. -- Best Wishes, Evgeny Grin
> >
> >
> > --
> > Silvio Clécio
>
--
Silvio Clécio
Re: [libmicrohttpd] MHD_quiesce_daemon() question
Done. The entire code is: https://pastebin.com/gNY2MwSY . I've used "curl
http://localhost:8080; echo" to test it.
Commenting the code: the locking happens at line 39 (MHD_stop_daemon(d)). I
don't know it the stop function offer some option to force the server
stopping, but I think we really don't need it. However, it would be nice to
have a "MHD_DAEMON_INFO_",
so the clients could check it in their processing, e.g:
...
static int dh(void *cls, struct MHD_Connection *con, const char *url,
const char *method, const char *version,
const char *upload_data, size_t *upload_data_size, void
**con_cls) {
const char *page = "I love MHD!";
struct MHD_Daemon *d;
struct MHD_Response *response;
int ret;
unsigned char i = 1;
bool stoped;
if (NULL != *con_cls) {
*con_cls = (void *) 1;
return MHD_YES;
}
/* simulating a long processing, like hard query in a database ... */
i = 1;
while (i < 100) {
MHD_get_connection_info(con, MHD_CONNECTION_INFO_DAEMON, &d);
if (d && MHD_get_daemon_info(d, *MHD_DAEMON_INFO_*))
break;
sleep(1);
i++;
}
response = MHD_create_response_from_buffer(strlen(page), (void *)
page, MHD_RESPMEM_PERSISTENT);
ret = MHD_queue_response(con, MHD_HTTP_OK, response);
MHD_destroy_response(response);
return ret;
}
...
It is just a idea, but I'm totally open to get new ideas. :-)
Anyway, if you see that *MHD_DAEMON_INFO_* could be an useful feature, I can try to send a patch
implementing it, something like *MHD_DAEMON_INFO_**WAS_QUIESCED*.
On Tue, Mar 28, 2017 at 12:51 PM, silvioprog wrote:
> Dude, thanks for fixing it, I'm going to study the changes and apply it
> here. :-)
>
> I've got a situation that locks the shutdown even using this fix, but I
> need to create a small project simulating the problem, I'm going to do it
> now ...
>
> On Tue, Mar 28, 2017 at 12:08 PM, Evgeny Grin wrote:
>
>> Fixed version of you code:
>>
>> bool bf_httpsrv_shutdown(struct bf_httpsrv *srv) { MHD_socket fd;
>> uint8_t shutdown_attempts; /* Max 256. */ if (srv && srv->listening) {
>> fd = MHD_quiesce_daemon(srv->mhd); if (fd != MHD_INVALID_SOCKET) {
>> srv->listening = false; close(fd); if (! srv->forced_shutdown) {
>> shutdown_attempts = 1; while (MHD_get_daemon_info(srv->mhd,
>> MHD_DAEMON_INFO_CURRENT_CONNECTIONS)->num_connections > 0) { if
>> (shutdown_attempts >= srv->shutdown_attempts) { /* Default
>> srv->shutdown_attempts is 10. */ _BF_LOG("Forced server shutdown.\n");
>> break; } shutdown_attempts++; sleep(1); } } } else _BF_LOG("Server
>> quiesce failed.\n"); MHD_stop_daemon(srv->mhd); return true; } return
>> false; }
>>
>> --
>> Best Wishes,
>> Evgeny Grin
>>
>> On 28.03.2017 4:46, silvioprog wrote:
>> > Thanks for replying Evgeny, you and Christian always saving my day. :-)
>> > I read and re-read your and Christian answer, and finally I found a
>> > possible way to never lock the server at shutdown: adding an option to
>> > exit the server after configurable attempts. It can solve the timeout
>> > problem too, because I can't ensure that the programmer have
>> > configured it (anyway my library will assume 15 seconds by default).
>> > So, the code below applies your tips and implements the attempts
>> approach:
>> > bool bf_httpsrv_shutdown(struct bf_httpsrv *srv) { MHD_socket fd;
>> > uint8_t shutdown_attempts; /* Max 256. */ if (srv && srv->listening) {
>> > fd = MHD_quiesce_daemon(srv->mhd); if (fd != MHD_INVALID_SOCKET)
>> > close(fd); else _BF_LOG("Server quiesce failed.\n"); if
>> > (srv->forced_shutdown) { shutdown_attempts = 1; while
>> > (MHD_get_daemon_info(srv->mhd,
>> > MHD_DAEMON_INFO_CURRENT_CONNECTIONS)->num_connections > 0) { if
>> > (shutdown_attempts >= srv->shutdown_attempts) { /* Default
>> > srv->shutdown_attempts is 10. */ _BF_LOG("Forced server shutdown.\n");
>> > exit(EINTR); } shutdown_attempts++; sleep(1); } }
>> > MHD_stop_daemon(srv->mhd); srv->listening = false; return true; }
>> > return false; }
>> > Unfortunately if exit was called the MHD_stop_daemon() will be not
>> > called, probably raising some memory leak, but current I have no idea
>> > how it could be solved.
>> > Feel totally free to point improvements about this new version. :-)
>> > On Sun, Mar 26, 2017 at 5:36 PM, Evgeny Grin > > > wrote:
>> >
>> > On 26.03.2017 8:33, silvioprog wrote: > I found the following
>> > related message: > >
>> > https://lists.gnu.org/archive/html/libmicrohttpd/2014-09/ms
>> g00012.html
>> > > sg00012.html>
>> > > > I've used a similar logic, but with item X below, because I
>> > need to wait > the client processing: > > 1) MHD_quiesce_daemon()
>> > > *X) while (info.num_connections > 0) sleep(0.5s) # pseudo code*
>> > > 2) MHD_stop_daemon() > 3) close() > > Real implementation: > >
>> > bool bf_httpsrv_shutdown(struct bf_httpsrv *
Re: [libmicrohttpd] MHD_quiesce_daemon() question
Oops, I meant:
static int dh(void *cls, struct MHD_Connection *con, const char *url,
const char *method, const char *version,
const char *upload_data, size_t *upload_data_size, void
**con_cls) {
const char *page = "I love MHD!";
struct MHD_Daemon *d;
struct MHD_Response *response;
int ret;
unsigned char i;
bool quiesced;
if (NULL != *con_cls) {
*con_cls = (void *) 1;
return MHD_YES;
}
/* simulating a long processing, like hard query in a database ... */
i = 1;
while (i < 100) {
MHD_get_connection_info(con, MHD_CONNECTION_INFO_DAEMON, &d);
if (d) {
MHD_get_daemon_info(d, *MHD_DAEMON_INFO_*, &quiesced);
if (quiesced)
break;
}
sleep(1);
i++;
}
response = MHD_create_response_from_buffer(strlen(page), (void *)
page, MHD_RESPMEM_PERSISTENT);
ret = MHD_queue_response(con, MHD_HTTP_OK, response);
MHD_destroy_response(response);
return ret;
}
On Tue, Mar 28, 2017 at 1:32 PM, silvioprog wrote:
> Done. The entire code is: https://pastebin.com/gNY2MwSY . I've used "curl
> http://localhost:8080; echo" to test it.
>
> Commenting the code: the locking happens at line 39 (MHD_stop_daemon(d)).
> I don't know it the stop function offer some option to force the server
> stopping, but I think we really don't need it. However, it would be nice to
> have a "MHD_DAEMON_INFO_",
> so the clients could check it in their processing, e.g:
>
> ...
>
> static int dh(void *cls, struct MHD_Connection *con, const char *url, const
> char *method, const char *version,
> const char *upload_data, size_t *upload_data_size, void
> **con_cls) {
> const char *page = "I love MHD!";
> struct MHD_Daemon *d;
> struct MHD_Response *response;
> int ret;
> unsigned char i = 1;
> bool stoped;
> if (NULL != *con_cls) {
> *con_cls = (void *) 1;
> return MHD_YES;
> }
> /* simulating a long processing, like hard query in a database ... */
> i = 1;
> while (i < 100) {
> MHD_get_connection_info(con, MHD_CONNECTION_INFO_DAEMON, &d);
> if (d && MHD_get_daemon_info(d, *MHD_DAEMON_INFO_ informing that daemon was quiesced>*))
> break;
> sleep(1);
> i++;
> }
> response = MHD_create_response_from_buffer(strlen(page), (void *) page,
> MHD_RESPMEM_PERSISTENT);
> ret = MHD_queue_response(con, MHD_HTTP_OK, response);
> MHD_destroy_response(response);
> return ret;
> }
>
> ...
>
> It is just a idea, but I'm totally open to get new ideas. :-)
>
> Anyway, if you see that *MHD_DAEMON_INFO_ daemon was quiesced>* could be an useful feature, I can try to send a
> patch implementing it, something like *MHD_DAEMON_INFO_**WAS_QUIESCED*.
>
> On Tue, Mar 28, 2017 at 12:51 PM, silvioprog wrote:
>
>> Dude, thanks for fixing it, I'm going to study the changes and apply it
>> here. :-)
>>
>> I've got a situation that locks the shutdown even using this fix, but I
>> need to create a small project simulating the problem, I'm going to do it
>> now ...
>>
>> On Tue, Mar 28, 2017 at 12:08 PM, Evgeny Grin wrote:
>>
>>> Fixed version of you code:
>>>
>>> bool bf_httpsrv_shutdown(struct bf_httpsrv *srv) { MHD_socket fd;
>>> uint8_t shutdown_attempts; /* Max 256. */ if (srv && srv->listening) {
>>> fd = MHD_quiesce_daemon(srv->mhd); if (fd != MHD_INVALID_SOCKET) {
>>> srv->listening = false; close(fd); if (! srv->forced_shutdown) {
>>> shutdown_attempts = 1; while (MHD_get_daemon_info(srv->mhd,
>>> MHD_DAEMON_INFO_CURRENT_CONNECTIONS)->num_connections > 0) { if
>>> (shutdown_attempts >= srv->shutdown_attempts) { /* Default
>>> srv->shutdown_attempts is 10. */ _BF_LOG("Forced server shutdown.\n");
>>> break; } shutdown_attempts++; sleep(1); } } } else _BF_LOG("Server
>>> quiesce failed.\n"); MHD_stop_daemon(srv->mhd); return true; } return
>>> false; }
>>>
>>> --
>>> Best Wishes,
>>> Evgeny Grin
>>>
>>> On 28.03.2017 4:46, silvioprog wrote:
>>> > Thanks for replying Evgeny, you and Christian always saving my day. :-)
>>> > I read and re-read your and Christian answer, and finally I found a
>>> > possible way to never lock the server at shutdown: adding an option to
>>> > exit the server after configurable attempts. It can solve the timeout
>>> > problem too, because I can't ensure that the programmer have
>>> > configured it (anyway my library will assume 15 seconds by default).
>>> > So, the code below applies your tips and implements the attempts
>>> approach:
>>> > bool bf_httpsrv_shutdown(struct bf_httpsrv *srv) { MHD_socket fd;
>>> > uint8_t shutdown_attempts; /* Max 256. */ if (srv && srv->listening) {
>>> > fd = MHD_quiesce_daemon(srv->mhd); if (fd != MHD_INVALID_SOCKET)
>>> > close(fd); else _BF_LOG("Server quiesce failed.\n"); if
>>> > (srv->forced_shutdown) { shutdown_attempts = 1; while
>>> > (MHD_get_daemon_info(srv->mhd,
>>> > MHD_DAEMON_INFO_CURRENT_CONNECTIONS)->num
Re: [libmicrohttpd] Reading data from named pipe
On 03/27/2017 08:03 PM, Alex Reynolds wrote: > I have an API that reads a subset of data from an archive and writes it to > a FILE*. I can use this API to write this data subset to a temporary file > via mkstemp() and fdopen(). > > I am currently using MHD_create_response_from_callback() to read chunks of > bytes from this temporary file and write them to the web client. This works > well on small files. > > To avoid the I/O cost of writing a larger temporary file, and then reading > from it, I set up a named pipe via mkfifo() and a FILE* that points to it. > When I have MHD_create_response_from_callback() try to read from this named > pipe, in the same way that it is set up to read from the temporary file, > the server and client hang. > > If this is possible, what is the correct way to serve data from a named > pipe? The answer depends on the style of event loop you are using. For external select, do a non-blocking (!) read from it, and return whatever you got to MHD via response_from_callback(), returning 0 if the pipe is still operational but you have no more data. To keep the event loop going, you need to manually add your pipe to the read-set, so that select/poll/epoll unblock as needed. For thread-per-connection, use a blocking (!) read from the pipe, and always return data to MHD. For thread-pool / internal select, you would have to suspend the connection when the pipe runs dry, and then have some _other_ thread check for the pipe becoming again available and calling resume on the connection. This one is usually messy. If you want to go experimental, you could try to get MHD_create_response_from_fd64() to work and use fileno() on your FILE* to get the underlying FD. I have _never_ tested this, and suspect it'll blow up in your face the moment the 'fd' blocks as MHD expects it to be a file and not a pipe. But, maybe it is possible to hack MHD to handle blocking 'sendfile()' --- I am pretty sure it does not manage this case today --- and thereby unify the 3 cases above, simplify the client and improve performance. Still, this is more like adding a medium-size feature to MHD than the "correct way" you asked for, so don't take this as advice for how to do it, but more as a pointer in case you ever need something even better and have too much time ;-). signature.asc Description: OpenPGP digital signature
Re: [libmicrohttpd] libmicrohttpd Digest, Vol 95, Issue 19
I need to compile a slightly older version of MHD, and manually adding -lrt at the end of my libraries string (i.e., LIB) in the build ended up fixing this, so I'd expect that to be the case with your patch as well (other changes notwithstanding). Thanks for the help! Regards, Alex > Date: Tue, 21 Mar 2017 13:53:49 +0100 > From: Christian Grothoff > To: libmicrohttpd@gnu.org > Subject: Re: [libmicrohttpd] '-lrt' not resolving undefined reference > to clock_gettime? > Message-ID: > Content-Type: text/plain; charset="utf-8" > glibc < 2.17 requires -lrt for clock_gettime(), older versions do not. > I've added "-lrt" in Git commit ed6509bf6ca46e39e3514680bfb81216e2a825bf > but without testing (as I don't have an ancient glibc on this system and > I am on a train...). Please let me know if it does not work... > Happy hacking! > Christian
