[libmicrohttpd] race condition on 0.9.73
Hi, My program sends answers in threads and runs the main MHD loop on external event in an other thread and I get crashes. I'm suspecting LMHD because valgrind point out that the memory for response is used after free in a kind of race condition between internals of MHD. Is it a known issue fixed since 0.9.73? I can either check why (if the program is correct to use multiple threads) or protect defensively (by removing implied concurrence externally -sub optimal probably-). Though? Best regard José
Re: [libmicrohttpd] race condition on 0.9.73
Hi José, For sure you can run MHD with "external poll" mode, but you must ensure that only single thread is calling MHD_run() at any given moment of time. If you want to call several copies of MHD_run(), make sure that you have individual copies of MHD for each thread (start MHD by MHD_start_daemon() for every thread). Alternatively, you can use MHD with its own threading mode. MHD can run treads per connection or use thread pool. Can you give more details how you use MHD? Which flags are used? Why do you use external threads instead of MHD internal treads? -- Wishes, Evgeny On 23.09.2021 13:17, José Bollo wrote: Hi, My program sends answers in threads and runs the main MHD loop on external event in an other thread and I get crashes. I'm suspecting LMHD because valgrind point out that the memory for response is used after free in a kind of race condition between internals of MHD. Is it a known issue fixed since 0.9.73? I can either check why (if the program is correct to use multiple threads) or protect defensively (by removing implied concurrence externally -sub optimal probably-). Though? Best regard José OpenPGP_signature Description: OpenPGP digital signature
Re: [libmicrohttpd] race condition on 0.9.73
On Thu, 23 Sep 2021 18:14:24 +0300 Evgeny Grin wrote: > Hi José, Hi Evgeny, > For sure you can run MHD with "external poll" mode, but you must > ensure that only single thread is calling MHD_run() at any given > moment of time. Okay, that is a good hint, thanks. I'm going to first track concurrent use of MHD_run. FYI, I tried the latest git and got the same issue. > If you want to call several copies of MHD_run(), make sure that you > have individual copies of MHD for each thread (start MHD by > MHD_start_daemon() for every thread). I don't want that, because it implies differents port I suppose. > Alternatively, you can use MHD with its own threading mode. MHD can > run treads per connection or use thread pool. > > Can you give more details how you use MHD? Which flags are used? flags = MHD_USE_EPOLL | MHD_ALLOW_UPGRADE | MHD_USE_TCP_FASTOPEN | MHD_USE_NO_LISTEN_SOCKET | MHD_ALLOW_SUSPEND_RESUME; > Why do you use external threads instead of MHD internal treads? The program has a job scheduler that manages threads on its own. It also serves other protocols that HTTP(S). best regards josé
Re: [libmicrohttpd] race condition on 0.9.73
> ... because valgrind point out ... You should replace (slow) valgrind with (fast) address-sanatizer. Simply tell GCC (or clang) to compile with: -fsanitize=address -fno- omit-frame-pointer
Re: [libmicrohttpd] race condition on 0.9.73
On 23.09.2021 18:43, José Bollo wrote: For sure you can run MHD with "external poll" mode, but you must ensure that only single thread is calling MHD_run() at any given moment of time. Okay, that is a good hint, thanks. I'm going to first track concurrent use of MHD_run. FYI, I tried the latest git and got the same issue. If you want to call several copies of MHD_run(), make sure that you have individual copies of MHD for each thread (start MHD by MHD_start_daemon() for every thread). I don't want that, because it implies differents port I suppose. Even if MHD is doing internal accept(), you still can use MHD_OPTION_LISTENING_ADDRESS_REUSE parameter to bind to the same port. Or provide your own socket. However as I can see below, it's not applicable for your case as you are not using MHD to listen for new connections. Alternatively, you can use MHD with its own threading mode. MHD can run treads per connection or use thread pool. Can you give more details how you use MHD? Which flags are used? flags = MHD_USE_EPOLL | MHD_ALLOW_UPGRADE | MHD_USE_TCP_FASTOPEN | MHD_USE_NO_LISTEN_SOCKET | MHD_ALLOW_SUSPEND_RESUME; Do you use suspend/resume of connections? Why do you use external threads instead of MHD internal treads? The program has a job scheduler that manages threads on its own. It also serves other protocols that HTTP(S). It is still hard to see where is the problem. Actually responses are thread-safe and shouldn't create any such kind of problems. Could you share you access handler callback code? Where do you generate responses? Where do you queue responses? -- Evgeny OpenPGP_signature Description: OpenPGP digital signature
Re: [libmicrohttpd] race condition on 0.9.73
On Thu, 23 Sep 2021 19:57:23 +0300 Evgeny Grin wrote: > On 23.09.2021 18:43, José Bollo wrote: > >> For sure you can run MHD with "external poll" mode, but you must > >> ensure that only single thread is calling MHD_run() at any given > >> moment of time. > > > > Okay, that is a good hint, thanks. I'm going to first track > > concurrent use of MHD_run. FYI, I tried the latest git and got the > > same issue. > >> If you want to call several copies of MHD_run(), make sure that you > >> have individual copies of MHD for each thread (start MHD by > >> MHD_start_daemon() for every thread). > > > > I don't want that, because it implies differents port I suppose. > > Even if MHD is doing internal accept(), you still can use > MHD_OPTION_LISTENING_ADDRESS_REUSE parameter to bind to the same > port. Or provide your own socket. > However as I can see below, it's not applicable for your case as you > are not using MHD to listen for new connections. > > > >> Alternatively, you can use MHD with its own threading mode. MHD can > >> run treads per connection or use thread pool. > >> > >> Can you give more details how you use MHD? Which flags are used? > > > > flags = MHD_USE_EPOLL > > | MHD_ALLOW_UPGRADE > > | MHD_USE_TCP_FASTOPEN > > | MHD_USE_NO_LISTEN_SOCKET > > | MHD_ALLOW_SUSPEND_RESUME; > > Do you use suspend/resume of connections? yes for websocket upgrade but it is not relevant in the case. > > > >> Why do you use external threads instead of MHD internal treads? > > > > The program has a job scheduler that manages threads on its own. It > > also serves other protocols that HTTP(S). > > It is still hard to see where is the problem. Actually responses are > thread-safe and shouldn't create any such kind of problems. > > Could you share you access handler callback code? > Where do you generate responses? > Where do you queue responses? I think that the clue about re-entering MHD_run is the good one. I'm going to follow that path tomorrow and will let you know. The code is public but you know, it is a pain to read someone else code and this part is poorly commented. Best regards José
