This is happening with the latest version (0.9.73) and on the master branch.
erik
On Thu, Dec 9, 2021 at 11:06 PM Evgeny Grin <k...@yandex.ru> wrote:
> Hi Erik,
>
> It's hard to move forward without knowing exact MHD version used.
>
> Please share information about your MHD version.
>
> I suspect that your "Reply-To" header may confuse mailing list system.
> Do not use "Reply-To" headers with mailing list.
>
> --
> Evgeny
>
> -------- Original Message --------
> From: Erik Smith <cruiserco...@gmail.com>
> Sent: Friday, December 10, 2021, 03:29 UTC+3
> Subject: [libmicrohttpd] epoll and memory leaks
>
> > I've been able to reproduce this with a modified program from the
> > examples to show the memory consumption I'm seeing. I'm using jemalloc
> > to capture memory consumption and here's what it looks like for the
> > program below when repeatedly hitting the endpoint:
> >
> > allocated: 120072, active: 163840, resident: 9150464
> > allocated: 150536, active: 196608, resident: 9228288
> > allocated: 181000, active: 229376, resident: 9306112
> > allocated: 211464, active: 262144, resident: 9383936
> > allocated: 211464, active: 262144, resident: 9383936
> > allocated: 241928, active: 294912, resident: 9461760
> > allocated: 272392, active: 327680, resident: 9539584
> > allocated: 272392, active: 327680, resident: 9539584
> > allocated: 302856, active: 360448, resident: 9617408
> >
> > The delay in the handler and the use of ASAN tend to inflate the memory
> > growth. The key factor here seems to be the use of the thread poll
> > with either poll or epoll. Without the thread pool, there is no memory
> > growth at all. The growth happens on low connection rates (manual
> > refreshing in the browser). I haven't yet tried compiling MHD with
> ASAN.
> >
> > I'm also not getting responses to my threads in email for some reason
> > but I'm checking the archive.
> >
> > #include <cstring>
> > #include <iostream>
> > #include <jemalloc/jemalloc.h>
> > #include <microhttpd.h>
> > #include <sstream>
> > #include <thread>
> >
> > static enum MHD_Result handler(void *, struct MHD_Connection *connection,
> > const char *url, const char *method,
> > const char *, const char *, size_t *,
> > void **ptr) {
> > static int aptr;
> >
> > if (&aptr != *ptr) {
> > *ptr = &aptr;
> > return MHD_YES;
> > }
> > *ptr = NULL;
> >
> > std::this_thread::sleep_for(std::chrono::milliseconds(40));
> >
> > size_t sz = sizeof(size_t);
> > uint64_t epoch = 1;
> > mallctl("thread.tcache.flush", NULL, NULL, NULL, 0);
> > mallctl("epoch", &epoch, &sz, &epoch, sz);
> >
> > std::size_t allocated, active, metadata, resident, mapped;
> > mallctl("stats.allocated", &allocated, &sz, NULL, 0);
> > mallctl("stats.active", &active, &sz, NULL, 0);
> > mallctl("stats.resident", &resident, &sz, NULL, 0);
> >
> > std::stringstream s;
> > s << "allocated: " << allocated << ", active: " << active
> > << ", resident: " << resident << "\n";
> > auto msg = s.str();
> >
> > std::cout << msg;
> >
> > struct MHD_Response *response = MHD_create_response_from_buffer(
> > msg.size(), msg.data(), MHD_RESPMEM_MUST_COPY);
> > MHD_Result ret = MHD_queue_response(connection, MHD_HTTP_OK,
> response);
> > MHD_destroy_response(response);
> > return ret;
> > }
> >
> > int main(int argc, char *argv[]) {
> > struct MHD_Daemon *d;
> >
> > int port = argc > 1 ? atoi(argv[1]) : 10000;
> >
> > // epoll mode with thread pool
> > unsigned int concurrency = std::thread::hardware_concurrency();
> > std::cout << "concurrency: " << concurrency << "\n";
> >
> > d = MHD_start_daemon(MHD_USE_EPOLL_INTERNAL_THREAD |
> > MHD_USE_ERROR_LOG, port,
> > NULL, NULL, handler, NULL,
> > MHD_OPTION_CONNECTION_TIMEOUT,
> > (unsigned int)120, MHD_OPTION_STRICT_FOR_CLIENT,
> > (int)1,
> > MHD_OPTION_THREAD_POOL_SIZE, concurrency, NULL,
> > MHD_OPTION_END);
> >
> > if (d == NULL)
> > return 1;
> > std::cout << "listening on port: " << port << "\n";
> > std::cout << "hit key to stop"
> > << "\n";
> >
> > // type a key to end
> > (void)getc(stdin);
> > MHD_stop_daemon(d);
> > return 0;
> > }
> >
> >
> > Hi Erik,
> > Which MHD version are you using?
> > Some problems with externally added connections with epoll mode were
> > fixed in v0.9.72.
> > If you have any blocking calls, make sure that you use connection
> > suspend/resume. Alternatively, you can you use thread-per-connection
> > mode, this is less efficient, but simpler to implement.
> > epoll mode does not have special memory allocation, connections are
> > processed in the same way, like in other modes. MHD typically does
> > not allocate memory during connection processing, except when new
> > connection is started.
> > Do you use postprosessor or authentication functions? MHD has some
> > memory allocs in these functions.
> > The issue is not connected with quoted comment definitely. It is
> > just a bad wording. Actually nothing is leaked, but may be locked
> > until end of sending of response. Moreover, MHD does not use memory
> > pool in the way where such lock is possible. Memory pool is reset
> > after each request-reply cycle. Memory pool size for each connection
> > is fixed and cannot grow.
> > A few suggestions:
> > * make sure that you are using the latest MHD version (0.9.73 at the
> > moment), * check whether you destroy responses and free all
> > resources connected to responses, * if you are testing your code
> > with ASAN, make sure that leak detector is enabled. You can build
> > static MHD lib with ASAN and link it with our application compiled
> > with ASAN,
> > * use Valgrind or simpler tools like memstat or memprof.
> > --
> > Wishes,
> > Evgeny
> > On 22.11.2021 22:56, Erik Smith wrote:
> > /* Reallocate a block of memory obtained from the pool.
> > * This is particularly efficient when growing or
> > * shrinking the block that was last (re)allocated.
> > * If the given block is not the most recently
> > * (re)allocated block, the memory of the previous
> > * allocation may be leaked until the pool is
> > * destroyed or reset. */
> > Can anyone confirm whether this might be related?
> > ASAN does not seem to detect any issues in our code presently (not
> > sure about MHD)
> >
> > We have started to experiment with running MHD with epoll + thread
> > pool as we do the FD limit in certain situations. We understand
> > that there are caveats to this given that we have some
> > blocking database calls. This seems to get us past the FD limit
> > errors and the performance is similar. However, we are running
> > into growing memory consumption in our server over time
> > running epoll+threads that require a restart frequently. This does
> > not seem to occur with just epoll (without the thread pool). We
> > are running jemalloc, but it does not seem to be related to the leak
> > when it is disabled. There is the following comment in the MHD code
> > for the MHD_pool_reallocate function that might be connected to this
> > issue:
> >
>
