Hi - I manage a libmicrohttpd based public service, and the software that runs it (elfutils debuginfod & its fedora service). Under some usage patterns, it can get into a state where it starts rejecting new incoming connections with $SUBJECT stderr message. But it's a mystery why.
The server is throttled to a few dozen tasks via systemd limits. It uses MHD_USE_THREAD_PER_CONNECTION | MHD_USE_INTERNAL_POLLING_THREAD, and received front-end reverse-proxy services with keepalive etc. That part's fine most of the time. So we're servicing some dozen concurrent connections, no problem. But somehow, libmicrohttpd (0.9.73, fedora34) gets into a state where the daemon->connections number is weirdly bloated (1020), and so internal_add_connection reports $SUBJECT. But there are still only a few dozen keepalive sockets alive (and their threads), and matching number of entries on the daemon->connections list, and zero or one entry on the daemon->cleanup list. I wonder if the thread throttling (via linux cgroups) or other transient problem can let that number bloat. In internal_add_connection(), ->connections++ is done, but in case of later error, the cleanup: path does not propagate the halfbroken connection object to MHD_connection_cleanup_() for a subsequent ->connections--. So that ->connections number could grow and grow, and eventually saturate to the ->global_connection_limit. Do I read the code right? Other than a fix to this part of libmicrohttpd, do I have an option other than using MHD_OPTION_THREAD_POOL_SIZE instead of MHD_USE_THREAD_PER_CONNECTION? - FChE
