Yes I am using 'select' in event loop. After my tests, your diagnosis and solutions are definitely right. I fixed the issue by changing select to epoll. Thank you very much! And thank you all for the hard work of this great project:)
Best regards, Yatong Zhang On Sat, Oct 21, 2017 at 4:45 PM, Christian Grothoff <[email protected]> wrote: > Hi, > > Might it be that you are using 'select()' in your event loop? (The rest > of the answer does not apply if you do not, but it is almost certain > that this is your problem). There, the FD_SIZE limit applies (not just > the ulimit), so once you hit near 1000 FDs open from mmap, the server > won't be able to accept() incoming connections and pass them to select() > anymore. As MHD accepted(), then got a "bad" (high) FD, its only choice > is to close() the FD, which leads exactly to the behavior your describe. > > > In this case, you have three possible solutions: > 1) change to poll or epoll, also improving performance and scalabiltiy > 2) Increase your ulimit (as you did presumably in point 4) and then > use 'dup2' for the files you open for mmap() to move the FD into > the range *above* 1024. Make sure to close() the file FDs below 1024 > afterwards. That way, MHD can keep using the low FDs for the TCP > sockets. > 3) Depending on your situation, you might also be able to simply > open the file, call mmap(), then *close* the file but not call > munmap() until the response is done. That way, you don't burn an > FD at all and don't need to do either (1) or (2). > > > Note that once you have more than 1000 concurrent network connections, > only (1) will really fix your problems. > > > Happy hacking! > > Christian > > On 10/21/2017 10:38 AM, Yatong Zhang wrote: > > Hi Christian, > > I built a http server with libmicrohttpd and it serves as a file content > > server. My http server opens lots of files using 'mmap' and the files > > increase daily. At first the server works fine but recently when the > > files opened about more than 1000, it begins to reply nothing. curl > > indicates that 'Server returned nothing (no headers, no data)' or > > 'Failure when receiving data from the peer‘. So, > > 1. When the files opened are more then 1000, server sometimes close > > connections immediately. It's 'sometimes', not 'always' > > 2. The files are opened with mmap. > > 3. If the files are reduced to about 950, server works fine. > > 4. I have modified system limits, such as number opened files, but don't > > help. > > Here the limits of the server process: > > > > cat /proc/10758/limits > > Limit Soft Limit Hard Limit > > Units > > Max cpu time unlimited unlimited > > seconds > > Max file size unlimited unlimited > > bytes > > Max data size unlimited unlimited > > bytes > > Max stack size 8388608 unlimited > > bytes > > Max core file size 0 0 > > bytes > > Max resident set unlimited unlimited > > bytes > > Max processes 514482 514482 > > processes > > Max open files 65535 65535 > > files > > Max locked memory 65536 65536 > > bytes > > Max address space unlimited unlimited > > bytes > > Max file locks unlimited unlimited > > locks > > Max pending signals 514482 514482 > > signals > > Max msgqueue size 819200 819200 > > bytes > > Max nice priority 0 0 > > Max realtime priority 0 0 > > Max realtime timeout unlimited unlimited > us > > > > > > I have tuned all the parameters I can find including 'sysctl' stuff, > > nothing helps. I know this must have something to do with the number of > > opened files but I don't know why and how to fix it. So would you please > > help and give me some advices? > > Thanks very much. > > > > Best regards, > > > > Yatong Zhang > >
