Re: [libmicrohttpd] Server reached connection limit. Closing inbound connection. how to increase the limits?

[Evgeny Grin]

In addition, consider using MHD_OPTION_CONNECTION_TIMEOUT so old connection will be dropped automatically. -- Best Wishes,Evgeny Grin   20.01.2018, 20:53, "Christian Grothoff" :Hi Pascal,There are several "limits" that may apply.(1) MHD_OPTION_CONNECTION_LIMIT -- usually 1020 (or FD_SETSIZE - 4) islikely the first one you may hit. You can pass a different value toMHD_start_daemon() in the varargs;(2) FD_SETSIZE is another limit you are likely to hit if you use theselect() event loop; tricky to bump, you ought to switch to poll/epollinstead.(3) ulimit (-n) may also impose a 1024 limit on your processHappy hacking!ChristianOn 01/20/2018 08:56 AM, Menaxerius wrote: Hello, I'm using microlibhttpd for our Miningpool Server application. So far so good. But due to more and more active miners is reaching I start reaching this limit. Can someone explain me where those limit is set and how to increase it?  Thanks in advance. Regards Pascal 

Re: [libmicrohttpd] Unable to connect to libmicrohttpd in embedded system.

[Evgeny Grin]

Hi Marcel,

Why do you think that you have some race condition? You are using single
thread, unlikely it could produce any internal race condition.

With recent versions of MHD, I suggest to use MHD_USE_AUTO.

In your case it could be MHD_USE_AUTO | MHD_USE_INTERNAL_POLLING_THREAD
| MHD_USE_DEBUG combination.

'daemon->connections_tail' should be NULL while MHD do not serve any
connection (only listening).

I suggest you to try to debug 'MHD_accept_connection ()' - this function
should accept new connection.

Alternatively, you could try to run any of provided examples to ensure
that MHD function properly.

-- 

Best Wishes,
Evgeny Grin

22.01.2018 1:09, Marcel Rutten пишет:
> Thanks, Silvio, it would be great if you could send me that working
> code. Probably saves a lot of time :-). My environment is an
> openembedded tree, as published by the manufacturer of the thermostat,
> back in 2012. Parts of it are really old, so sometimes I have to do
> quite some patching to backport newer code to that tree and toolchain.
> Nevertheless, your working code is probably fairly easy to build.
> libmicrohttpd wasn't so hard in terms of passing it through this
> toolchain. Getting it to work is a different story :-).
>
> In the meantime, I ploughed through a lot of routines, working my way
> up from the routines that have daemon->default_handler (which is
> called from ozwcp) as argument.
> I have tested the following options for the server:
>
> MHD_USE_EPOLL | MHD_USE_INTERNAL_POLLING_THREAD | MHD_USE_DEBUG
>
> This ends up in a race condition in MHD_epoll(), as called in
> MHD_polling_thread().
>
> MHD_USE_POLL | MHD_USE_INTERNAL_POLLING_THREAD | MHD_USE_DEBUG
>
> gives a race condition in MHD_poll()
>
> and finally:
> |
> MHD_USE_INTERNAL_POLLING_THREAD | MHD_USE_DEBUG
> This gives a race condition in MHD_select(). The race conditions at
> least explain why the client keeps trying, while nothing gets actually
> loaded. Only after killing the daemon, the connection is reset. I did
> some follow-up on the race condition in MHD_select(), and found that
> the connections in the daemon struct are not filled:
> (excerpt from internal_run_from_select() ):
> ... (FD_ISSET (ds, read_fd_set)) ) (void) MHD_accept_connection
> (daemon); if (0 == (daemon->options & MHD_USE_THREAD_PER_CONNECTION))
> { /* do not have a thread per connection, process all connections now
> */ prev = daemon->connections_tail; < prev = 0x00 while (NULL !=
> (pos = prev)) < pos is also 0x00, so the loop never starts,
> resulting in a new call to MHD_select(), and so on and on .. { prev =
> pos->prev; ds = pos->socket_fd; ...
> Both current and previous connections are NULL pointers, which is not
> good :-(, but it already narrows the issue down to a much smaller part
> of the daemon. So the next question is: where are these connections
> filled, and how do I check if it's done properly?
> |
>
> On 21 January 2018 at 22:28, silvioprog  > wrote:
>
> Hello Marcel, welcome to the list!
>
> I used MHD on ARM in a small HTTP server testing for
> Android: 
> https://lists.gnu.org/archive/html/libmicrohttpd/2017-02/msg00014.html
> 
> . This link contains an attached picture showing the results. I
> think I have the sources in my old machine.
>
> I don't know if my environment can be useful for you, but I built
> MHD for ARM via integrated Android Studio's CMake, and used MHD in
> Java project with JNI. I can search and send sources if they could
> be useful for you ...
>
> On Sun, Jan 21, 2018 at 7:48 AM, Marcel Rutten
> mailto:mcmrut...@gmail.com>> wrote:
>
> Hi, I'm new to this list, and I have been trying to get
> libmicrohttpd to work on a linux-based smart thermostat.
>
> To no avail, so far. I should say that I'm not an expert on
> web connections, but fairly fluent in C.
>
> The thermostat has a Freescale imx 27 processor, (ARM-926 EJS)
> and has an openembedded image, developed around 2011, with
> linux kernel 2.6.26. The reason I want to get libmicrohttpd
> working on it, is because it features a zwave controller,
> which may be used to control TRVs (Thermostatic Radiator
> Valves), a feature which is not available in the standard
> firmware of the thermostat.  The zwave interface can be
> operated through openzwave, and its web interface ozwcp (open
> zwave control panel). ozwcp is linked against libmicrohttpd to
> provide the web interface.
>
> I have tested the combination libopenzwave-1.4.164,
> libmicrohttpd-0.9.58 and ozwcp (latest version from github) on
> a CentOS7 machine, x86_64, with a USB zwave controller and it
> works flawlessly. External access to the webinterface is OK.
>
> I built the same libraries and code for the thermostat, a

Re: [libmicrohttpd] Unable to connect to libmicrohttpd in embedded system.

[Marcel Rutten]

Thanks, Evgeny,

My statement was a bit off. The race condition is not inside MHD_epoll()
and its cousins, but in the loop surrounding them.  So I get an incessant
series of calls to e.g., MHD_epoll() at a very high rate, without anything
actually happening. I have landed inside MHD_accept_connection(), and as
you suggest, the problem probably resides somewhere in that routine.
For some reason, the examples are not built on my system, although I
haven't switched them off through configure. Could be the bitbake build
environment, but I'm not sure.
For now, I'm not too worried about that.

I just tried the MHD_USE_AUTO etc., it makes no difference.
Will keep on digging in MHD_accept_connection(). It returns 0 on its very
first call. Not good. I have a feeling that I'm getting closer.

Marcel

Re: [libmicrohttpd] Unable to connect to libmicrohttpd in embedded system.

[Marcel Rutten]

Found it. accept4() returns an invalid socket.
I will carry on tomorrow, it's almost midnight in my time zone ...

On 22 January 2018 at 23:13, Marcel Rutten  wrote:

> Thanks, Evgeny,
>
> My statement was a bit off. The race condition is not inside MHD_epoll()
> and its cousins, but in the loop surrounding them.  So I get an incessant
> series of calls to e.g., MHD_epoll() at a very high rate, without anything
> actually happening. I have landed inside MHD_accept_connection(), and as
> you suggest, the problem probably resides somewhere in that routine.
> For some reason, the examples are not built on my system, although I
> haven't switched them off through configure. Could be the bitbake build
> environment, but I'm not sure.
> For now, I'm not too worried about that.
>
> I just tried the MHD_USE_AUTO etc., it makes no difference.
> Will keep on digging in MHD_accept_connection(). It returns 0 on its very
> first call. Not good. I have a feeling that I'm getting closer.
>
> Marcel
>

Re: [libmicrohttpd] Unable to connect to libmicrohttpd in embedded system.

[Christian Grothoff]

-1 can be *normal*, as MHD calls 'accept()' (possibly via multiple
threads when using a thread pool) until there are no more clients.  You
should check what errno says.

Also, in case accept4() is buggy on your platform, it is an
optimization; so you could #undef HAVE_ACCEPT4 if accept() +
setsockopt() works better. But I've not yet heard about a buggy
accept4() implementation...

Happy hacking!

Christian

On 01/22/2018 11:32 PM, Marcel Rutten wrote:
> Found it. accept4() returns an invalid socket.
> I will carry on tomorrow, it's almost midnight in my time zone ...



signature.asc
Description: OpenPGP digital signature

Re: [libmicrohttpd] Unable to connect to libmicrohttpd in embedded system.

[silvioprog]

Hello Marcel, sorry for the late answer.

The primary example was deleted from my old machine, but I downloaded the
new Android Studio version and created a new example using MHD 0.9.58 as
static library. This new example is very, very, very simple, no
sophisticated implementation, just an app with two buttons "Start/Stop" to
test the server running on Android in a Java application with MHD via JNI:

https://img42.com/UmKpx

I tried to auto-download MHD's tar.gz with CMake's ExternalProject_Add()
 feature, but I've
got a problem with Gradle and I left to solve it for a while. :-/

So, to test this rustic example, download it (it is a little large because
I kept entire project with generated APKs) from my Google Drive:

https://drive.google.com/open?id=1fGy3hTSjaRjykrcBHHXvY3ha-YUU4jEV

then, open it on Android Studio and run the application (Shift+F10), the
studio will ask you to run it in a smartphone or in builtin android
emulator.

The commands I've used to build MHD for ARM was:

./configure --host=arm-linux-androideabi --enable-shared=no
--enable-static=yes --enable-https=no --disable-curl --disable-doc
--disable-examples
make

(feel free to customize it)

Notice a line in the CMakeLists.txt file:

target_link_libraries( # Specifies the target library.
   native-lib

   # Links the target library to the log library
   # included in the NDK.
   ${log-lib}

*"/home/silvioprog/AndroidStudioProjects/MHDServer/app/src/main/cpp/libmicrohttpd-0.9.58/src/microhttpd/.libs/libmicrohttpd.a"*
...

it was intentional to show the directory with the generated static library,
please change it on your machine.

I'm not sure if this example will help you, but I'll keep it available a
week in my drive. :-)

On Sun, Jan 21, 2018 at 7:09 PM, Marcel Rutten  wrote:

> Thanks, Silvio, it would be great if you could send me that working code.
> Probably saves a lot of time :-). My environment is an openembedded tree,
> as published by the manufacturer of the thermostat, back in 2012. Parts of
> it are really old, so sometimes I have to do quite some patching to
> backport newer code to that tree and toolchain. Nevertheless, your working
> code is probably fairly easy to build. libmicrohttpd wasn't so hard in
> terms of passing it through this toolchain. Getting it to work is a
> different story :-).
>
> In the meantime, I ploughed through a lot of routines, working my way up
> from the routines that have daemon->default_handler (which is called from
> ozwcp) as argument.
> I have tested the following options for the server:
>
> MHD_USE_EPOLL | MHD_USE_INTERNAL_POLLING_THREAD | MHD_USE_DEBUG
>
> This ends up in a race condition in MHD_epoll(), as called in
> MHD_polling_thread().
>
> MHD_USE_POLL | MHD_USE_INTERNAL_POLLING_THREAD | MHD_USE_DEBUG
>
> gives a race condition in MHD_poll()
>
> and finally:
>
> MHD_USE_INTERNAL_POLLING_THREAD | MHD_USE_DEBUG
>
> This gives a race condition in MHD_select().
> The race conditions at least explain why the client keeps trying, while 
> nothing gets actually loaded. Only after killing the daemon, the connection 
> is reset.
> I did some follow-up on the race condition in MHD_select(), and found that 
> the connections in the daemon struct are not filled:
>
> (excerpt from internal_run_from_select() ):
>
> ...
>(FD_ISSET (ds,
>   read_fd_set)) )
> (void) MHD_accept_connection (daemon);
>
>   if (0 == (daemon->options & MHD_USE_THREAD_PER_CONNECTION))
> {
>   /* do not have a thread per connection, process all connections now */
>   prev = daemon->connections_tail; < 
> prev = 0x00
>   while (NULL != (pos = prev)) < 
> pos is also 0x00, so the loop never starts, resulting in a new call to 
> MHD_select(), and so on and on ..
> {
> prev = pos->prev;
>   ds = pos->socket_fd;
> ...
>
> Both current and previous connections are NULL pointers, which is not good 
> :-(, but it already narrows the issue down to a much smaller part of the 
> daemon. So the next question is: where are these connections filled, and how 
> do I check if it's done properly?
>
>
> On 21 January 2018 at 22:28, silvioprog  wrote:
>
>> Hello Marcel, welcome to the list!
>>
>> I used MHD on ARM in a small HTTP server testing for Android:
>> https://lists.gnu.org/archive/html/libmicrohttpd/2017-02/msg00014.html .
>> This link contains an attached picture showing the results. I think I have
>> the sources in my old machine.
>>
>> I don't know if my environment can be useful for you, but I built MHD for
>> ARM via integrated Android Studio's CMake, and used MHD in Java project
>> with JNI. I can search and send sources if they could be useful for you ...
>>
>> On Sun, Jan 21, 2018 at 7:48 AM, Marcel Rutten 
>> wrote:
>>
>>> Hi, I'm new to this list, a