I use uhttpd with the embedded lua engine and uploading files with
standard POST / multipart/form-data / type=file has never been speedy.
However, I've reached a point that the file upload completes but
uhttpd times out (in 60 seconds) before it can transfer the file to
the lua script that is using the luci.http.setfilehandler() method. To
be clear, uhttpd times out before the first call to the filehander
function-- it times out in the CGI boundary between the server and
sgi/uhttp.lua.
I tracked this down to uhttpd/lua.c:uh_lua_recv(). A single
uhttpd.read(4096) from lua reads *the entirety of stdin* up until it
is exhausted instead of the requested size. Note that the underlying
reads are LUAL_BUFFERSIZE in size, which is 1024 bytes. With a large
file upload (or any upload really), the entire content will be
contained in a single lua string which is contatenated together 1KB at
a time, and contatentated together in 10x gathered chunks off the lua
stack in the luaL_buffer implementation.
Example, bcm2708 target, 8876KB upload:
-- 18 seconds between browser data transfer complete and CGI boundary
crossing into user lua code
-- 89MB RAM usage in uhttpd process as the giant string is built
Example, bcm2708 target, 28992KB upload:
-- 68 seconds between browser data transfer complete and CGI boundary
crossing into user lua code
-- 320MB RAM usage in uhttpd process as the giant string is built
The solution is pretty simple, to only return as much data as is
requested by the lua method uhttpd.read():
--- uhttpd-2017-08-19-3fd58e9b/lua.c.orig 2018-01-24
10:24:34.288327793 -0500
+++ uhttpd-2017-08-19-3fd58e9b/lua.c 2018-01-24 10:28:31.817731660 -0500
@@ -72,6 +72,7 @@
luaL_addsize(&B, r);
data_len += r;
+ len -= r;
if (r != LUAL_BUFFERSIZE)
break;
}
For my 8876KB example, this reduces the execution time from 18 seconds
to 1 second and the RAM usage from 89MB to 3.7MB. The 28992KB example
drops from 68 seconds to 4 seconds, and 320MB RAM to 3.7MB as well.
The size of the process working set is really just the size of the
running lua interpreter reusing the same 4K over and over again,
instead of trying to hold an entire 29MB file in a string,
concatenating it over and over. The sha256sum and md5sum of the
uploaded files are identical.
_______________________________________________
Lede-dev mailing list
Lede-dev@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/lede-dev
