It takes me less time using fopen() to download big files, but, after
reading your answer, I'm not sure if it is really related to fopen vs fd or
to my test environment. Anyway, I did a small patch in MHD that performs
better results using fd. If you agree with that, I finally could upgrade my
library to use MHD_response_from_fd-style functions and abandon fopen
definitely. Firstly, consider the following example (errors handling was
omitted to make it clear):
#define FILENAME "Win7.iso" // *about 4.9 GB*
#define USE_FOPEN
static ssize_t fileread_cb(void *handle, uint64_t offset, char *buf, size_t
size) {return fread(buf, 1, size, handle);}
static void filefree_cb(void *handle) {fclose(handle);}
static int ahc_echo(void *cls, struct MHD_Connection *con, const char *url,
const char *verb, const char *ver,
const char *ud, size_t *uds, void **p) {
struct MHD_Response *res;
int ret, fd;
struct stat64 buf;
FILE *file;
if (!*p) {
*p = (void *) 1;
return MHD_YES;
}
*p = NULL;
#if defined(USE_FOPEN)
file = fopen64(FILENAME, "rb");
fd = fileno(file);
fstat64(fd, &buf);
res = MHD_create_response_from_callback((uint64_t) buf.st_size, 16 *
1024, fileread_cb, file, filefree_cb);
#else
fd = open64(FILENAME, O_RDONLY);
fstat64(fd, &buf);
res = MHD_create_response_from_fd64((uint64_t) buf.st_size, fd);
#endif
ret = MHD_queue_response(con, MHD_HTTP_OK, res);
MHD_destroy_response(res);
return ret;
}
using the example above, the fopen saves 8 ~ 11 seconds (the total average
time download is ~35 seconds). However, if you agree to add a new define in
mhd_limits.h:
...
#ifndef MHD_FD_BLOCK_SIZE
#define MHD_FD_BLOCK_SIZE 4 * 1024
#endif
...
so MHD_create_response_from_callback() could use it:
...
response = MHD_create_response_from_callback (size,
MHD_FD_BLOCK_SIZE,
&file_reader,
NULL,
&free_callback);
...
and we could change MHD_FD_BLOCK_SIZE at build time to use a custom value
in such environments.
What do you think?
On Fri, Nov 30, 2018 at 7:18 PM Christian Grothoff <groth...@gnunet.org>
wrote:
> silvioprog: I don't understand why you'd use fopen() here instead of
> just using the MHD_response_from_fd-style response generation. On
> platforms where sendfile() is unavailable, MHD falls back to the usual
> reading of the file into a buffer, so I don't quite see how you
> re-implementing that logic helps. Sure, the fopen()-API might do some
> additional buffering, but if you give MHD enough memory per connection,
> you should get exactly the same effect. So this just seems
> unnecessarily convoluted (but I didn't try to understand the motivation
> behind every branch in that code).
>
> Happy hacking!
>
> Christian
>
> On 11/30/18 4:32 AM, silvioprog wrote:
> > Hi Santos and Christian.
> >
> > Santos, the Christian's answer was complete and clarified me a lot too. ☺
> >
> > Christian, it seems sendfile() is not available in mingw and in the
> > Google's NDK (Android), but my library must work at least on
> > Windows, Linux, Raspbian and Android, so I've used the fopen() in this
> > way: link
> > <https://github.com/risoflora/libsagui/blob/master/src/sg_httpres.c#L112
> > (sorry
> > for put a link, but the code is a little bit large). Do you recommend to
> > use fopen() in those cases? I would like to use sendfile() because it is
> > very fast, but I can't implement it for Windows.
> >
> > Thanks in advance for your help!
> >
> > On Wed, Nov 28, 2018 at 1:55 PM Christian Grothoff <groth...@gnunet.org
> > <mailto:groth...@gnunet.org>> wrote:
> >
> > Dear Santos,
> >
> > MHD can send chunked response to the client. That said, this is not a
> > question of how large the file is. Especially, if it is simply a very
> > large file, MHD allows you to use sendfile() to avoid having a copy
> of
> > the data in userspace. Chunked encoding is usually only used if the
> > application simply does not know the size of the response ahead of
> time.
> >
> > Happy hacking!
> >
> > Christian
> >
> > On 11/28/18 4:09 PM, Santos Das wrote:
> > > Hi,
> > >
> > > How can MHD send the chunked message to the client ?
> > >
> > > When a server receives a simple GET request, and the response is so
> > > large that it must be sent back using the "chunked" procedure.
> > >
> > > An example would be if a server received a file GET request, and
> the
> > > response is a large data file.
> > >
> > > Can you please point me to some example on how this can be done
> > using MHD ?
> > >
> > > Thanks in advance.
> > >
> > > Regards, Santos
> >
> >
> > --
> > Silvio Clécio
>
--
Silvio Clécio
