Re: multi_socket and epoll example

2018-08-09 Thread James Read via curl-library
On Thu, Aug 9, 2018 at 7:56 AM, Daniel Stenberg wrote: > On Thu, 9 Aug 2018, James Read wrote: > > Everything seems to work fine. With a single URL and with multiple URLs. >> The only issue I have is the throughput. >> > > Could be vphiperfifo.c example issues. > > For example, I don't see how it

Re: multi_socket and epoll example

2018-08-09 Thread Daniel Stenberg via curl-library
On Thu, 9 Aug 2018, James Read wrote: Everything seems to work fine. With a single URL and with multiple URLs. The only issue I have is the throughput. Could be vphiperfifo.c example issues. For example, I don't see how it handles API timeouts properly: the timeout that libcurl tells the CUR

Re: multi_socket and epoll example

2018-08-08 Thread James Read via curl-library
> > and make sure the application works correctly when using a small amount of >> transfers - perhaps a single one to start with - and then add more when >> you've confirmed that. > > Everything seems to work fine. With a single URL and with multiple URLs. The only issue I have is the throughput. -

Re: multi_socket and epoll example

2018-08-08 Thread James Read via curl-library
On Wed, Aug 8, 2018 at 10:56 AM, Daniel Stenberg wrote: > On Tue, 7 Aug 2018, James Read wrote: > > As long as libcurl is built to use the threaded resolver (default) or the >>> c-ares backend, it will do name resolving in a non-blocking manner. >>> >> >> I just did a standard install using the l

Re: multi_socket and epoll example

2018-08-08 Thread Daniel Stenberg via curl-library
On Tue, 7 Aug 2018, James Read wrote: As long as libcurl is built to use the threaded resolver (default) or the c-ares backend, it will do name resolving in a non-blocking manner. I just did a standard install using the latest git clone. Then it probably built with the threaded resolver. At

Re: multi_socket and epoll example

2018-08-07 Thread James Read via curl-library
On Tue, Aug 7, 2018 at 11:11 PM, Daniel Stenberg wrote: > On Tue, 7 Aug 2018, James Read wrote: > > What about DNS resolution? Is this already asynchronous or do I need to >> make some changes? >> > > As long as libcurl is built to use the threaded resolver (default) or the > c-ares backend, it w

Re: multi_socket and epoll example

2018-08-07 Thread Daniel Stenberg via curl-library
On Tue, 7 Aug 2018, James Read wrote: What about DNS resolution? Is this already asynchronous or do I need to make some changes? As long as libcurl is built to use the threaded resolver (default) or the c-ares backend, it will do name resolving in a non-blocking manner. -- / daniel.haxx.s

Re: multi_socket and epoll example

2018-08-07 Thread James Read via curl-library
On Tue, Aug 7, 2018 at 9:37 PM, Daniel Stenberg wrote: > On Tue, 7 Aug 2018, James Read via curl-library wrote: > > Does anyone have a working example using multi_socket with epoll? >> > > Josh Bialkowski provided one not too long ago, available here: > > https://curl.haxx.se/libcurl/c/ephiperf

Re: multi_socket and epoll example

2018-08-07 Thread Daniel Stenberg via curl-library
On Tue, 7 Aug 2018, James Read via curl-library wrote: Does anyone have a working example using multi_socket with epoll? Josh Bialkowski provided one not too long ago, available here: https://curl.haxx.se/libcurl/c/ephiperfifo.html -- / daniel.haxx.se

multi_socket and epoll example

2018-08-07 Thread James Read via curl-library
I have a list of 135M .com domains and want to implement a basic web crawler to crawl this list of domains. My early design decisions are that I want to use multi_socket libcurl library with epoll. However, the book everything curl doesn't contain any working examples of using multi_socket with epo