On Tue, Feb 9, 2021 at 12:56 PM Fulup Ar Foll via curl-library <
curl-library@cool.haxx.se> wrote:
> Hi,
>
> I posted on github a template for libcurl asynchronous usage with
> libuv+libsystemd mainloop abstraction:
> https://github.com/fulup-bzh/libcurl-mainloop .
>
> While I spend in significant
On Tue, Dec 1, 2020 at 3:34 PM Daniel Stenberg via curl-library <
curl-library@cool.haxx.se> wrote:
> Hi,
>
> Here's an idea: a function that can return the equivalent curl command
> line
> based on the options set in the easy handle!
>
> For debugging and more.
>
>https://github.com/curl/curl
Hi,
On Sun, Nov 29, 2020 at 4:36 PM Patrick Schlangen
wrote:
> Hi,
>
> Am 29.11.2020 um 15:28 schrieb James Read :
>
> Setting CURLOPT_CAINFO to NULL causes thousands of errors of the type:
>
> DONE:
> https://www.gosunoob.com/witcher-3/ballad-heroes-dlc-gwent-card-set/attachment/witcher_3_ville
Hi,
On Sun, Nov 29, 2020 at 2:16 PM James Read wrote:
> Hi,
>
> On Fri, Nov 27, 2020 at 7:56 AM Patrick Schlangen
> wrote:
>
>> On Fri, 27 Nov 2020, James Read wrote:
>> > Has anybody ever actually succeeded in making a high performance
>> application
>> > with epoll/libcurl as the back end.
>>
Hi,
On Fri, Nov 27, 2020 at 7:56 AM Patrick Schlangen
wrote:
> On Fri, 27 Nov 2020, James Read wrote:
> > Has anybody ever actually succeeded in making a high performance
> application
> > with epoll/libcurl as the back end.
>
> Yes. Some things I've learned (for my usecase):
>
What's your usec
Hi,
On Fri, Nov 27, 2020 at 6:22 PM Felipe Gasper
wrote:
>
>
> > On Nov 27, 2020, at 5:29 AM, James Read via curl-library <
> curl-library@cool.haxx.se> wrote:
> >
> > Hi,
> >
> > On Fri, Nov 27, 2020 at 7:22 AM Daniel Stenberg wrote:
> > O
>
>
> Does the ephiperfifo.c example show the problem you experience if you pipe
> in
> say a 100 different URLs?
>
>
With 100 URLs I get speed of 8Mbps. With 1000 URLs I get average speed of
about 16Mbps.
James Read
---
Unsubscribe:
Hi,
On Fri, Nov 27, 2020 at 7:22 AM Daniel Stenberg wrote:
> On Fri, 27 Nov 2020, James Read wrote:
>
> > Has anybody ever actually succeeded in making a high performance
> application
> > with epoll/libcurl as the back end.
>
> Yes. Although most people I know of use an event library in between
Hi,
On Thu, Nov 26, 2020 at 10:20 PM Daniel Stenberg wrote:
> On Thu, 26 Nov 2020, James Read via curl-library wrote:
>
> > I don't know if this helps but I would be willing to become a silver
> sponsor
> > of the libcurl project if I can find a decent solution to
Hi,
On Tue, Nov 24, 2020 at 7:50 PM James Read wrote:
> Hi,
>
> On Tue, Nov 24, 2020 at 5:37 PM Tomalak Geret'kal via curl-library <
> curl-library@cool.haxx.se> wrote:
>
>> On 23/11/2020 20:16, James Read via curl-library wrote:
>> > I have
Hi,
On Tue, Nov 24, 2020 at 5:37 PM Tomalak Geret'kal via curl-library <
curl-library@cool.haxx.se> wrote:
> On 23/11/2020 20:16, James Read via curl-library wrote:
> > I have attempted to make two minimal codes that
> > demonstrate my problem.
> >
> > The
Hi,
On Tue, Nov 24, 2020 at 5:37 PM Tomalak Geret'kal via curl-library <
curl-library@cool.haxx.se> wrote:
> On 23/11/2020 20:16, James Read via curl-library wrote:
> > I have attempted to make two minimal codes that
> > demonstrate my problem.
> >
> > The
Hi,
I have attempted to make two minimal codes that demonstrate my problem.
The first can be downloaded from https://github.com/JamesRead5737/fast
It basically recursively downloads http://www.google.com,
http://www.yahoo.com and http://www.bing.com
I am able to achieve download speeds of up to 7
Hi,
when using curl_multi_socket_action combined with epoll are connect calls
done non blocking under the hood?
James Read
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.h
Hi,
I have some error messages in libcurl logs like as follows:
== Info: Couldn't parse CURLOPT_RESOLVE entry '
http://kitcheneroktoberfest.com/:443:69.172.201.153'!
== Info: Couldn't parse CURLOPT_RESOLVE entry '
http://kitcheneroktoberfest.com/:80:69.172.201.153'!
The following is the relevant
Hi,
are there any undocumented calls such that I can inspect the DNS cache to
ensure things are working as expected?
James Read
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.se/mail/etique
On Mon, Nov 9, 2020 at 10:52 PM Daniel Stenberg wrote:
> On Mon, 9 Nov 2020, Daniel Stenberg wrote:
>
> > There's fixed limit to what amount of entries that can be added.
>
> Sorry, I meant to write: there's NO fixed limit.
>
>
So, will that lead to a memory leak?
James Read
> --
>
> / danie
Hi,
how many entries can the DNS cache hold?
James Read
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
On Wed, Jun 10, 2020 at 9:01 AM Daniel Stenberg wrote:
> On Wed, 10 Jun 2020, James Read via curl-library wrote:
>
> > /* abort if slower than 30 bytes/sec during 60 seconds */
> > curl_easy_setopt(curl, CURLOPT_LOW_SPEED_TIME, 60L);
> > curl_easy_setopt(curl, CURL
Hi,
I am investigating using CURLOPT_LOW_SPEED_TIME and CURLOPT_LOW_SPEED_LIMIT
to speed up my web crawler. I am a little confused though by a comment in
the documentation. According to
https://curl.haxx.se/libcurl/c/CURLOPT_LOW_SPEED_TIME.html the following
code segment says:
/* abort if slower
On Mon, Jun 8, 2020 at 10:35 PM Daniel Stenberg wrote:
> On Mon, 8 Jun 2020, James Read via curl-library wrote:
>
> > This is because my web crawler uses this function and the read() call
> seems
> > to be causing problems with my web crawler.
> >
> > What is t
I am trying to understand the function timer_cb in
https://curl.haxx.se/libcurl/c/ephiperfifo.html
This is because my web crawler uses this function and the read() call seems
to be causing problems with my web crawler.
What is the read() call for?
It has been suggested to me elsewhere that I cou
On Fri, Jun 5, 2020 at 9:30 AM Zakrzewski, Jakub via curl-library <
curl-library@cool.haxx.se> wrote:
> My wild guess is that Valgrind have something against the uninitialized
> bytes in the "ev" objects. The union epoll_data is 64 bits but you set only
> the 32-bit "fd" member.
> Try zeroing "ev"
Hi,
I have some errors reported by valgrind that may or may not be related to
some strange behaviour with my crawler. Here is the output from valgrind:
==299487== Memcheck, a memory error detector
==299487== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==299487== Using Valgrind
On Tue, May 26, 2020 at 4:26 PM Daniel Stenberg wrote:
> On Tue, 26 May 2020, James Read wrote:
>
> > So, I guess that's problem solved. Thanks.
>
> Lovely!
>
> With that issue nailed and perhaps with some newfound knowledge in your
> head,
> is there something we should think about to clarify or
On Tue, May 26, 2020 at 3:43 PM Daniel Stenberg wrote:
> On Tue, 26 May 2020, James Read via curl-library wrote:
>
> > When parallel connections reaches 0 g->still_running is still reporting
> a
> > number of easy handles in progress. How can this be? Surely, the answer
&
On Tue, May 26, 2020 at 2:31 PM James Read wrote:
>
>
> On Tue, May 26, 2020 at 12:47 PM James Read
> wrote:
>
>>
>>
>> On Tue, May 26, 2020 at 7:30 AM Patrick Monnerat via curl-library <
>> curl-library@cool.haxx.se> wrote:
>>
>&g
On Tue, May 26, 2020 at 12:47 PM James Read wrote:
>
>
> On Tue, May 26, 2020 at 7:30 AM Patrick Monnerat via curl-library <
> curl-library@cool.haxx.se> wrote:
>
>> On 5/26/20 1:15 AM, James Read via curl-library wrote:
>> >
>> > git clone https://g
On Tue, May 26, 2020 at 7:30 AM Patrick Monnerat via curl-library <
curl-library@cool.haxx.se> wrote:
> On 5/26/20 1:15 AM, James Read via curl-library wrote:
> >
> > git clone https://github.com/JamesRead5737/libcurlmemoryleak.git
> >
> > No need to make. Just co
On Tue, May 26, 2020 at 12:32 AM Jeffrey Walton wrote:
> On Mon, May 25, 2020 at 7:16 PM James Read
> wrote:
> >
> >
> >
> > On Tue, May 26, 2020 at 12:02 AM Jeffrey Walton
> wrote:
> >>
> >> On Mon, May 25, 2020
On Tue, May 26, 2020 at 12:02 AM Jeffrey Walton wrote:
> On Mon, May 25, 2020 at 6:27 PM James Read via curl-library
> wrote:
> >
> > ...
> >
> > Gmail seems to have taken out all the formatting. Apologies. It should
> still compile though.
>
> I ca
On Mon, May 25, 2020 at 11:15 PM James Read wrote:
>
>
> On Mon, May 25, 2020 at 10:37 PM Daniel Stenberg wrote:
>
>> On Mon, 25 May 2020, James Read wrote:
>>
>> > I call curl_multi_cleanup here:
>>
>> > I call curl_easy_cleanup here:
>>
>> > What am I missing?
>>
>> I don't think we'll be able
On Mon, May 25, 2020 at 10:37 PM Daniel Stenberg wrote:
> On Mon, 25 May 2020, James Read wrote:
>
> > I call curl_multi_cleanup here:
>
> > I call curl_easy_cleanup here:
>
> > What am I missing?
>
> I don't think we'll be able to tell you that. We can't reproduce this
> problem.
> We don't see
On Mon, May 25, 2020 at 7:56 AM Daniel Stenberg wrote:
> On Sun, 24 May 2020, James Read via curl-library wrote:
>
> > ==78076==by 0x48BBEE0: curl_dbg_calloc (memdebug.c:205)
> > ==78076==by 0x490A1D0: Curl_ssl_initsessions (vtls.c:608)
>
> This is the TLS
On Sun, May 24, 2020 at 9:56 PM Patrick Schlangen
wrote:
> Am 24.05.2020 um 21:56 schrieb James Read via curl-library <
> curl-library@cool.haxx.se>:
> > ...
> > On closer inspection my valgrind output has the following lines:
> >
> > --69689-- Reading
On Sun, May 24, 2020 at 5:43 PM James Read wrote:
>
>
> On Sun, May 24, 2020 at 4:47 PM James Read
> wrote:
>
>>
>>
>> On Sun, May 24, 2020 at 4:07 PM Daniel Stenberg wrote:
>>
>>> On Sun, 24 May 2020, James Read via curl-library wrote:
&g
On Sun, May 24, 2020 at 4:47 PM James Read wrote:
>
>
> On Sun, May 24, 2020 at 4:07 PM Daniel Stenberg wrote:
>
>> On Sun, 24 May 2020, James Read via curl-library wrote:
>>
>> > Valgrind reports a memory leak in my web crawler:
>>
>> ...
>>
&
On Sun, May 24, 2020 at 4:07 PM Daniel Stenberg wrote:
> On Sun, 24 May 2020, James Read via curl-library wrote:
>
> > Valgrind reports a memory leak in my web crawler:
>
> ...
>
> > What is memory being allocated for?
>
> Since your stack trace has no debug
On Sun, May 24, 2020 at 1:02 PM Aleksandar Lazic
wrote:
> Hi.
>
> On 24.05.20 02:44, James Read via curl-library wrote:
> > Valgrind reports a memory leak in my web crawler:
> >
> > ==36126== 923,440 bytes in 485 blocks are possibly lost in loss record
> 56 of 56
Valgrind reports a memory leak in my web crawler:
==36126== 923,440 bytes in 485 blocks are possibly lost in loss record 56
of 56
==36126==at 0x483DD99: calloc (in
/usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==36126==by 0x4896414: ??? (in
/usr/lib/x86_64-linux-gnu
On Thu, May 21, 2020 at 8:58 PM James Read wrote:
>
>
> On Thu, May 21, 2020 at 4:18 PM Dan Fandrich via curl-library <
> curl-library@cool.haxx.se> wrote:
>
>> On Thu, May 21, 2020 at 03:46:33PM +0100, James Read via curl-library
>> wrote:
>> > I
On Thu, May 21, 2020 at 4:18 PM Dan Fandrich via curl-library <
curl-library@cool.haxx.se> wrote:
> On Thu, May 21, 2020 at 03:46:33PM +0100, James Read via curl-library
> wrote:
> > I'm implementing a simple web crawler with curl and want to retrieve the
> >
Hi,
I'm implementing a simple web crawler with curl and want to retrieve the
Last-Modified header so I can implement a sensible recrawl policy. I've
found https://curl.haxx.se/libcurl/c/getinfo.html which is a nice easy way
to retrieve the Content-Type header. Is there a similarly easy way to
retr
On Thu, Aug 9, 2018 at 7:56 AM, Daniel Stenberg wrote:
> On Thu, 9 Aug 2018, James Read wrote:
>
> Everything seems to work fine. With a single URL and with multiple URLs.
>> The only issue I have is the throughput.
>>
>
> Could be vphiperfifo.c example issues.
>
> For example, I don't see how it
>
> and make sure the application works correctly when using a small amount of
>> transfers - perhaps a single one to start with - and then add more when
>> you've confirmed that.
>
>
Everything seems to work fine. With a single URL and with multiple URLs.
The only issue I have is the throughput.
-
On Wed, Aug 8, 2018 at 10:56 AM, Daniel Stenberg wrote:
> On Tue, 7 Aug 2018, James Read wrote:
>
> As long as libcurl is built to use the threaded resolver (default) or the
>>> c-ares backend, it will do name resolving in a non-blocking manner.
>>>
>>
>> I just did a standard install using the l
On Tue, Aug 7, 2018 at 11:11 PM, Daniel Stenberg wrote:
> On Tue, 7 Aug 2018, James Read wrote:
>
> What about DNS resolution? Is this already asynchronous or do I need to
>> make some changes?
>>
>
> As long as libcurl is built to use the threaded resolver (default) or the
> c-ares backend, it w
On Tue, Aug 7, 2018 at 9:37 PM, Daniel Stenberg wrote:
> On Tue, 7 Aug 2018, James Read via curl-library wrote:
>
> Does anyone have a working example using multi_socket with epoll?
>>
>
> Josh Bialkowski provided one not too long ago, available here:
>
> htt
I have a list of 135M .com domains and want to implement a basic web
crawler to crawl this list of domains. My early design decisions are that I
want to use multi_socket libcurl library with epoll. However, the book
everything curl doesn't contain any working examples of using multi_socket
with epo
49 matches
Mail list logo
