Re: CURLOPT_READFUNCTION performance issue

2013-07-29 Thread Clemens Gruber
>>> it will not benefit a carefully written application that takes proper care >>> of buffering; the algorithm has either no effect, or negative effect on the >>> application. >> >> Does curl take "proper care of buffering"? If yes, could it be beneficial to >> set CURLOPT_TCP_NODELAY per defau

Re: CURLOPT_READFUNCTION performance issue

2013-07-29 Thread Daniel Stenberg
On Sun, 28 Jul 2013, Clemens Gruber wrote: Would be interesting to know if the Nagle algorithm improves libcurl performance at all. Maybe it does in some protocol handlers, but not in others.., etc. libcurl is a pretty thin layer too in some aspects. I can very well imagine that an applicati

Re: CURLOPT_READFUNCTION performance issue

2013-07-28 Thread Clemens Gruber
>> Is there any way curl could automatically detect when it would be >> appropriate to use this option by default? > > Not that I can think of. I would guess that it is hard for us to second-guess > the TCP stack's behavior generically. I think we should start with > considering some mention ab

Re: CURLOPT_READFUNCTION performance issue

2013-07-28 Thread Daniel Stenberg
On Sat, 27 Jul 2013, Michael Dowling wrote: I reran my various performance tests, and I can confirm that adding CURLOPT_TCP_NODELAY resolves the performance issue. Great! Is there any way curl could automatically detect when it would be appropriate to use this option by default? Not that I

Re: CURLOPT_READFUNCTION performance issue

2013-07-27 Thread Michael Dowling
I wish I had thought of the Nagle slow start algorithm earlier! I reran my various performance tests, and I can confirm that adding CURLOPT_TCP_NODELAY resolves the performance issue. Is there any way curl could automatically detect when it would be appropriate to use this option by default?

Re: CURLOPT_READFUNCTION performance issue

2013-07-27 Thread Daniel Stenberg
On Fri, 26 Jul 2013, Daniel Stenberg wrote: tl;dr: I fixed the problem by setting CURLOPT_TCP_NODELAY. Using strace -T, I can also see that the poll() call actually takes that long: 23:58:38.816031 poll([{fd=3, events=POLLIN}], 1, 1002) = 1 ([{fd=3, revents=POLLIN}]) <0.037972> This was th

Re: CURLOPT_READFUNCTION performance issue

2013-07-26 Thread Daniel Stenberg
On Thu, 25 Jul 2013, Daniel Stenberg wrote: However, at line 75 we can see the second HTTP response being received 36817 microseconds after the data was sent! Going down the log we can see that all subsequent responses show this exessive time. ... I can also confirm these numbers when straci

Re: CURLOPT_READFUNCTION performance issue

2013-07-25 Thread Michael Dowling
Hi Daniel, Thanks again for the test script. I was able to reproduce the performance issue by simply pointing the test script you provided to a simple web server that runs on node.js (change the URL in the test script to "http://localhost:8124";). I created a gist that contains the source of the t

Re: CURLOPT_READFUNCTION performance issue

2013-07-23 Thread Michael Dowling
server. Thanks, Michael Message: 4 Date: Tue, 23 Jul 2013 20:18:28 +0200 (CEST) From: Daniel Stenberg To: libcurl development Subject: Re: CURLOPT_READFUNCTION performance issue Message-ID: Content-Type: text/plain; charset="us-ascii"; Format="flowed" On Mon, 22 Jul 201

Re: CURLOPT_READFUNCTION performance issue

2013-07-23 Thread Daniel Stenberg
On Mon, 22 Jul 2013, Michael Dowling wrote: I've noticed that there appears to be a significant performance hit when using CURLOPT_READFUNCTION. This issue seems to be platform dependent as I've only been able to get poor performance on Linux (Amazon Linux m1.large 64-bit ami-0358ce33) across

CURLOPT_READFUNCTION performance issue

2013-07-22 Thread Michael Dowling
I've noticed that there appears to be a significant performance hit when using CURLOPT_READFUNCTION. This issue seems to be platform dependent as I've only been able to get poor performance on Linux (Amazon Linux m1.large 64-bit ami-0358ce33) across multiple versions of cURL and PHP. I've not seen