On Tue, May 10, 2011 at 04:04:27PM -0700, Scott Lamb wrote: > I don't see the point in libevent implementing spooling to a file > (descriptor). If an application wants that, it can implement it on top > of the existing evhttp_request_set_chunked_cb + the fairly small > patches being proposed (call the chunked cb more often, offer some > sort of flow control, make it possible for the server to set a chunked > callback for POST bodies). It's not possible to do the reverse in a > satisfactory way:
The problem - and the main reason I never actually posted the patch - was because there is a flaw in the set_chunked_cb logic. It only makes sense when the chunks are of reasonable size. You are only getting data after that one chunk has been fully processed. > * fork()+exec() in the request path is slow (think CGI). Maybe this > isn't significant for long requests but if you also get any small > requests it'd be problematic. Also even ignoring performance, it's > surprisingly hard to fork() correctly in threaded programs. > I can't figure out whether you are working through some other problems by typing them out, or if this is a response to something I said. > * Likewise, directly spooling to a file on disk is not satisfying for > several reasons: > > ** It doesn't solve the problem of failing on an infinite stream; it > makes some larger requests possible but for infinite ones just makes > the failure mode running out of disk instead of running out of memory. > Lets put this issue to rest.. *NOTHING* solves the problem of infinite inputs. > ** It could as much as double the response time for proxy servers: > they must receive the entire request then send the entire request > instead of doing them simultaneously. And it'd break the downstream > client's progress indicator. > Then keep it in memory. > ** It's too slow to be done synchronously in the network thread (a > single seek is ~10 ms under even ideal conditions, and I was just > reading about cases at work where it could be more like a second) and > there's no satisfactory async API for plain files, so it would require > threading which complicates things. Currently I believe libevent > doesn't manage any thread pools; the application is required to do all > that. And libevent can even be compiled without threading support so > you'd need to this support to be conditional on threading. > A majority of developers understand the issues that coincide with disk IO. Again, I must reiterate - this would be an optional flag. Having a cross compatible mechanism for transparently spooling data for large streams (note: not chunks) of data isn't bad, it's an alternative. > ** As previously mentioned, there are embedded cases where it's > impossible because there is no disk. > Then keep it in memory. > I think libevent should provide the minimal API that satisfies all use > cases, and that's chunked callback + flow control. > Option A: use the partial data flag method I previously posted (which solves both chunking and large data issues). Option B: depend only on the chunk callback mechanism, which does not solve the issue with large data streams. Option C: Use a spooling method. Option D: Implement all of the above options, allow the developer to choose their own doom. *********************************************************************** To unsubscribe, send an e-mail to majord...@freehaven.net with unsubscribe libevent-users in the body.
