[Twisted-Python] Reactor Spinning?
Hey Twisted folks,
I'm using the twisted.web framework for a high performance HTTP proxy
server that very closely resembles the HTTP proxy server example that
comes with the twisted package.
Under heavy load, I occasionally run into a problem where the reactor
appears to start spinning on an "empty" write. An strace of the process
reveals these system calls:
epoll_wait(3, {{EPOLLOUT, {u32=81, u64=13194703840475086929}}}, 269, 8) = 1
write(81, "", 0)= 0
gettimeofday({1272980323, 567613}, NULL) = 0
gettimeofday({1272980323, 567696}, NULL) = 0
epoll_wait(3, {{EPOLLOUT, {u32=81, u64=13194703840475086929}}}, 269, 8) = 1
write(81, "", 0)= 0
gettimeofday({1272980323, 567988}, NULL) = 0
gettimeofday({1272980323, 568071}, NULL) = 0
epoll_wait(3, {{EPOLLOUT, {u32=81, u64=13194703840475086929}}}, 269, 7) = 1
write(81, "", 0)= 0
gettimeofday({1272980323, 568501}, NULL) = 0
gettimeofday({1272980323, 568613}, NULL) = 0
epoll_wait(3, {{EPOLLOUT, {u32=81, u64=13194703840475086929}}}, 269, 7) = 1
write(81, "", 0)= 0
gettimeofday({1272980323, 568978}, NULL) = 0
gettimeofday({1272980323, 569084}, NULL) = 0
epoll_wait(3, {{EPOLLOUT, {u32=81, u64=13194703840475086929}}}, 269, 6) = 1
write(81, "", 0)= 0
...it seems to be waiting for write availability on a FD, trying to
write an empty buffer, seeing that it wrote zero bytes, and so waiting
for availability again.
I've confirmed that when this occurs, whatever FD it is spinning on does
correspond with a network socket connected to the proxy. I've seen this
with both Twisted 8.2 and 9.0 running on Linux using both the epoll and
select reactors.
If anyone has any thoughts about this problem or has seen it before, I'd
appreciate any insight that anyone might have.
Thanks,
- moxie
--
http://www.thoughtcrime.org
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Reactor Spinning?
Does it spin forever or does it clear up after some time?
I have occasional 'hickups' with heavy loads with responses taking
several ten seconds for no apparent reason, the server then working
again like nothing happened.
Werner
Moxie Marlinspike wrote:
> Hey Twisted folks,
>
> I'm using the twisted.web framework for a high performance HTTP proxy
> server that very closely resembles the HTTP proxy server example that
> comes with the twisted package.
>
> Under heavy load, I occasionally run into a problem where the reactor
> appears to start spinning on an "empty" write. An strace of the process
> reveals these system calls:
>
> epoll_wait(3, {{EPOLLOUT, {u32=81, u64=13194703840475086929}}}, 269, 8) = 1
> write(81, "", 0)= 0
> gettimeofday({1272980323, 567613}, NULL) = 0
> gettimeofday({1272980323, 567696}, NULL) = 0
> epoll_wait(3, {{EPOLLOUT, {u32=81, u64=13194703840475086929}}}, 269, 8) = 1
> write(81, "", 0)= 0
> gettimeofday({1272980323, 567988}, NULL) = 0
> gettimeofday({1272980323, 568071}, NULL) = 0
> epoll_wait(3, {{EPOLLOUT, {u32=81, u64=13194703840475086929}}}, 269, 7) = 1
> write(81, "", 0)= 0
> gettimeofday({1272980323, 568501}, NULL) = 0
> gettimeofday({1272980323, 568613}, NULL) = 0
> epoll_wait(3, {{EPOLLOUT, {u32=81, u64=13194703840475086929}}}, 269, 7) = 1
> write(81, "", 0)= 0
> gettimeofday({1272980323, 568978}, NULL) = 0
> gettimeofday({1272980323, 569084}, NULL) = 0
> epoll_wait(3, {{EPOLLOUT, {u32=81, u64=13194703840475086929}}}, 269, 6) = 1
> write(81, "", 0)= 0
>
> ...it seems to be waiting for write availability on a FD, trying to
> write an empty buffer, seeing that it wrote zero bytes, and so waiting
> for availability again.
>
> I've confirmed that when this occurs, whatever FD it is spinning on does
> correspond with a network socket connected to the proxy. I've seen this
> with both Twisted 8.2 and 9.0 running on Linux using both the epoll and
> select reactors.
>
> If anyone has any thoughts about this problem or has seen it before, I'd
> appreciate any insight that anyone might have.
>
> Thanks,
>
> - moxie
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Reactor Spinning?
I've seen it clear up before, but it will normally spin for days. This
state doesn't really impact performance at all (requests are still
snappy), but it is annoying how much CPU time it eats through before a
restart.
- moxie
--
http://www.thoughtcrime.org
Werner Thie wrote:
> Does it spin forever or does it clear up after some time?
>
> I have occasional 'hickups' with heavy loads with responses taking
> several ten seconds for no apparent reason, the server then working
> again like nothing happened.
>
> Werner
>
> Moxie Marlinspike wrote:
>> Hey Twisted folks,
>>
>> I'm using the twisted.web framework for a high performance HTTP proxy
>> server that very closely resembles the HTTP proxy server example that
>> comes with the twisted package.
>>
>> Under heavy load, I occasionally run into a problem where the reactor
>> appears to start spinning on an "empty" write. An strace of the process
>> reveals these system calls:
>>
>> epoll_wait(3, {{EPOLLOUT, {u32=81, u64=13194703840475086929}}}, 269, 8) = 1
>> write(81, "", 0)= 0
>> gettimeofday({1272980323, 567613}, NULL) = 0
>> gettimeofday({1272980323, 567696}, NULL) = 0
>> epoll_wait(3, {{EPOLLOUT, {u32=81, u64=13194703840475086929}}}, 269, 8) = 1
>> write(81, "", 0)= 0
>> gettimeofday({1272980323, 567988}, NULL) = 0
>> gettimeofday({1272980323, 568071}, NULL) = 0
>> epoll_wait(3, {{EPOLLOUT, {u32=81, u64=13194703840475086929}}}, 269, 7) = 1
>> write(81, "", 0)= 0
>> gettimeofday({1272980323, 568501}, NULL) = 0
>> gettimeofday({1272980323, 568613}, NULL) = 0
>> epoll_wait(3, {{EPOLLOUT, {u32=81, u64=13194703840475086929}}}, 269, 7) = 1
>> write(81, "", 0)= 0
>> gettimeofday({1272980323, 568978}, NULL) = 0
>> gettimeofday({1272980323, 569084}, NULL) = 0
>> epoll_wait(3, {{EPOLLOUT, {u32=81, u64=13194703840475086929}}}, 269, 6) = 1
>> write(81, "", 0)= 0
>>
>> ...it seems to be waiting for write availability on a FD, trying to
>> write an empty buffer, seeing that it wrote zero bytes, and so waiting
>> for availability again.
>>
>> I've confirmed that when this occurs, whatever FD it is spinning on does
>> correspond with a network socket connected to the proxy. I've seen this
>> with both Twisted 8.2 and 9.0 running on Linux using both the epoll and
>> select reactors.
>>
>> If anyone has any thoughts about this problem or has seen it before, I'd
>> appreciate any insight that anyone might have.
>>
>> Thanks,
>>
>> - moxie
>>
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Reactor Spinning?
> > Hey Twisted folks, > > I'm using the twisted.web framework for a high performance HTTP proxy > server that very closely resembles the HTTP proxy server example that > comes with the twisted package. > > Under heavy load, I occasionally run into a problem where the reactor > appears to start spinning on an "empty" write. An strace of the process > reveals these system calls: Can you produce a minimal reproducing example of a program that has this issue? I can imagine a code bug where you're calling transport.resumeProducing() or startWriting() sometime when you shouldn't, except the empty write() does look wrong... a minimal code sample would help. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Reactor Spinning?
On 01:48 pm, mo...@thoughtcrime.org wrote:
>
>Hey Twisted folks,
>
>I'm using the twisted.web framework for a high performance HTTP proxy
>server that very closely resembles the HTTP proxy server example that
>comes with the twisted package.
>
>Under heavy load, I occasionally run into a problem where the reactor
>appears to start spinning on an "empty" write. An strace of the
>process
>reveals these system calls:
>
>epoll_wait(3, {{EPOLLOUT, {u32=81, u64=13194703840475086929}}}, 269, 8)
>= 1
>write(81, "", 0)= 0
It might be interesting to learn what's in the reactor that's associated
with this file descriptor and what the state of its send buffer is.
Some strategically placed log messages might help with that, or a
manhole in the process that you can connect to when you notice the
problem has started in order to poke around.
Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] win32com client in a twisted thread
Andrew, Itamar, thanks. I've taken the hint to CoInitialize the thread. That gets me a bit further, but there's still some strange stuff going on. I'm going to have to put this on the back burner for a while in favor of higher priority stuff. -- Don Dwiggins Advanced Publishing Technology ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
