Re: [Twisted-Python] Need help for structured application development
On Fri, Feb 19, 2010 at 5:45 AM, Johann Borck wrote: > I see. So if I get these requirements right you have 5-10 services with > 1-30 instances of each, with following properties relevant to the task > at hand: > > 1. those pollable by your existing program. > 2. those incompatible with your existing program. > 3. those that do not stream additional data. > 4. those that do stream additional data. > > since sets (1 and 2) and (3 and 4) are distinct, the combinations > (1,3),(1,4),(2,3),(2,4) are possible. > > (1,3) the easy one, obviously you just need an object that polls the > service using your program in intervals specific to the service. > (1,4) Question: will the data you're interested in be collected by your > existing program or by twisted? In the former case, it's basically the > same as (1,3), in the latter you'll have to implement a protocol. > (2,3) for these you'll need to implement a protocol class. > (2,4) here you'll have to implement one or two protocols, depending on > how the service is implemented. > > Is the above about correct? I think it would be a good idea to have an > object OB that keeps references to all objects that gather data from the > services, grouped by the type of service they're responsible for > (defaultdict(list or dict) comes to mind). And then you'll probably > either want factories that take care of handing newly created protocol > instances over to OB or some instances (one or two per service in sets > (2,3,4) ) of a multipurpose factory that can be initialized with the > respective protocol and the information how to pass the created protocol > over to OB, maybe just a simple method that is able to distinguish the > protocols by the interfaces they implement. > > One pitfall might be your polling program in case you're using > t.i.u.getProcessOutput. It (t.i.u...) provides an asynchronous > interface, so the worst that can happen is a stray process that doesn't > return, but you still might want to consider implementing a > ProcessProtocol > (http://twistedmatrix.com/documents/current/core/howto/process.html) > with a reasonable timeout, to be able to kill the spawned process in > case it doesn't terminate. Thinking about it, it's the better solution > anyway, because process protocols are just another type of protocol in > twisted, and can be integrated consistently with the rest of your app. > > Another utility you might want is t.i.task.LoopingCall, for obvious > reasons. Given your requirements something along these lines would be my > approach, although I'd probably reimplement the polling thingy in > twisted if it's not too complex :) > > hope that makes sense, > Johann Johann, Thank you very much for your detailed response. It is greatly appreciated. I think what you describe makes sense, and I hope I understand what you mean and how I can implement it: I need to create a "suberobject" OB which references all the poller-objects. I need to create one object per detected server instance, which takes care of creating a timed spawning of the external poller-process, and passing it along to OB. For those servers that require streamlisteners, I really only need one factory per server type, with the ablity to match streamed data with polled data via the poller-object. Is this somewhat on the right track? I absolutely hate creating lots of code just to find that it was done in the "wrong" way, structurally speaking, and that it requires a lot of work to rewrite in the "correct way" to allow for further expansion of the application. Your answers are very helpful in letting me avoid spaghetti-code. Cheers, Einar ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] fast high load protocol
Thanks a lot, Stephen. AMP probably is what I looking for. Now I can handle a much more clients (2 times more without significantly increasing response time). Server can also handle 3 times more clients without any errors, but response time grow up. I check for netstat and find out Recv-Q about 1706013 on client-side of AMP and Send-Q about 642288 on AMP server-side. Is this meant: - OS needs tuning (ulimit or ifconfig) - or Twisted/Python can't handle so much amp-connections? CPU utilization < 10%, free memory about 500M from 1.7G. I'd appreciate any help. On Wed, Feb 17, 2010 at 1:32 AM, Stephen Thorne wrote: > > Vlad, > > Have you considered using twisted.protocols.amp ? > > -- > Regards, > Stephen Thorne > Development Engineer > Netbox Blue > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -- WBR, Vlad Shevchenko ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] fast high load protocol
Vlad Shevchenko wrote: > Thanks a lot, Stephen. > > AMP probably is what I looking for. Now I can handle a much more > clients (2 times more without significantly increasing response time). > Server can also handle 3 times more clients without any errors, but > response time grow up. I check for netstat and find out Recv-Q about > 1706013 on client-side of AMP and Send-Q about 642288 on AMP > server-side. Is this meant: > - OS needs tuning (ulimit or ifconfig) > - or Twisted/Python can't handle so much amp-connections? > > How many connections are there? Are the queues that big for all of them? > CPU utilization < 10%, free memory about 500M from 1.7G. > > Is this the total system CPU load (how many cores/CPUs)? Johann ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] fast high load protocol
Hi, Johann Few words about load script: each "client" is a thread, which wait from 7 to 15 seconds and then make request to nginx, nginx proxy requests to api-server. On every request api-server makes 3 callRemote to memory-db. Delay between creating clients - 0.1 sec. Currently I use 1 amp-connection for api<=>memory-db. Response time dependency: clients < 1500 = api-response < 1s 1500 < clients < 3000 = api-response < 14s Recv-q/Send-q appears on 1500 clients line. API-server utilization on 3K clients = Recv-Q ~ 75...100 # top top - 17:38:47 up 222 days, 23:29, 2 users, load average: 1.05, 0.77, 0.34 Tasks: 45 total, 3 running, 42 sleeping, 0 stopped, 0 zombie Cpu(s): 37.1%us, 2.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 60.3%st Mem: 1747764k total, 574376k used, 1173388k free, 157632k buffers Swap: 917496k total,0k used, 917496k free, 273236k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 2492 root 25 0 151m 46m 3568 R 39.7 2.7 2:19.66 python Memory-db-server utilization on 3K clients == Send-Q ~ 40...45 # top top - 17:38:08 up 16 days, 22:07, 4 users, load average: 16.59, 9.40, 4.00 Tasks: 65 total, 6 running, 56 sleeping, 0 stopped, 3 zombie Cpu(s): 37.6%us, 5.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 56.4%st Mem: 1747764k total, 1308636k used, 439128k free, 281048k buffers Swap: 917496k total,0k used, 917496k free, 616676k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 31804 root 15 0 56596 51m 2820 R 25.7 3.0 16:44.27 python <- memory-db 30788 root 15 0 1622m 164m 2648 S 6.9 9.6 0:31.89 python <- load script 1449 root 25 0 000 Z 4.3 0.0 0:00.35 python I tried tests with 30/50/100 amp connections (with round-robin algorithm of choosing conn). Thats partially solve the problem: server can keeps response time 2-4s, but max number of clients going down with more amp connections. With 100 amp connections load script got 270 errors when achieve 2.7K clients (nginx reported about dropped connections because of timeout, my nginx proxy timeout settings 120 sec). Recv-Q still exists on most from 100 connections, but proportionally smaller. Note about environment: Amazon EC2, both servers are small instances -- 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 32-bit platform One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor OS - Ubuntu Hardy On Sun, Feb 21, 2010 at 8:18 PM, Johann Borck wrote: > Vlad Shevchenko wrote: >> Thanks a lot, Stephen. >> >> AMP probably is what I looking for. Now I can handle a much more >> clients (2 times more without significantly increasing response time). >> Server can also handle 3 times more clients without any errors, but >> response time grow up. I check for netstat and find out Recv-Q about >> 1706013 on client-side of AMP and Send-Q about 642288 on AMP >> server-side. Is this meant: >> - OS needs tuning (ulimit or ifconfig) >> - or Twisted/Python can't handle so much amp-connections? >> >> > How many connections are there? Are the queues that big for all of them? >> CPU utilization < 10%, free memory about 500M from 1.7G. >> >> > Is this the total system CPU load (how many cores/CPUs)? > > Johann > > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -- WBR, Vlad Shevchenko ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] multiprocessing capability?
Hi, I looked at the source for the threads module and wondered if the current Twisted supports Python's (2.6) multiprocessing threading? If not, is there a stable package somewhere that patches Twisted to support this? I saw one from last summer but not sure if its stable. Thanks! Darren ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] multiprocessing capability?
On Sun, Feb 21, 2010 at 1:43 PM, Darren Govoni wrote: > Hi, > I looked at the source for the threads module and wondered if the current > Twisted supports Python's (2.6) multiprocessing threading? > If not, is there a stable package somewhere that patches Twisted to support > this? I saw one from last summer but not sure if its stable. > > Hi Darren, I don't think there's any explicit support for multiprocessing, although I have seen some people using multiprocessing to run twisted in multiple processes. This doesn't answer your question, but you might be interested in ampoule as this provides a nice process protocol implemented on twisted or specifically twisted.protocols.amp: https://launchpad.net/ampoule -Drew ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Nevow question
> -Original Message- > From: twisted-python-boun...@twistedmatrix.com > [mailto:twisted-python-boun...@twistedmatrix.com] On Behalf Of Marco Giusti > Sent: Friday, 19 February 2010 10:12 PM > To: Twisted general discussion > Subject: Re: [Twisted-Python] Nevow question > > On Fri, Feb 19, 2010 at 03:17:07PM +1100, Sury Soni wrote: > > Hi, > > > > How do we disable traceback on Nevow production site? > > > > Right now, by default, any nevow site will print full length of > > traceback on web browser if there is some exception within the > > application. > [cut] > > something like this? > > http://divmod.org/trac/wiki/DivmodNevow/ErrorHandling Thank you Marco, this is exactly, what I was looking for. The example mentions ticket, http://divmod.org/trac/ticket/526 … … site = appserver.NevowSite(root) # You should be able to do this instead of in locateChild, but there seems to be a bug # http://divmod.org/trac/ticket/526 #site.remember(The404Page(), inevow.ICanHandleNotFound) #site.remember(The500Page(), inevow.ICanHandleException) … … There was a last post on above ticket 4 years ago. And this ticket is not closed. But seems to be working fine. I am using: twistd 8.2.0 nevow 0.9.33 python 2.6.3 Any potential bug should I be worried about, if I use site.remember stuff in production? Regards, Surya ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] multiprocessing capability?
Hello everyone, I have done something similar to this, but I used the children IO stream to control them. Maybe I should have done that using some higher level protocol, such as AMP or PB. (I think AMP is more robust than PB, though) The project that uses the children IO and process protocols is Lunch. See http://svn.sat.qc.ca/trac/lunch a 2010/2/21 Drew Smathers : > On Sun, Feb 21, 2010 at 1:43 PM, Darren Govoni wrote: >> >> Hi, >> I looked at the source for the threads module and wondered if the >> current Twisted supports Python's (2.6) multiprocessing threading? >> If not, is there a stable package somewhere that patches Twisted to >> support this? I saw one from last summer but not sure if its stable. >> > > Hi Darren, > I don't think there's any explicit support for multiprocessing, although I > have seen some people using multiprocessing to run twisted in multiple > processes. This doesn't answer your question, but you might be interested > in ampoule as this provides a nice process protocol implemented on twisted > or specifically twisted.protocols.amp: > https://launchpad.net/ampoule > -Drew > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > -- Alexandre Quessy http://alexandre.quessy.net/ ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Many connections and TIME_WAIT
I've been meaning to update this for a while - it turned out to be caused by a bug in my code. X-o Sorry guys - & thanks for helping me work it out. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] [ANNOUNCE] Twisted 10.0.0pre1 is now released
Live from PyCon Atlanta, I'm pleased to herald the approaching footsteps of the 10.0 release. Tarballs for the first Twisted 10.0.0 pre-release are now available at: http://people.canonical.com/~jml/Twisted/ This release is the first release ever with the new NEWS building system, which turns out to be utterly fantastic. We're also using this release to actually hammer out a release process. You can find the draft at: http://twistedmatrix.com/trac/wiki/ReleaseProcess Please feel free to update it with questions, thoughts, corrections and advice. Thanks, jml ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] [ANNOUNCE] Twisted 10.0.0pre1 is now released
I'm not opposed to Jinja2 as a templating engine, though Django would need to be the default: http://jinja.pocoo.org/2/ Jeff Bauer Rubicon, Inc. 615.297.3511 615.301.6533 fax On 02/21/2010 08:50 PM, Jonathan Lange wrote: > Live from PyCon Atlanta, I'm pleased to herald the approaching > footsteps of the 10.0 release. > > Tarballs for the first Twisted 10.0.0 pre-release are now available at: >http://people.canonical.com/~jml/Twisted/ > > This release is the first release ever with the new NEWS building > system, which turns out to be utterly fantastic. > > We're also using this release to actually hammer out a release > process. You can find the draft at: >http://twistedmatrix.com/trac/wiki/ReleaseProcess > > Please feel free to update it with questions, thoughts, corrections and > advice. > > Thanks, > jml > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] [ANNOUNCE] Twisted 10.0.0pre1 is now released
Hi Jeff, On Feb 21, 2010, at 9:57 PM, Jeff Bauer wrote: > I'm not opposed to Jinja2 as a templating engine, though > Django would need to be the default: This reply appears to have nothing to do with the release announcement. Wrong list, maybe? > http://jinja.pocoo.org/2/ > > Jeff Bauer > Rubicon, Inc. > 615.297.3511 > 615.301.6533 fax > > On 02/21/2010 08:50 PM, Jonathan Lange wrote: >> Live from PyCon Atlanta, I'm pleased to herald the approaching >> footsteps of the 10.0 release. ... -glyph ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] [ANNOUNCE] Twisted 10.0.0pre1 is now released
Sorry, wrong list (slippery fingers). Congrats on 10.0, BTW! Jeff Bauer Rubicon, Inc. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] [ANNOUNCE] Twisted 10.0.0pre1 is now released
On Feb 21, 2010, at 9:50 PM, Jonathan Lange wrote: > Live from PyCon Atlanta, I'm pleased to herald the approaching > footsteps of the 10.0 release. Woo! > Tarballs for the first Twisted 10.0.0 pre-release are now available at: > http://people.canonical.com/~jml/Twisted/ WOO! > This release is the first release ever with the new NEWS building > system, which turns out to be utterly fantastic. WOOO! > We're also using this release to actually hammer out a release > process. You can find the draft at: > http://twistedmatrix.com/trac/wiki/ReleaseProcess YYYEEAAHHH > Please feel free to update it with questions, thoughts, corrections and > advice. Can you make "a public website" a bit more specific? What kind of public website? (I realize that the answer is probably "any arbitrary public website, not twistedmatrix.com", but it would be good to say that.) ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] [ANNOUNCE] Twisted 10.0.0pre1 is now released
On Sun, Feb 21, 2010 at 10:15 PM, Glyph Lefkowitz wrote: > On Feb 21, 2010, at 9:50 PM, Jonathan Lange wrote: > ... >> Please feel free to update it with questions, thoughts, corrections and >> advice. > > Can you make "a public website" a bit more specific? What kind of public > website? (I realize that the answer is probably "any arbitrary public > website, not twistedmatrix.com", but it would be good to say that.) I'll say that for now. Ideally, I'd like there to be a standard place to upload the pre-release tarballs. jml ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] [ANNOUNCE] Twisted 10.0.0pre1 is now released
Hi, This is great! Is there a list of new features or improvements for this release? I'd like to start planning my update. Darren On Sun, 2010-02-21 at 21:50 -0500, Jonathan Lange wrote: > Live from PyCon Atlanta, I'm pleased to herald the approaching > footsteps of the 10.0 release. > > Tarballs for the first Twisted 10.0.0 pre-release are now available at: > http://people.canonical.com/~jml/Twisted/ > > This release is the first release ever with the new NEWS building > system, which turns out to be utterly fantastic. > > We're also using this release to actually hammer out a release > process. You can find the draft at: > http://twistedmatrix.com/trac/wiki/ReleaseProcess > > Please feel free to update it with questions, thoughts, corrections and > advice. > > Thanks, > jml > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] [ANNOUNCE] Twisted 10.0.0pre1 is now released
On Sun, Feb 21, 2010 at 10:57 PM, Darren Govoni wrote: > Hi, > This is great! > > Is there a list of new features or improvements for this release? I'd like > to start planning my update. > Each of the tarballs has a NEWS file. jml ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Try/catching yielded Exceptions and getting proper tracebacks
I'm frequently using the inlineCallbacks idiom for writing code, but am
often finding myself frustrated with rather counterintuitive tracebacks.
Let's say I'm writing an XML-RPC handler, and even if an exception
occurs, I want it to return something to the user and dump the exception
to the log file. However, if an exception occurs when I'm using the
inlineCallbacks decorator, I sometimes get lacking tracebacks. Maybe
I'll have the line number of the function that yielded the exception,
but I'll have no detailed information as to where the exception really
came from.
For example, in the example I've pasted at the end of this mail, I get a
traceback as follows:
Traceback (most recent call last):
File "exceptions.py", line 13, in xmlrpc_dosomething
result = yield self._do_something_else()
Exception: die die die!!!
However, the exception happens within _do_something_else on line 20.
I have a simple complete example copy/pasted below. Is there a better
way to accomplish this?
Thanks.
- Paul Goins
---
from twisted.web.xmlrpc import XMLRPC
from twisted.web.server import Site
from twisted.web.resource import Resource
from twisted.internet import reactor, defer
import sys, traceback
class MyApp(XMLRPC):
@defer.inlineCallbacks
def xmlrpc_dosomething(self):
try:
result = yield self._do_something_else()
defer.returnValue(result)
except Exception:
print >> sys.stderr, traceback.format_exc()
@defer.inlineCallbacks
def _do_something_else(self):
yield
raise Exception("die die die!!!")
@defer.inlineCallbacks
def cause_trouble(app):
val = yield app.xmlrpc_dosomething()
print val
reactor.stop()
if __name__ == "__main__":
app = MyApp()
r = Resource()
r.putChild("RPC2", app)
s = Site(r)
reactor.listenTCP(8123, s)
reactor.callLater(0.2, cause_trouble, app)
reactor.run()
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
