On Jul 8, 2013, at 3:31 PM, Justin Chiu wrote: > Hi all, > > *I posted this to python-list and tutor-list and received no replies. Any > advice would be much appreciated. Thank you.* > > What is the best approach to writing a concurrent daemon that can execute > callbacks for different types of events (AMQP messages, parsed output of a > subprocess, HTTP requests)? > > I am considering [twisted][1], the built-in [threading][2] module, and > [greenlet][3]. I must admit that I am very unfamiliar with concurrent > programming and Python programming in general (formerly a data analysis > driven procedural programmer). Any resources on threaded/concurrent > programming (specifically daemons...not just multi-threading a single task) > would be much appreciated. >
From the details below, it would appear that Twisted would be a great fit for this application. The asynchronous nature of the framework may cause some head scratching if you're in a procedural mindset, but on the other hand, attempting to facilitate all of the different service interactions you are proposing in a coherent manner can be challenging regardless. One of Twisted's strengths is that it provides that coherence...once you grok the paradigm, you'll be able to look at this type of application and quickly come up with a reasonable sketch of how all of the pieces will fit together, with confidence. I highly recommend Dave Peticolas' tutorial: http://krondo.com/?page_id=1327 You'll get a great introduction to asynchronous programming and dive into Twisted. And of course the Twisted documentation, esp. the Getting Started section of http://twistedmatrix.com/documents/current/core/howto/. Finally, I'd stress that you don't need to know *all* of Twisted to benefit from using it. There is a great deal of functionality in the framework and I'm discovering new uses for it all of the time. > Details: > > 1) Listens into AMQP messaging queues and executes callbacks when messages > arrive. > Example: Immediately after startup, the daemon continuously listens to the > [Openstack Notifications messaging queue][4]. When a virtual machine is > launched, a notification is generated by Openstack with the hostname, IP > address, etc. The daemon should read this message and write some info to a > log (or POST the info to a server, or notify the user...something simple). > Others have mentioned txAMQP. It should be fairly straightforward to consume messages from the queue and hook it up to your user interface. There are numerous asynchronous APIs in the framework, so you'll have options for doing things just as you propose, without blocking other parts of your application. Twisted has its own logging system and twisted.web.client.Agent can be used to POST to a web server, etc. > 2) Parse the output of a subprocess and execute callbacks based on the output. > Example: Every 30 seconds, a system command "[qstat][5]" is run to query a > job resource manager (e.g. TORQUE). Similar callbacks to 1). Twisted can help you with scheduled tasks and there is a ProcessProtocol that, combined, will do exactly what you propose. http://twistedmatrix.com/documents/current/core/howto/time.html http://twistedmatrix.com/documents/current/core/howto/process.html > 3) Receive requests from a user and process them. I think this will be via > WSGI HTTP. > Example: User submits an XML template with virtual machine templates. The > daemon does some simple XML parsing and writes a job script for the job > resource manager. The job is submitted to the resource manager and the daemon > continually checks for the status of the job with "qstat" and for messages > from AMQP. It should return "live" feedback to the user and write to a log. > Twisted has a built-in programmable web server, and a WSGI container is available. http://twistedmatrix.com/documents/current/web/howto/web-in-60/index.html is a good starter for web programming with Twisted, as is the general web howto: http://twistedmatrix.com/documents/current/web/howto/using-twistedweb.html There are many ways to construct what you describe, so it's probably not appropriate to give specific recommendations other than to say what you want is absolutely feasible. It appears you've already got a good handle on what you want to achieve, which should make it easier when it comes to seeking more specific help, etc. from the mailing list once you get your feet wet. Lucas
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
