[Twisted-Python] how to create state machines?
Hi all, I'm still getting my 'sea legs' with Twisted, so this may be a novice question, but I'm wondering what would be a good way to create a state machine with Twisted? For instance I've written quite a bit of code in different languages that creates co-operative, context switching schemes using state machines. Here would be a primative example in psuedocode: Switch(state): Case INIT: Initialize() state = CMD1 Case CMD1: CommandOne() state = WAITCMD1 Case WAITCMD1: If CommandOneDone() state = DONE Else state = WAITCMD1 Case DONE: state = EXIT This state machine initializes a system, issues an asynchronous command that takes time to complete (say a motor move command) and then waits for that command to be done before exiting. In the context of a framework that is calling this state machine, the WAITCMD1 is executed repeatedly (polling) while the asynchronous command completes. A system can be constructed with lots of little state machines like this and be made to look like it is doing many things at once even though the execution of the program is single threaded. I understand (pretty much) the Twisted framework is like this and implmenting event handlers like connectionMade(), etc., is a state machine, but I'm wondering how to implement what I've outlined above in one of Twisted's state event handlers, or inside something like callLater(). For example, let's say I want to use Twisted to run a long running daemon process that has an XMLRPC interface. That XMLRPC interface is an interface to a state machine inside Twisted that allows the caller to change state, or start new state machines, or get the status of a running state machine. In that case I'm thinking I want a state machine the runs continuously in the Twisted loop, structured like the example above; co-operatively yielding back to Twisted, but running non-stop. Something like callLater(1, stateMachine), but non-stop, without even the 1 second call loop. Is this something I can do with callLater()? Would using a deferred be a better option? I see deferred's callBack() system as a connecting states together, but I'm not sure how a WAITONCMD1 kind of thing would work, where the deferred callBack(d, waitTillDone) would have to keep coming back to itself till some condition is met? Thanks in advance for your help, Doug ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] RE: how to create state machines?
Hi Andrew, > > Hi Doug: > > >This state machine initializes a system, issues an asynchronous > command > >that takes time to complete (say a motor move command) and then waits > >for that command to be done before exiting. In the context of a > >framework that is calling this state machine, the WAITCMD1 is executed > >repeatedly (polling) while the asynchronous command completes. A > system > >can be constructed with lots of little state machines like this and be > >made to look like it is doing many things at once even though the > >execution of the program is single threaded. > > >I understand (pretty much) the Twisted framework is like this and > >implmenting event handlers like connectionMade(), etc., is a state > >machine, but I'm wondering how to implement what I've outlined above > in > >one of Twisted's state event handlers, or inside something like > >callLater(). For example, let's say I want to use Twisted to run a > long > >running daemon process that has an XMLRPC interface. That XMLRPC > >interface is an interface to a state machine inside Twisted that > allows > >the caller to change state, or start new state machines, or get the > >status of a running state machine. In that case I'm thinking I want a > >state machine the runs continuously in the Twisted loop, structured > like > >the example above; co-operatively yielding back to Twisted, but > running > >non-stop. Something like callLater(1, stateMachine), but non-stop, > >without even the 1 second call loop. > > If I understand your correctly, I don't think you need to implement a > state machine to simulate concurrency with Twisted - Twisted does a lot > of that for you. You can think of a Twisted application as a state > machine - the callback being the state and the completion of the > operation and the calling of the callback is the transition. These > callbacks at runtime act like a thread of execution. > > def Initialize(...): > # do something > deferred = someFunctionThatReturnsADeferred() > deferred.addCallback(State2) > > def State2(...): > # do something > deferred = someFunctionThatReturnsADeferred() > deferred.addCallback(State3) > > def State3(someData): > # do something > if someData == 'State4': >deferred = someFunctionThatReturnsADeferred() >func = State4 > elif someData == 'State5': >deferred = someOtherFunctionThatReturnsADeferred() >func = State5 > ... > deferred.addCallback(func) > > if __name__ = "__main__": >initialize() >reactor.run() > > As for the Twisted loop. Well, you don't really see the Twisted loop > since that is hidden in the reactor. Also you should distinguish > between writing a new protocol and using an existing one. > > In the case of XMLRPC, creating the server isn't the problem. > > http://twistedmatrix.com/projects/web/documentation/howto/xmlrpc.html > > Once a XMLRPC server is created, Twisted will take responsibility for > creating new instances (or threads if you want to see it that way). If > you still need a state machine, then the only hiccup I can see is > sharing state machine (if you really need one) between XMLRPC method > invocations. > > Cheers, > Andrew > [Doug Farrell] Thank you very much for your detailed response, I'm kinda getting it and am going to try out a prototype to make sure I do. As you mention, Twisted does take care of a lot of the issues dealing with concurrency. Having written a few threaded applications, I didn't really want to get into starting my state machine in a thread and then having to deal with all the cross thread data safety issues. One thing I'm still a little confused by in your reply is how you're getting the deferred instance. Is there a particular reason your example does this: deferred = someOtherFunctionThatReturnsADeferred() rather than this: deferred = defer.deferred() I just want to understand if there a reason the deferred is being created in the other function. Again, thanks for your help and response! Doug ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Question about deferreds
Hi all, I consider myself a Twisted "newbie", so this might seem like a simple couple of questions, hope you'll indulge me. When I first began experimenting with deferreds I was breaking up tasks into sequences of steps something like this: d = Deferred() d.succeed(True) d.addCallback(step1) d.addCallback(step2) d.addCallback(step3) etc. When this runs I was under the mistaken impression that at the end of each callback (step1, step2, step3, etc) the deferred would yield back to the main loop (reactor) and allow other deferreds to run. Now I know that the callback functions run sequentially one after another with no break. So my first question is this: what is the advantage of this over just combining the callback functions into one big function?, something like this: d = Deferred() d.succeed(True) d.addCallback(OneBigStep) If step1(), step2() and step3() take a fair bit of time to complete, doesn't this effectively block the reactor loop for the entire exection of all the steps? My second question is related to this; what is a recommended way to create the cooperative state machine I'm thinking of using deferreds? For example if I create something like this: d1 = Deferred() d1.succeed(True) d1.addCallback(d1_step1) d1.addCallback(d1_step2) d1.addCallback(d1_step3) d2 = Deferred() d2.succeed(True) d2.addCallback(d2_step1) d2.addCallback(d2_step2) d2.addCallback(d2_step3) How do I get the execution of this to be this: 1) d1_step1 2) d2_step1 3) d1_step2 4) d2_step2 5) d1_step3 6) d2_step3 I want to have the state machines I create cooperatively 'multi-task', yielding control back to the main loop at each state transition. Thanks in advance for any help, guidance or references you can provide. I really enjoy Twisted, I'm still working on getting the 'Zen' of it. Doug ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Question about deferreds
Glyph, Thank you very much for your response, it has provided me with material that will help me take a big step in my Twisted education. > > Deferreds are not about breaking your work up into steps to save time. > They're about breaking it up so that different systems can deal with > the > same asynchronous event in their own way. > > For example, let's say you're working on a REST client that deals with > invoices. The server gives you data at a particular URL and you want > to > convert it into an "Invoice" instance. > (snip) This whole section of your answer really helped me get a handle on a deferred, well beyond the "promise that a function will have a result later". J I was thinking kind of monolithically about how to use deferreds, in that if I was writing everything why would I break it up? However, the real case is more along the lines of what you said in your answer; more than likely the eventual total behavior is made up of a string of smaller behaviors 'glued' together by the deferred. And these other behaviors would be code generated by others in my group or libraries. (snip) > > If you actually wanted to interleave Deferred callbacks like that it > would be tricky. But if you just want interleaved callbacks, use > Cooperator as I mentioned above :). > This reference to the Cooperator really helped me out, I just didn't know what to look for in the Twisted system to do what I wanted with co-operating state machines. I did build a little prototype using deferreds that does provide the behavior I was after, but it was awkward at best to set up. I felt like there should be something more elegant, and the Cooperator looks like exactly what I need. I'm going to try and set up a prototype to experiment with so I understand it more. Again, thanks for your reply and help! Doug ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] twisted.internet.task.LoopingCall
Hi all, I'm using the twisted.internet.task.LoopingCall system to run periodic tasks in a Twisted server. If I had code like this: from twisted.internet.task import LoopingCall lp = LoopingCall(someFunction) lp.start(5.0) # run every 5 seconds Is there anything in LoopingCall to keep it from trying to run someFunction() if the previous call is still running? For instance a call to someFunction() takes longer than 5 seconds, will LoopingCall hold off or will it call someFuction() anyway causing two 'instances' to run? Thanks, Doug ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted-Python Digest, Vol 63, Issue 7
Terry > Hi Doug > > Sorry if it sounds like I'm being a PITA, but I really recommend > reading > the source code and writing & playing with a tiny example. Also see > http://twistedmatrix.com/documents/8.2.0/api/twisted.internet.task.Loop > ingCall.html > > I suggest this because it's by far the best way to learn these things. > And > if you're using Twisted in any serious way, it's hard to get by without > getting closely acquainted with some of the code. > > BTW, yes, LoopingCall will reschedule in a sensible way, supposing your > task returns a deferred that takes a long time to fire :-) Try it. > > Terry > [Doug Farrell] Don't mind the PITA at all, I actuall did read the source code and just didn't 'get it' so much. In my case the function that LoopingCall was passed was actually doing the work, not returning a Deferred. However, I'm going to look into doing that. Thanks for your feedback and advice, Doug > per > http://twistedmatrix.com/documents/current/api/twisted.internet.task.Lo > opingCall.html > > If someFunction returns a Deferred, it will not be rescheduled if the > Deferred has not fired. > > e.g. > > def someFunction(): > d = someDeferredReturningOperation() > d.addCallbacks(success, error) > return d > > lp = LoopingCall(someFunction) > lp.start(5.0) # run every 5 seconds > > > Lucas > [Doug Farrell] Lucas, Thanks for your response and feedback. In my sample code the someFunction() code was actually doing the work I wanted, not creating a deferred as your sample shows. However, I'm going to try that out and give it a run. Thanks! Doug ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Twisted synchronous and asynchronous db connections
Hi all, Our projects at work include synchronous applications (short lived) and asynchronous Twisted applications (long lived). We're re-factoring our database and are going to build an API module to decouple all of the SQL in that module. I'd like to create that API so both synchronous and asynchronous applications can use it. For the synchronous applications I'd like calls to the database API to just return data (blocking) just like using MySQLdb, but for the asynchronous applications I'd like calls to the same API functions/methods to be non-blocking, probably returning a deferred. Anyone have any hints, suggestions or help they might offer me to do this? Thanks in advance, Doug ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted synchronous and asynchronous db connections
Jean-Paul, Thanks for the link, I'm looking at it right now. Doug On Mon, Nov 9, 2009 at 10:43 PM, wrote: > On 03:24 am, doug.farr...@gmail.com wrote: >>Hi all, >> >>Our projects at work include synchronous applications (short lived) >>and asynchronous Twisted applications (long lived). We're re-factoring >>our database and are going to build an API module to decouple all of >>the SQL in that module. I'd like to create that API so both >>synchronous and asynchronous applications can use it. For the >>synchronous applications I'd like calls to the database API to just >>return data (blocking) just like using MySQLdb, but for the >>asynchronous applications I'd like calls to the same API >>functions/methods to be non-blocking, probably returning a deferred. >>Anyone have any hints, suggestions or help they might offer me to do >>this? > > Chris Armstrong wrote a synchronous implementation of Deferred. That > might be an approach to investigate: > > http://radix.twistedmatrix.com/2007/12/supporting-both-asynchronous- > and.html > > Jean-Paul > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] synchronous/asynchronous api: possible interface
Hi all,
I'd like to get some comments on the code below (including, "don't be
a bonehead!" ). I'm trying to write a API library for database
access that can be used by both synchronous (non-twisted, no reactor)
code and asynchronous (twisted, with reactor) code where the methods
of the library can called in either environment. What I'd like is the
method to return the results of the call (database query results) when
in synchronous mode, and a deferred that is a result of the database
call in asynchronous mode.
What I've done below is create a decorator class called CallerMode
that is used to decorate a function/method that normally returns a
deferred. If the CallerMode class is in synchronous mode it starts a
reactor to get the default callbacks to get called, which gets the
results or exceptions, and stops the reactor.
Am I being a dunce doing this kind of thing? Any suggestions,
comments, etc. are welcome.
Thanks!
Doug
import sys
from functools import wraps
from twisted.internet import reactor
from twisted.python import log
from twisted.web.client import getPage
class CallerMode(object):
'''This is a decorator class to make calls to an api play nice
in synchronous or asynchronous mode'''
ASYNCH = 'asynch'
SYNCH = 'synch'
STATE = SYNCH
def __init__(self):
self.retval = None
def __call__(self, f):
'''This provides the decorator function wrapper, which will return
a deferred for asynchronous mode and the results of the wrapped
function in synchronous mode'''
def CB(result):
self.retval = result
reactor.stop()
def EB(failure):
reactor.stop()
failure.raiseException()
@wraps(f)
def func(*args, **kwargs):
d = f(*args, **kwargs)
if CallerMode.STATE == CallerMode.SYNCH:
d.addCallback(CB).addErrback(EB)
reactor.run()
return self.retval if CallerMode.STATE == CallerMode.SYNCH else d
return func
class Library(object):
'''This is just a class to provide an API to call for data,
in this case just using getPage() to get a deferred that gets some
data, vs. having to set up a database for this test program
'''
def __init__(self):
log.msg('MyLib has been initialized')
self._retval = None
@CallerMode()
def page(self):
return getPage('http://www.google.com')
def doSomethingWithData(data):
log.msg(data)
log.FileLogObserver.timeFormat = '%Y-%m-%d %H:%M:%S'
log.startLogging(sys.stdout)
lib = Library()
#CallerMode.STATE = CallerMode.ASYNCH
CallerMode.STATE = CallerMode.SYNCH
# call the library API method
result = lib.page()
# depending on the CallerMode, handle the results differently
if CallerMode.STATE == CallerMode.SYNCH:
doSomethingWithData(result)
elif CallerMode.STATE == CallerMode.ASYNCH:
result.addCallback(doSomethingWithData)
reactor.run()
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] SQLAlchemy and Twisted
Hi all, I've been doing some searching about how to get SQLAlchemy and Twisted working together in a Twisted application. Though I've found a lot of information, I haven't seen (or figured out) a good working solution or definitive answer. The most promising one I've run across concerns running the SQLAlchemy queries in a separate process (rather than a separate thread) and communicating the queries between the Twisted application in one process and the SQLAlchemy application in another. Does anyone have any good pointers, suggestions, ideas, links to how I might go about setting something like this up? Here's a couple questions that come to mind: 1) Would the SQLAlchemy process also be a Twisted application with all the queries running as deferreds in the main thread, and blocking? 2) How would the Twisted process communicate with the SQLAlchemy process, using something like XMLRPC, calling methods to perform the queries? Or would the XMLRPC methods convey something more generic like SQL? Thanks in advance for any help! Doug ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
