[Twisted-Python] stopping LoopCall and check if running
I have a question on LoopCall.
I have a process that uses a sequence of LoopCalls, once one finishes
I call the next to check for the next critical event to continue:
e.g.
def checkForAvailableProfile(self):
profile = fs_profiles_manager.get_next_available_profile()
if profile is None:
return
else:
self.checkForAvailableProfileLoop.stop()
self.profile = profile
logging.debug("using profile: %s", self.profile['name'])
self.ff = FirefoxProcess(self.profile, self.params)
self.ff.run()
self.checkForEventsFileLoop =
LoopingCall(self.checkIfEventsFileExists)
self.checkForEventsFileLoop.start(0.5)
self.state = self.STATE_WAIT_FOR_EVENTS
The problem is that I'm getting errors, errors when I try to call stop
when stop perhaps has already been called.
I want advice on how to manage LoopCalls. So far I'm using state variables.
e.g.
self.state = self.STATE_WAIT_FOR_EVENTS
if something goes haywire or if the process is done I have a function
that cleans up all the LoopCalls so it's not checking constantly:
def cleanExit(self):
if self.state == self.STATE_WAIT_FOR_PROFILE:
self.checkForAvailableProfileLoop.stop()
elif self.state == self.STATE_WAIT_FOR_EVENTS:
self.checkForEventsFileLoop.stop()
elif self.state == self.STATE_WAIT_FOR_PROC_KILLED:
self.checkIfFirefoxWasKilledLoop.stop()
self.ff.cleanExit()
the above code is what I believe is causing my exceptions, but how
else do I check for the LoopCalls?
=
2011-08-31 12:30:11-0700 [FiresharkProtocol,1278,127.0.0.1] Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.6/dist-packages/twisted/application/app.py",
line 348, in runReactorWithLogging
reactor.run()
File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py",
line 1170, in run
self.mainLoop()
File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py",
line 1182, in mainLoop
self.doIteration(t)
File
"/usr/lib/python2.6/dist-packages/twisted/internet/selectreactor.py",
line 140, in doSelect
_logrun(selectable, _drdw, selectable, method, dict)
--- ---
File "/usr/lib/python2.6/dist-packages/twisted/python/log.py",
line 84, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File "/usr/lib/python2.6/dist-packages/twisted/python/log.py",
line 69, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/usr/lib/python2.6/dist-packages/twisted/python/context.py",
line 59, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python2.6/dist-packages/twisted/python/context.py",
line 37, in callWithContext
return func(*args,**kw)
File
"/usr/lib/python2.6/dist-packages/twisted/internet/selectreactor.py",
line 156, in _doReadOrWrite
self._disconnectSelectable(selectable, why, method=="doRead")
File "/usr/lib/python2.6/dist-packages/twisted/internet/posixbase.py",
line 194, in _disconnectSelectable
selectable.connectionLost(f)
File "/usr/lib/python2.6/dist-packages/twisted/internet/tcp.py",
line 519, in connectionLost
protocol.connectionLost(reason)
File "/usr/sbin/fireshark.py", line 101, in connectionLost
self.cleanExit()
File "/usr/sbin/fireshark.py", line 43, in cleanExit
self.checkIfFirefoxWasKilledLoop.stop()
File "/usr/lib/python2.6/dist-packages/twisted/internet/task.py",
line 171, in stop
assert self.running, ("Tried to stop a LoopingCall that was "
exceptions.AssertionError: Tried to stop a LoopingCall that was not
running.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] How can LoopCall not be defined?
I'm running a server with about 20 client connections going to it right now and monitoring the error log eventually I see LoopCall is not defined. how can that be? Thanks in advance. 2011-09-07 22:47:26-0700 [-] Unhandled Error Traceback (most recent call last): File "/usr/lib/python2.6/dist-packages/twisted/application/app.py", line 445, in startReactor self.config, oldstdout, oldstderr, self.profiler, reactor) File "/usr/lib/python2.6/dist-packages/twisted/application/app.py", line 348, in runReactorWithLogging reactor.run() File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1170, in run self.mainLoop() File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1179, in mainLoop self.runUntilCurrent() --- --- File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 778, in runUntilCurrent call.func(*call.args, **call.kw) File "/usr/sbin/firefoxprocess.py", line 48, in onTimer self.firesharkProtocol.handleProcessTimedOut() File "/usr/sbin/fireshark.py", line 32, in handleProcessTimedOut self.dirtyExit(); File "/usr/sbin/fireshark.py", line 68, in dirtyExit self.checkIfFirefoxWasKilledLoop = LoopCall(self.checkIfFirefoxWasKilled) exceptions.NameError: global name 'LoopCall' is not defined ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] How can LoopCall not be defined?
sorry for wasting everyone's time.
script worked, made a change and somehow changed the one line from
LoopingCall to LoopCall, I should have noticed that, but I've been
coding for a while and just missed. Thanks.
changed now and works
if self.checkIfFirefoxWasKilledLoop.running:
logging.debug("FiresharkProtocol.checkIfFirefoxWasKilled is running.")
else:
logging.debug("FiresharkProtocol.checkIfFirefoxWasKilled is NOT
running. starting..\
.")
self.checkIfFirefoxWasKilledLoop =
LoopingCall(self.checkIfFirefoxWasKilled)
self.checkIfFirefoxWasKilledLoop.start(0.5, now=True)
except AttributeError:
logging.debug("FiresharkProtocol.checkIfFirefoxWasKilled exist but is
NOT running. starting\
...")
self.checkIfFirefoxWasKilledLoop =
LoopingCall(self.checkIfFirefoxWasKilled)
self.checkIfFirefoxWasKilledLoop.start(0.5, now=True)
try:
On Wed, Sep 7, 2011 at 11:34 PM, Laurens Van Houtven <_...@lvh.cc> wrote:
> I've never heard of LoopCall. Maybe you meant LoopingCall?
>
> cheers
> lvh
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] feedback on next version of Fireshark using twisted framework....
Hi I have an open source project that is pretty well-known in the security community. http://www.fireshark.org it's a service wrapper that runs a headless version of firefox to visit malicious sites to store telemetry data. The old service used PERL for threading (very painful), so a while ago I changed everything to python using the twisted framework. the service's job is to launch a choose a firefox profile, launch the process and handle closes and crashes approprietly. the launch of version 2.5 is next week but I sure could use some feedback on the service code before releasing it. Keep in mind I haven't finishing commenting for public release. here are the files for the new service: 1) fireshark.py http://pastebin.mozilla.org/1614427 this is the main service. opens up a port to listen on, and uses the following two classes to manage profiles and launch a firefox process. 2) firesharkprofilemanager.py http://pastebin.mozilla.org/1614428 this is the firefox profile manager, it manages what firefox profiles are available. 3) firefoxprocess.py http://pastebin.mozilla.org/1614431 this is the firefox process class, to check if the launched process has been killed as well, as check if the logging from the internal plugin (another part of the fireshark project) has completed logging. Thanks in advance! Twisted has been a great Framework! Stephan ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] feedback on next version of Fireshark using twisted framework....
thanks for the feed back Jasper! More feedback welcome. Yes the code def. needs to be cleaned up before public release, and I will make more use of python libs for path concatenation. The feedback I'm specifically looking for is anything related to how I'm using twisted and process management. The reason I have everything in there from timing out, to killing a process, is because I'm dealing with malicious websites, so bad things happen and so I need to take extreme measures to for resource monitoring. Any suggestions on how I can do things better with the service in terms of how i'm using Twisted is welcome! Jasper provided great comments and I'll be working to make those changes. Thanks, Stephan On Thu, May 3, 2012 at 11:12 AM, Jasper St. Pierre wrote: > You seem to have some indentation issues. You have a random assortment > of Python style issues (some lines end with a semicolon). You use > 'foo' + '/' + 'bar' to generate paths. While not evil (I think all > major platforms will support that and do the translation on their > own), you're better off using os.path.join. The "~/.mozilla/firefox" > won't work on certain platforms (and you could just use > os.path.expanduser if you don't want to support those platforms). > > Your process spawning code is ugly, and could be open to injection if > not handled properly. Do it the better way: > >reactor.spawnProcess(self.processProtocol, 'xfvb-run', > ['xvfv-run', '--auto-servernum', 'firefox', '-p', profile_id], env = > os.environ, usePTY=1) > > Don't do 'foo == True'. Please use new-style classes (inherit from > 'object'). I don't know why the timeout stuff makes me a bit scared, > but it does. I don't really think timeouts are necessary here. > > I don't think you should be killing a process with os.kill. I'm quite > sure whatever you're trying to do has a better Twisted idiom. > > Google for PEP 8, ignore the ones that don't make sense (like foo_case > instead of camelCase in Twisted code), and try to follow the rest. > When using Twisted code, always look for things in Twisted that might > combat the standard library, and try to use Twisted's whenever > possible. > > On Thu, May 3, 2012 at 11:56 AM, Stephan wrote: > > > > Hi I have an open source project that is pretty well-known in the > security > > community. http://www.fireshark.org > > it's a service wrapper that runs a headless version of firefox to visit > > malicious sites to store telemetry data. > > > > The old service used PERL for threading (very painful), so a while ago I > > changed everything to python using > > the twisted framework. the service's job is to launch a choose a firefox > > profile, launch the process and > > handle closes and crashes approprietly. > > > > the launch of version 2.5 is next week but I sure could use some > feedback on > > the service code before releasing it. > > Keep in mind I haven't finishing commenting for public release. > > > > here are the files for the new service: > > > > 1) > > fireshark.py http://pastebin.mozilla.org/1614427 > > > > this is the main service. opens up a port to listen on, and uses the > > following two classes to manage profiles and launch > > a firefox process. > > > > 2) > > firesharkprofilemanager.py http://pastebin.mozilla.org/1614428 > > > > this is the firefox profile manager, it manages what firefox profiles are > > available. > > > > 3) > > firefoxprocess.py http://pastebin.mozilla.org/1614431 > > > > this is the firefox process class, to check if the launched process has > > been killed as well, as check if the logging from > > the internal plugin (another part of the fireshark project) has completed > > logging. > > > > Thanks in advance! Twisted has been a great Framework! > > Stephan > > > > ___ > > Twisted-Python mailing list > > Twisted-Python@twistedmatrix.com > > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > > > > -- > Jasper > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] how to daemonize twisted...and detatch from shell process
hey Jean-Paul, I'm using twistd - if you see the daemon code it's in there to start and manage the service. For some reason the OC Daemon pastebin linkted didn't work but this one does: OC Daemon that starts the OC Service http://pastebin.com/g0UE7An8 still know don't know what to change to get the service to run and detach from the parent process. Any help is appreciated. Stephan On Sat, Oct 6, 2012 at 6:56 PM, wrote: > On 6 Oct, 10:30 pm, schene...@gmail.com wrote: > >Hi, > > > >I'm attempting to create an ubuntu service that when started will run > >in > >the background processing beanstalk jobs. > >I've taken the code from beanstalk-client twisted to act as my > >beanstalk > >client ( > >https://github.com/dustin/beanstalk-client- > >twisted/blob/master/beanstalk.py) > > > >The issue I'm having is how do I set up my service (ocapi.py) so that > >when > >the daemon (ocapi) starts it detaches from the process, which isn't' > >happening right now. > > Use twistd, which comes with daemonization features. From the looks of > your code, you're halfway there already. > > http://twistedmatrix.com/documents/current/core/howto/basics.html#auto1 > http://twistedmatrix.com/documents/current/core/howto/application.html > > Jean-Paul > >ocapi.py (service) <--this is what should detach and I put it in > >/usr/sbin/ocapi.py code found at http://pastebin.com/29VnXnrm > >ocapi (daemon) <-- goes int /etc/init.d/ocapi code found at > >http://pastebin.com/0QgRfTfu > > > >stephan@oc:~$ sudo /etc/init.d/ocapi start > >* Starting ocapi... Removing stale pidfile /var/run/ocapi.pid > >Connected! > > > >...problem is I don't get my shell prompt back when i start the > >service, > >meaning it didn't detach from the parent process...hopefully someone > >can > >shed some light on this. > >Thanks, > >Stephan > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
