[Twisted-Python] Persisted protocol?

[anatoly techtonik]

Hi again,


>From http://twistedmatrix.com/documents/current/core/howto/udp.html

"""As you can see, the protocol is registered with the reactor. This means it
may be persisted if it's added to an application..."""

What does 'persisted` here mean?

--
anatoly t.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Persisted protocol?

[Stephen Thorne]

On 2011-04-06, anatoly techtonik wrote:
> From http://twistedmatrix.com/documents/current/core/howto/udp.html
> 
> """As you can see, the protocol is registered with the reactor. This means it
> may be persisted if it's added to an application..."""
> 
> What does 'persisted` here mean?

It sounds like this is a rememant of when the reactor shutdown it used
to serialise its state so it could be restarted later.

-- 
Regards,
Stephen Thorne
Development Engineer
Netbox Blue

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Persisted protocol?

[Tim Allen]

On Wed, Apr 06, 2011 at 10:16:03AM +0300, anatoly techtonik wrote:
> From http://twistedmatrix.com/documents/current/core/howto/udp.html
> 
> """As you can see, the protocol is registered with the reactor. This means it
> may be persisted if it's added to an application..."""
> 
> What does 'persisted` here mean?

At some point in the past, somebody decided it would be a great idea to
have Twisted's "twistd" start applications from Python pickles; you
could start up a server, configure it interactively, and when you shut
it down the entire server state would be persisted in a pickle-file;
when you started it up it would resume exactly as it had been when you
shut it down.

The idea seems to have fallen from grace, possibly because restarting
a process is often the simplest way to resolve a production problem.
These days, new Twisted users are encouraged to use Python ".tac" files
rather than ".tap" pickles, and I expect that sentence is just a relic
of that bygone era.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Instrumenting Reactors

[Paul Thomas]

I have a reactor which is getting busier over time and I'd like to find out 
where the cycles are going. Using the profiler isn't really practical on a 
server running for days, so I'd like to instrument the reactor to show me which 
I/O events are taking the longest to deal with. 

Should I just hack into the reactor somewhere? Or is there something sitting in 
a library I haven't seen that will help with this?

Paul. 

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Instrumenting Reactors

[Laurens Van Houtven]

Hi!


No guarantee it's an optimal solution, but when I wanted to inspect a
running process (which sounds like what you're doing), I had a pretty good
time with manhole. (If you're not familiar with that: it's in Twisted Conch,
and it's basically just a way to SSH in to a running box and get a REPL).

In that REPL, you can of course do things like inspect the running reactor
:-)


cheers
lvh

On Wed, Apr 6, 2011 at 11:55 AM, Paul Thomas
wrote:

> I have a reactor which is getting busier over time and I'd like to find out
> where the cycles are going. Using the profiler isn't really practical on a
> server running for days, so I'd like to instrument the reactor to show me
> which I/O events are taking the longest to deal with.
>
> Should I just hack into the reactor somewhere? Or is there something
> sitting in a library I haven't seen that will help with this?
>
> Paul.
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>



-- 
cheers
lvh
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Persisted protocol?

[Laurens Van Houtven]

On Wed, Apr 6, 2011 at 9:30 AM, Tim Allen  wrote:

> On Wed, Apr 06, 2011 at 10:16:03AM +0300, anatoly techtonik wrote:
> > From http://twistedmatrix.com/documents/current/core/howto/udp.html
> >
> > """As you can see, the protocol is registered with the reactor. This
> means it
> > may be persisted if it's added to an application..."""
> >
> > What does 'persisted` here mean?
>
> At some point in the past, somebody decided it would be a great idea to
> have Twisted's "twistd" start applications from Python pickles; you
> could start up a server, configure it interactively, and when you shut
> it down the entire server state would be persisted in a pickle-file;
> when you started it up it would resume exactly as it had been when you
> shut it down.
>

Whoa hang on. Without trying to hijack the thread, this is the entire
premise of infobarb, the IRC bot I'm building for #python-*, except
s/pickle/sqlite/, so if this is a horrible idea I'd like to know before I
build it.

>
> The idea seems to have fallen from grace, possibly because restarting
> a process is often the simplest way to resolve a production problem.
> These days, new Twisted users are encouraged to use Python ".tac" files
> rather than ".tap" pickles, and I expect that sentence is just a relic
> of that bygone era.
>

I suppose the problem you're suggesting is that the pickle is in a state
where running the server is impossible: do you mean like the pickle is
corrupted, or pickle has a bunch of settings that just can't work... ?

Would this problem be fixed by implementing features like differential
snapshots? Maybe a replayable or reverse-replayable event log?

cheers
lvh
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Persisted protocol?

[Tim Allen]

On Wed, Apr 06, 2011 at 12:41:08PM +0200, Laurens Van Houtven wrote:
> Whoa hang on. Without trying to hijack the thread, this is the entire
> premise of infobarb, the IRC bot I'm building for #python-*, except
> s/pickle/sqlite/, so if this is a horrible idea I'd like to know before I
> build it.

I'm not familiar with the reasons Twistd moved from .tap to .tac (it's
rather before my time), but as a user I'd be worried about things like
"what happens if, due to a bug, the server winds up choking on some
unexpected input, or deadlocked, or just mis-files some piece of state?"
Restarting the process in question is a sledgehammer approach, but
that's often what you want if the alternative is unscheduled downtime.

The difference between Pickle and SQLite is that a SQLite database has
probably had some thought put into its schema, and is much less likely
to accidentally scoop up random other objects by reference. Also, if
your database *does* pick up some unwanted state, you have the option of
tinkering with the database manually; something that's much more
difficult to do with pickles.

I'd be interested in hearing from Twisted greybeards why .tap was
deprecated, but I don't think your IRC bot has a fatal design flaw.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Persisted protocol?

[Laurens Van Houtven]

On Wed, Apr 6, 2011 at 1:27 PM, Tim Allen  wrote:

> On Wed, Apr 06, 2011 at 12:41:08PM +0200, Laurens Van Houtven wrote:
> > Whoa hang on. Without trying to hijack the thread, this is the entire
> > premise of infobarb, the IRC bot I'm building for #python-*, except
> > s/pickle/sqlite/, so if this is a horrible idea I'd like to know before I
> > build it.
>
> I'm not familiar with the reasons Twistd moved from .tap to .tac (it's
> rather before my time), but as a user I'd be worried about things like
> "what happens if, due to a bug, the server winds up choking on some
> unexpected input, or deadlocked, or just mis-files some piece of state?"
> Restarting the process in question is a sledgehammer approach, but
> that's often what you want if the alternative is unscheduled downtime.


Right -- I don't think this applies to my bot since if it does the argument
can also be extended to "don't have any persistence at all, state is bad"
;-) (It is, but that's not the most pragmatic of answers.)

The difference between Pickle and SQLite is that a SQLite database has
> probably had some thought put into its schema, and is much less likely
> to accidentally scoop up random other objects by reference. Also, if
> your database *does* pick up some unwanted state, you have the option of
> tinkering with the database manually; something that's much more
> difficult to do with pickles.
>

Right -- I'm giving plugins to the bot a SQLite database, it's up to them if
they use it. All state serialization is explicit. I'm hoping it's going to
be more sensible as a result.

I'd be interested in hearing from Twisted greybeards why .tap was
> deprecated, but I don't think your IRC bot has a fatal design flaw.
>

Whew :) Thanks for confirming anyway :)

-- 
cheers
lvh
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Instrumenting Reactors

[Itamar Turner-Trauring]

On Wed, 2011-04-06 at 10:55 +0100, Paul Thomas wrote:
> I have a reactor which is getting busier over time and I'd like to
> find out where the cycles are going. Using the profiler isn't really
> practical on a server running for days, so I'd like to instrument the
> reactor to show me which I/O events are taking the longest to deal
> with. 

You may just want to wrap manual profiling code around your code's entry
points (e.g. dataReceived/connectionMade/connectionLost/whatever gets
callLatered). It would be nice if you could get the reactor to turn this
sort of info on/off in a running system, though.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] AMP Client disconnecting from server on high memory usage

[Dan-Cristian Bogos]

Hey Jean-Paul,

For record's sake, I thought I announce here that my issue got fixed.

The problem was that I was loading the data in startService method and
it looks like there are some timers which prevent the method to block
for a longer period of time. Loading the data in __init__ of
service.Service object did the trick and I have no problem anymore
with large data.

Have a good one!

DanB

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Persisted protocol?

[Tristan Seligmann]

On Wed, Apr 6, 2011 at 1:32 PM, Laurens Van Houtven <_...@lvh.cc> wrote:
> On Wed, Apr 6, 2011 at 1:27 PM, Tim Allen  wrote:
>> The difference between Pickle and SQLite is that a SQLite database has
>> probably had some thought put into its schema, and is much less likely
>> to accidentally scoop up random other objects by reference. Also, if
>> your database *does* pick up some unwanted state, you have the option of
>> tinkering with the database manually; something that's much more
>> difficult to do with pickles.
>
> Right -- I'm giving plugins to the bot a SQLite database, it's up to them if
> they use it. All state serialization is explicit. I'm hoping it's going to
> be more sensible as a result.

For what it's worth, you might find this interesting:

https://launchpad.net/eridanus

It's an IRC bot implemented on Mantissa, which uses Axiom databases
(SQLite under the hood) in order to store and persist service
configuration as well as user data.
-- 
mithrandi, i Ainil en-Balandor, a faer Ambar

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Persisted protocol?

[exarkun]

On 07:16 am, techto...@gmail.com wrote:
>Hi again,
>> From http://twistedmatrix.com/documents/current/core/howto/udp.html
>
>"""As you can see, the protocol is registered with the reactor. This 
>means it
>may be persisted if it's added to an application..."""
>
>What does 'persisted` here mean?

As others have pointed out, this is referring to an old feature which is 
being phased out.  It would be great if you could file a ticket for 
cleaning up the docs; they mostly shouldn't talk about application 
persistence (certainly not in the UDP section).

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Persisted protocol?

[Laurens Van Houtven]

Cool, thanks for the link to Eridanus! I hadn't seen it yet :-)


cheers
lvh
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Persisted protocol?

[Laurens Van Houtven]

Urgh, thanks gmail -- that was supposed to be to mithrandi, not to the
entire mailing list. My apologies.


cheers
lvh
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Instrumenting Reactors

[Phil Mayers]

On 06/04/11 13:19, Itamar Turner-Trauring wrote:
> On Wed, 2011-04-06 at 10:55 +0100, Paul Thomas wrote:
>> I have a reactor which is getting busier over time and I'd like to
>> find out where the cycles are going. Using the profiler isn't really
>> practical on a server running for days, so I'd like to instrument the
>> reactor to show me which I/O events are taking the longest to deal
>> with.
>
> You may just want to wrap manual profiling code around your code's entry
> points (e.g. dataReceived/connectionMade/connectionLost/whatever gets
> callLatered). It would be nice if you could get the reactor to turn this
> sort of info on/off in a running system, though.

I've done this with a decorator & dictionary before, including counting 
time inside deferred return values separately from the original function 
call.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Instrumenting Reactors

[anatoly techtonik]

On Wed, Apr 6, 2011 at 1:35 PM, Laurens Van Houtven <_...@lvh.cc> wrote:
> Hi!
>
>
> No guarantee it's an optimal solution, but when I wanted to inspect a
> running process (which sounds like what you're doing), I had a pretty good
> time with manhole. (If you're not familiar with that: it's in Twisted Conch,
> and it's basically just a way to SSH in to a running box and get a REPL).
>
> In that REPL, you can of course do things like inspect the running reactor
> :-)

It would be nice to be able to attach signals emitters to some parts
of running reactor in real-time. If manhole can allow to attach to
these emitters - it will be possible to visualize them in funny way.
Does anybody know how a twisted reactor model should look like in
Blender? =)
--
anatoly t.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Persisted protocol?

[anatoly techtonik]

On Wed, Apr 6, 2011 at 3:58 PM,   wrote:
>
> As others have pointed out, this is referring to an old feature which is
> being phased out.  It would be great if you could file a ticket for
> cleaning up the docs; they mostly shouldn't talk about application
> persistence (certainly not in the UDP section).

Tried to do as clear as possible.
http://twistedmatrix.com/trac/ticket/5031
--
anatoly t.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Twisted Plugins - Implementation Discussion

[Stephen Thorne]

G'day,

So Glyph and I had a discussion about the architecture and
implementation of plugins on IRC this week, I raised some issues that
I've seen with implementing plugins in that discussion, and he said that
I should take the discussion to the list because IRC wasn't the right
place for it.

First of all, a quick discussion of the current plugin architecture, so
that we're on the same page.

'twistd' automatically imports python modules from under twisted/plugins
relative to sys.path[1], or it loads a cache of those plugins from
dropin.cache, or it loads a cache of those plugins from dropin.cache.

The reason it loads all of those plugins is so that 'tapname' and
'description' can be grabbed out of all of the serviceMaker attributes
of all those modules.

Then running 'twistd' shows a helpful list of commands, and 'twistd
$tapname' uses the correct serviceMaker to start whatever service is
specified.

Part of the discussion was about how to rewrite this in such a way that
no python code needs to be run in order to discover all the
tapname+description combinations that are available to twistd, this is
because of a perceived performance and sanity deficit in using 'twistd'.

In the course of the discussion I raised several things that I consider
annoyances in the twisted plugin system. I will repeat them here.

First, the reason i use twisted plugins is because they're the a way to
easily do sensible things with logging, daemonisation and interaction
from init.d files.

* The number of imports required to compose a plugin is annoying. 2
  interfaces from two different packages, plus needing
  zope.interface.implements.

* I've never liked the twisted arg parser, I use it only grudingly, it
  would be nice to be able to throw argv at my make_service call.

* The entire task of having this python plugin is to link up the
  metadata with a more or less standard

* It's very easy to accidentally make your plugin load your package for
  every other twistd daemon running out of the same plugin cache.

* The practice of putting a module under twisted/plugins/$mymodulehere.py
  upsets my equilibrium, the only reason I tolerate this kind of thing
  is that I install my python code via rpms and I automatically get
  installation, ownership and uninstallation done in a stable manner.

For your consideration, and (constructive) critcism, here is a twisted
plugin that is nearly identical to 6 that I have running in production:

from zope.interface import implements
from twisted.python import usage
from twisted.plugin import IPlugin
from twisted.application.service import IServiceMaker

class Options(usage.Options):
optFlags = [['debug', 'd', 'Emit debug messages']]

class ExampleServiceMaker(object):
implements(IServiceMaker, IPlugin)
tapname = 'example'
description = 'Example Twistd Plugin'
options = Options

def makeService(self, options):
from examplepackage.examplemodule import make_service
return make_service(debug=options['debug'])

serviceMaker = ExampleServiceMaker()

>From these 19 lines of code there are 4 things that are relevent:
 tapname = example
 description = 'Example Twistd Plugin'
 options = {'debug':True} if args == ['--debug'] else {'debug':False}
 service = examplepackage.examplemodule.make_service(**options)

Only 2 of which are relevent for running a twistd daemon other than
'example'

So the goal of my post to this mailing list is:

* I would like glyph's goal of having less arbitary code executed at
  twistd launch time to become a realisation,

* I would like the process of creating a twisted plugin to be less of a
  cut+paste+fill-in-blanks hassle.

-- 
Regards,
Stephen Thorne
Development Engineer
Netbox Blue

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Plugins - Implementation Discussion

[Phil Christensen]

On Apr 6, 2011, at 8:35 PM, Stephen Thorne wrote:
> For your consideration, and (constructive) critcism, here is a twisted
> plugin that is nearly identical to 6 that I have running in  
> production:
[snip]
> serviceMaker = ExampleServiceMaker()
>
> From these 19 lines of code there are 4 things that are relevent:
> tapname = example
> description = 'Example Twistd Plugin'
> options = {'debug':True} if args == ['--debug'] else {'debug':False}
> service = examplepackage.examplemodule.make_service(**options)
>
> Only 2 of which are relevent for running a twistd daemon other than
> 'example'
>
> So the goal of my post to this mailing list is:
>
> * I would like glyph's goal of having less arbitary code executed at
>  twistd launch time to become a realisation,

Makes sense. My first inclination is to suggest creating metadata  
files that are found via pkg_resources.

> * I would like the process of creating a twisted plugin to be less  
> of a
>  cut+paste+fill-in-blanks hassle.

This doesn't bother me so much. To go back in time a bit:

> First, the reason i use twisted plugins is because they're the a way  
> to
> easily do sensible things with logging, daemonisation and interaction
> from init.d files.
>
> * The number of imports required to compose a plugin is annoying. 2
>  interfaces from two different packages, plus needing
>  zope.interface.implements.
>
> * I've never liked the twisted arg parser, I use it only grudingly, it
>  would be nice to be able to throw argv at my make_service call.

I've got no issues with t.p.usage, and tend to keep its definitions in  
the plugin class. But I agree you should be able to skip it, although  
I have a suspicion you probably already can by being sneaky.

> * The entire task of having this python plugin is to link up the
>  metadata with a more or less standard

Not sure if I get this part. I tend to think of plugins as the service  
bootstrap file, like an int main(). I would probably *not* be into the  
idea of passing sys.argv directly to a service, for example.

> * It's very easy to accidentally make your plugin load your package  
> for
>  every other twistd daemon running out of the same plugin cache.

Never had this happen, but I think I can see where it might. It would  
be solved by some kind of external metadata, though, right?

> * The practice of putting a module under twisted/plugins/ 
> $mymodulehere.py
>  upsets my equilibrium, the only reason I tolerate this kind of thing
>  is that I install my python code via rpms and I automatically get
>  installation, ownership and uninstallation done in a stable manner.

The only issue I've found with this is the issue of having to create  
plugin cache files. Apart from the spurious error messages that were  
(partially?) covered at the sprint recently, the benefit they provide  
is definitely negated by bad permissions. I've done some acrobatics  
inside my setup.py to make it work, but it involves different steps  
depending on whether you're installing or building a package.

Still, it seems like most of their necessity would be negated by using  
external metadata files. It seems to be the way most plugin systems  
end up going in some way or another.

Anyways, that's just my 2/100ths. I'm pretty happy with most of the  
plugin model, but I think there's a lot of room for improvement,  
particularly in the area of dropin.cache files.

-phil

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Plugins - Implementation Discussion

[Tim Allen]

On Thu, Apr 07, 2011 at 10:35:18AM +1000, Stephen Thorne wrote:
> So the goal of my post to this mailing list is:
> 
> * I would like glyph's goal of having less arbitary code executed at
>   twistd launch time to become a realisation,
> 
> * I would like the process of creating a twisted plugin to be less of a
>   cut+paste+fill-in-blanks hassle.

I notice that Tarek Ziadé's "distutils2" is moving away from "load
a Python module and probe for well-known attributes" toward "define all
metadata in a static file". It sounds like you want a similar thing for
twistd plugins. Perhaps an implementation might look something like
this:

- At startup, twistd scans twisted/plugin directories on sys.path
  looking for files whose filenames end with '.twistd'.
- Each such file is loaded with Python's ConfigParser module.
- Each section in the ConfigParser module represents a plugin whose
  'tapname' is the section name.
- Each section has a 'description' option, whose value is
  a human-readable string describing the plugin.
- Each section has a 'module' option, whose value is a string that
  can be passed to Python's __import__ builtin to get a Python
  module.

...where the module defined by 'module' exposes a
'make_service(options)' function, and an 'options' global variable that
is an instance of t.p.usage.Options.

I know you said you didn't like t.p.usage.Options, but I'd be sad to
lose the ability for twistd to support "twistd $PLUGIN --help", and for
that kind of introspection to work, the options data needs to be in
*some* known format. Maybe this might be the time to move to the
stdlib's optparse - or maybe not, now that optparse is (presumably)
deprecated in favour of argparse. Maybe twistd could examine a number of
different well-known variable-names, for different option-parsing
libraries.

Just tossing this out as a strawman for people to point and laugh at.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Plugins - Implementation Discussion

[Glyph Lefkowitz]

On Apr 7, 2011, at 12:31 AM, Tim Allen wrote:

> On Thu, Apr 07, 2011 at 10:35:18AM +1000, Stephen Thorne wrote:
>> So the goal of my post to this mailing list is:
>> 
>> * I would like glyph's goal of having less arbitary code executed at
>>  twistd launch time to become a realisation,
>> 
>> * I would like the process of creating a twisted plugin to be less of a
>>  cut+paste+fill-in-blanks hassle.
> 
> I notice that Tarek Ziadé's "distutils2" is moving away from "load
> a Python module and probe for well-known attributes" toward "define all
> metadata in a static file". It sounds like you want a similar thing for
> twistd plugins. Perhaps an implementation might look something like
> this:
> 
>- At startup, twistd scans twisted/plugin directories on sys.path
>  looking for files whose filenames end with '.twistd'.

While I'm sympathetic to the goal here, I don't like this particular 
implementation strategy for several reasons.

Right now, in order to properly install a 'twistd' plugin (including those that 
come with Twisted), you have to do two things:

  1. install some .py files into a package
  2. as the user doing the installation (probably root), run the one-liner at 
the bottom of 
.  If 
invoking python code in your installation process is too hard, this can be 
approximated with 'twistd --help 2>&1 > /dev/null'.

This is rocket science.  Nobody can manage it.  Seriously.  After literally 
_years_ of fighting with conflicting python installation techniques in Debian 
and Ubuntu, I think that we finally have something that works about half of the 
time.  I haven't checked up on RedHat in a while and I don't know if they have 
a working system to do this yet, but they didn't last I checked.

If we invent our own file extension which has to be separately installed, we 
have to teach distutils, and setuptools, and distribute, and pip, and 
distutils2, and 'packaging' (as I'm sure that will eventually be incompatible 
with distutils2 for some silly reason), and easy_install, and dpkg, and rpm, 
and yum, and apt, and probably five other horrible Python packaging things that 
I don't even know about yet, how to deal with it.  So I am strongly in favor of 
keeping everything in .py files and just making a minor tweak to what's stored 
in dropin.cache (and perhaps allowing dropin.cache to be stored in some 
location more likely to be writable by individual users, in case the 
installation process doesn't update it).

Because, frankly, Python installation tools REALLY REALLY SHOULD be able to 
install Python files into Python packages.  I'm not sure I can make any other 
assertions quite so strongly.  I'm pretty sure that this is a problem that more 
than one project is interested in solving.  No other projects are interested in 
installing '.twistd' files though, I can assure you of that :).

>- Each such file is loaded with Python's ConfigParser module.

The first rule of the Twisted cabal is of course "don't talk about the Twisted 
cabal", but the second and possibly even more important rule is "no '.ini' 
files".  I'd seriously much rather we use XML.  And you can ask Stephen how he 
feels about XML configuration files.  (Although I'd strongly recommend standing 
well clear of him when you do that, and making sure that no sharp or otherwise 
dangerous objects are within easy reach.)

>- Each section blah blah terrible user interface stuff about 'sections' 
> and other misfeatures of ini files.

I don't want a solution that is hard-coded to deal with the metadata that 
'twistd' specifically needs, as Twisted plugins are already used for more than 
just twistd plugins, and I'd like them to be used for even more.  An interface 
that limits the metadata to ConfigParser sections would make it awkward to fit 
into a management GUI or web page.

Plus, the quoting rules for long strings in ini files make it unsuitable for 
storing long descriptive strings (which is one of the primary use-cases here).

> ...where the module defined by 'module' exposes a
> 'make_service(options)' function, and an 'options' global variable that
> is an instance of t.p.usage.Options.
> 
> I know you said you didn't like t.p.usage.Options, but I'd be sad to
> lose the ability for twistd to support "twistd $PLUGIN --help", and for
> that kind of introspection to work, the options data needs to be in
> *some* known format. Maybe this might be the time to move to the
> stdlib's optparse - or maybe not, now that optparse is (presumably)
> deprecated in favour of argparse. Maybe twistd could examine a number of
> different well-known variable-names, for different option-parsing
> libraries.

For what it's worth, I don't care about this at all.  It's a completely 
separate issue from the main stuff I care about, and while we should be able to 
simply delegate to a function that takes a list of strings, I will insist that 
we fix that separately.

>

Re: [Twisted-Python] Twisted Plugins - Implementation Discussion

[Glyph Lefkowitz]

On Apr 7, 2011, at 12:06 AM, Phil Christensen wrote:

> On Apr 6, 2011, at 8:35 PM, Stephen Thorne wrote:
>> For your consideration, and (constructive) critcism, here is a twisted
>> plugin that is nearly identical to 6 that I have running in  
>> production:
> [snip]
>> serviceMaker = ExampleServiceMaker()
>> 
>> From these 19 lines of code there are 4 things that are relevent:
>> tapname = example
>> description = 'Example Twistd Plugin'
>> options = {'debug':True} if args == ['--debug'] else {'debug':False}
>> service = examplepackage.examplemodule.make_service(**options)
>> 
>> Only 2 of which are relevent for running a twistd daemon other than
>> 'example'
>> 
>> So the goal of my post to this mailing list is:
>> 
>> * I would like glyph's goal of having less arbitary code executed at
>> twistd launch time to become a realisation,
> 
> Makes sense. My first inclination is to suggest creating metadata  
> files that are found via pkg_resources.

We already use a similar mechanism, twisted.python.modules, which uses the same 
underlying standards as pkg_resources (PEP 302) but is somewhat more flexible.  
And we create a metadata file (dropin.cache) which is stored and retrieved 
using this mechanism.

>> * The entire task of having this python plugin is to link up the
>> metadata with a more or less standard
> 
> Not sure if I get this part. I tend to think of plugins as the service  
> bootstrap file, like an int main(). I would probably *not* be into the  
> idea of passing sys.argv directly to a service, for example.

Why not?  It's just a list of strings.  You should be able to deal with it how 
you like.  (But as I said in a previous message: this is a separate issue.)

>> * It's very easy to accidentally make your plugin load your package  
>> for every other twistd daemon running out of the same plugin cache.
> 
> Never had this happen, but I think I can see where it might. It would  
> be solved by some kind of external metadata, though, right?

It already is solved by the external metadata... sort of.  If you look at the 
implementation of CachedPlugin, you can see that it actually already has a name 
and description!  There are two problems though: first is that this is 
hard-coded to be the module's name and docstring, but more importantly, there's 
just no way to get at those attributes via the getPlugin interface, which 
implicitly invokes '__conform__' via adaptation (and therefore load()).

> * The practice of putting a module under twisted/plugins/ 
>> $mymodulehere.py
>> upsets my equilibrium, the only reason I tolerate this kind of thing
>> is that I install my python code via rpms and I automatically get
>> installation, ownership and uninstallation done in a stable manner.
> 
> The only issue I've found with this is the issue of having to create  
> plugin cache files. Apart from the spurious error messages that were  
> (partially?) covered at the sprint recently, the benefit they provide  
> is definitely negated by bad permissions. I've done some acrobatics  
> inside my setup.py to make it work, but it involves different steps  
> depending on whether you're installing or building a package.

What is "this kind of thing", though?  The plugins have to go in some defined 
namespace in order to be enumerated.  Even if we were to implement something 
based on purely static metadata, you'd still have to list a directory to get at 
that metadata.  Making the namespace be owned by the module doing the importing 
makes sense.

For what it's worth, bzrlib does this too, but by convention in a bzr plugin 
you put _all_ your code into bzrlib/plugins/foo/*.py, and your package is 
bzrlib.plugins.foo.  (This will work fine with Twisted if you want to do it 
that way.)

> Still, it seems like most of their necessity would be negated by using  
> external metadata files. It seems to be the way most plugin systems  
> end up going in some way or another.

dropin.cache is an external metadata file ;-).

> Anyways, that's just my 2/100ths. I'm pretty happy with most of the  
> plugin model, but I think there's a lot of room for improvement,  
> particularly in the area of dropin.cache files.

Thanks for your feedback.___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Plugins - Implementation Discussion

[Tim Allen]

On Thu, Apr 07, 2011 at 12:54:45AM -0400, Glyph Lefkowitz wrote:
> If we invent our own file extension which has to be separately
> installed, we have to teach distutils, and setuptools, and distribute,
> and pip, and distutils2, and 'packaging' (as I'm sure that will
> eventually be incompatible with distutils2 for some silly reason), and
> easy_install, and dpkg, and rpm, and yum, and apt, and probably five
> other horrible Python packaging things that I don't even know about
> yet, how to deal with it.  So I am strongly in favor of keeping
> everything in .py files and just making a minor tweak to what's stored
> in dropin.cache (and perhaps allowing dropin.cache to be stored in
> some location more likely to be writable by individual users, in case
> the installation process doesn't update it).

My understanding was that .py files have to be installed into
twisted/plugins as binary blobs, not as ordinary Python modules, because
of special rules like "twisted/plugins must not be a Python package". If
distutils/setuptools/etc. can handle a binary blob with a ".py"
extension, I figured it could handle a binary blob with any other
extension.

If that's wrong, then yeah, I guess that would be a problem.

> >- Each such file is loaded with Python's ConfigParser module.
> 
> The first rule of the Twisted cabal is of course "don't talk about the
> Twisted cabal", but the second and possibly even more important rule
> is "no '.ini' files".  I'd seriously much rather we use XML.  And you
> can ask Stephen how he feels about XML configuration files.  (Although
> I'd strongly recommend standing well clear of him when you do that,
> and making sure that no sharp or otherwise dangerous objects are
> within easy reach.)

Well, the nice thing about ConfigParser is that it's in the stdlib, and
people already know how to create them, and rolling
yet-another-config-file-format seems crazy in this day and age.

If you need a non-Turing-complete config language and rule out .ini and
XML, I'm not sure what's left. JSON, perhaps.

> I don't want a solution that is hard-coded to deal with the metadata
> that 'twistd' specifically needs, as Twisted plugins are already used
> for more than just twistd plugins, and I'd like them to be used for
> even more.

I've never actually come across anything that used Twisted plugins
besides twistd, so I'd forgotten they weren't twistd-specific.

> So as not to make this message too long, I'll defer a description my
> own preferred implementation strategy for a future post to this
> thread.

I'll look forward to it. :)

Tim.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Plugins - Implementation Discussion

[Stephen Thorne]

On 2011-04-07, Glyph Lefkowitz wrote:
> Because, frankly, Python installation tools REALLY REALLY SHOULD be
> able to install Python files into Python packages.  I'm not sure I can
> make any other assertions quite so strongly.  I'm pretty sure that
> this is a problem that more than one project is interested in solving.
> No other projects are interested in installing '.twistd' files though,
> I can assure you of that :).

This is entirely wrong. Python installation tools are barely capable of
putting entire existing working python packages into a directory that if
you mumble rhymes with "kite smackages".

To expect installation tools to be able to put a python-script into a
nested subdirectory of an entirely different subtree without putting
in __init__.py files, and working with .pth files, and not being insane,
you have to hack things.

kite-smackages/twisted/plugins/myplugin.py installed with any standard
tool without simply hard-coding is a disaster. Putting it outside of
that directory, more so.

-- 
Regards,
Stephen Thorne
Development Engineer
Netbox Blue

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Plugins - Implementation Discussion

[Glyph Lefkowitz]

On Apr 7, 2011, at 1:38 AM, Stephen Thorne wrote:

> On 2011-04-07, Glyph Lefkowitz wrote:
>> Because, frankly, Python installation tools REALLY REALLY SHOULD be
>> able to install Python files into Python packages.  I'm not sure I can
>> make any other assertions quite so strongly.  I'm pretty sure that
>> this is a problem that more than one project is interested in solving.
>> No other projects are interested in installing '.twistd' files though,
>> I can assure you of that :).
> 
> This is entirely wrong. Python installation tools are barely capable of
> putting entire existing working python packages into a directory that if
> you mumble rhymes with "kite smackages".

This is why I said "should".  Python installation tools are basically incapable 
of anything.  And yet.

My point is that it's hard enough to advocate for bugs to be fixed in 
installing .py files; let's focus on that, and avoid installing other stuff.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Plugins - Implementation Discussion

[Glyph Lefkowitz]

On Apr 7, 2011, at 1:08 AM, Tim Allen wrote:

> Well, the nice thing about ConfigParser is that it's in the stdlib, and
> people already know how to create them, and rolling
> yet-another-config-file-format seems crazy in this day and age.

My point was really that people think they know how to create these, but 
actually they don't.  Pop quiz, hot shot: what is the quoting rule to put a 
linebreak with preserved trailing whitespace into a value in a .ini file?  Into 
a key?

> If you need a non-Turing-complete config language and rule out .ini and
> XML, I'm not sure what's left. JSON, perhaps.

Pickle, of course.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Plugins - Implementation Discussion

[Tristan Seligmann]

On 7 Apr 2011 06:56, "Glyph Lefkowitz"  wrote:
>>- Each section blah blah terrible user interface stuff about
'sections' and other misfeatures of ini files.
>
>
> I don't want a solution that is hard-coded to deal with the metadata that
'twistd' specifically needs, as Twisted plugins are already used for more
than just twistd plugins, and I'd like them to be used for even more.  An
interface that limits the metadata to ConfigParser sections would make it
awkward to fit into a management GUI or web page.

I'd like to jump in with a +1000 here. I have literally written over 1000
plugins[1] using twisted.plugin, and not one of them was a twistd plugin. I
would very much like to take advantage of any improvements in this area in
my own projects.

[1]
http://bazaar.launchpad.net/~dosage-dev/dosage/trunk/files/head:/dosage/plugins/
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Plugins - Implementation Discussion

[Stephen Thorne]

On 2011-04-07, Glyph Lefkowitz wrote:
> 
> On Apr 7, 2011, at 1:38 AM, Stephen Thorne wrote:
> 
> > On 2011-04-07, Glyph Lefkowitz wrote:
> >> Because, frankly, Python installation tools REALLY REALLY SHOULD be
> >> able to install Python files into Python packages.  I'm not sure I can
> >> make any other assertions quite so strongly.  I'm pretty sure that
> >> this is a problem that more than one project is interested in solving.
> >> No other projects are interested in installing '.twistd' files though,
> >> I can assure you of that :).
> > 
> > This is entirely wrong. Python installation tools are barely capable of
> > putting entire existing working python packages into a directory that if
> > you mumble rhymes with "kite smackages".
> 
> This is why I said "should".  Python installation tools are basically
> incapable of anything.  And yet.
> 
> My point is that it's hard enough to advocate for bugs to be fixed in
> installing .py files; let's focus on that, and avoid installing other
> stuff.

Everything has the capability to put datafiles somewhere, even if the
location offset is annoying. In fact, most tools are /better/ at
installing arbitary files that don't have a first line of #!python or
.py extension than files that do.

For this reason, a static configuration file, such as a hand crafted
pickle, an xml file, or an ini file, is a reasonably viable alternative.

I hesitate to suggest it, but a .pth file or a setuptools entrypoint may
be an option here too for inserting plugins without writing to a doubly
nested non-package twisted ''package'' plugins directory.

-- 
Regards,
Stephen Thorne
Development Engineer
Netbox Blue

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Plugins - Implementation Discussion

[Stephen Thorne]

On 2011-04-07, Glyph Lefkowitz wrote:
> 
> On Apr 7, 2011, at 1:08 AM, Tim Allen wrote:
> 
> > Well, the nice thing about ConfigParser is that it's in the stdlib, and
> > people already know how to create them, and rolling
> > yet-another-config-file-format seems crazy in this day and age.
> 
> My point was really that people think they know how to create these,
> but actually they don't.  Pop quiz, hot shot: what is the quoting rule
> to put a linebreak with preserved trailing whitespace into a value in
> a .ini file?  Into a key?

Reality check. This are the plugins that are currently shipped:

ftp  An FTP server.
telnet   A simple, telnet-based remote debugging service.
socksA SOCKSv4 proxy service.
manhole-old  An interactive remote debugger service.
portforward  A simple port-forwarder.
web  A general-purpose web server which can serve from a
 filesystem or application resource.
inetdAn inetd(8) replacement.
news A news server.
xmpp-router  An XMPP Router server
wordsA modern words server
dns  A domain name server.
mail An email service
manhole  An interactive remote debugger service accessible via
 telnet and ssh and providing syntax coloring and basic line
 editing functionality.
conchA Conch SSH service.
procmon  A process watchdog / supervisor

Why do we care about complex quoting and linebreaks for descriptions? If
you can't remember, just keep typing and let it get wrapped.

-- 
Regards,
Stephen Thorne
Development Engineer
Netbox Blue

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Plugins - Implementation Discussion

[Glyph Lefkowitz]

On Apr 7, 2011, at 1:55 AM, Stephen Thorne wrote:

> On 2011-04-07, Glyph Lefkowitz wrote:
>> 
>> On Apr 7, 2011, at 1:08 AM, Tim Allen wrote:
>> 
>>> Well, the nice thing about ConfigParser is that it's in the stdlib, and
>>> people already know how to create them, and rolling
>>> yet-another-config-file-format seems crazy in this day and age.
>> 
>> My point was really that people think they know how to create these,
>> but actually they don't.  Pop quiz, hot shot: what is the quoting rule
>> to put a linebreak with preserved trailing whitespace into a value in
>> a .ini file?  Into a key?
> 
> Reality check. This are the plugins that are currently shipped:
> 
>ftp  An FTP server.
>telnet   A simple, telnet-based remote debugging service.
>socksA SOCKSv4 proxy service.
>manhole-old  An interactive remote debugger service.
>portforward  A simple port-forwarder.
>web  A general-purpose web server which can serve from a
> filesystem or application resource.
>inetdAn inetd(8) replacement.
>news A news server.
>xmpp-router  An XMPP Router server
>wordsA modern words server
>dns  A domain name server.
>mail An email service
>manhole  An interactive remote debugger service accessible via
> telnet and ssh and providing syntax coloring and basic 
> line
> editing functionality.
>conchA Conch SSH service.
>procmon  A process watchdog / supervisor
> 
> Why do we care about complex quoting and linebreaks for descriptions? If
> you can't remember, just keep typing and let it get wrapped.

None of those descriptions have non-ASCII characters in there either, but that 
doesn't mean I want to standardize on a format where I can't figure out how to 
type them.  I would like to provide more flexibility with a simpler API, not 
less flexibility and more complexity.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Plugins - Implementation Discussion

[Glyph Lefkowitz]

On Apr 6, 2011, at 8:35 PM, Stephen Thorne wrote:

> Part of the discussion was about how to rewrite this in such a way that
> no python code needs to be run in order to discover all the
> tapname+description combinations that are available to twistd, this is
> because of a perceived performance and sanity deficit in using 'twistd'.

My interest in this discussion is not so much in "no python code should be 
executed" but rather "the current constraints of the system should be preserved 
(your whole package doesn't get imported) but you shouldn't have to write hacks 
like ServiceMaker 
()to
 preserve them".  Or, for that matter, do inner imports, like this one from 
your example:

>def makeService(self, options):
>from examplepackage.examplemodule import make_service
>return make_service(debug=options['debug'])


Someone unfamiliar with the Twisted plugin system would probably not realize 
that the positioning of that import is critically important.  It seems kind of 
random, and maybe sloppy, and a refactoring for stylistic fixes might move it 
to the top of the module.

Of course, such a refactoring would make 'twistd --help' on any system with 
your code installed start executing gobs and gobs of additional code.  Also, as 
a result of such a change, every 'twistd' server on such a system would have 
your entire examplepackage.examplemodule imported, silently of course, 
increasing their memory footprint and so on.

As I have mentioned in other parts of this mailing list thread, there's already 
some caching going on, but it's never used.  Observe:

glyph@... twisted/plugins$ python
Python 2.6.1 (...)
>>> from cPickle import load
>>> plugins = load(file('dropin.cache'))
>>> plugins['twisted_names'].plugins
[]
>>> plugins['twisted_names'].plugins[0].name
'TwistedNames'
>>> plugins['twisted_names'].plugins[0].description
'\nUtility class to simplify the definition of L{IServiceMaker} plugins.\n  
  '
>>> plugins['twisted_names'].plugins[0].provided
[, ]
>>> import sys
>>> 'twisted.plugins' in sys.modules
False

The problem with this is that once you've loaded the plugins, you can't see it 
any more:

>>> from twisted.plugin import getPlugins
>>> from twisted.application.service import IServiceMaker
>>> allPlugins = list(getPlugins(IServiceMaker))
>>> plugin = [p for p in allPlugins if p.tapname == 'dns'][0]
>>> plugin.description
'A domain name server.'
>>> plugin.name
'Twisted DNS Server'

Those are the 'name' and 'description' attributes from the IServiceMaker 
provider, already implicitly loaded by getPlugins.  You can't see the 
CachedPlugin any more.

So, here's an idea, very similar to the one on the ticket.  Keeping in mind the 
state described above, hopefully it will communicate my idea better.

Right now, IPlugin is purely a marker.  It provides no methods.  I propose a 
new subinterface (designed to eventually replace it), IPlugin2, with one 
method, 'metadata()', that returns a dictionary mapping strings to strings.  
This _could_ be any object, limited only by what we think is a good idea to 
allow serializing.  The second method would be 'willProvide(I)' which returns a 
boolean, whether the result of load() will provide the interface 'I'.

Then there's a helper which you inherit which looks like:

class Plugin2(object):
implements(IPlugin2)
def metadata(self):
raise NotImplementedError("your metadata here")
def willProvide(self, I):
return I.providedBy(self)
def load(self):
return self

The one rule here is that 'metadata()' must always return the same value for a 
particular version of the code.  We will then serialize the metadata from 
calling metadata() into dropin.cache, and expose it to application code.

My idea for exposing it is that if you then do 'getPlugins(IPlugin2)', you will 
get back an iterable of IPlugin2 providers, but not necessarily instances of 
your classes: they could be cached plugins, with cached results for metadata() 
and willProvide() - the latter based on the list currently saved as the 
'provided' attribute.  So a loop like this to load a twistd plugin by name:

def twistdPluginByTapname(name):
for p2 in getPlugins(IPlugin2):
if p2.willProvide(IServiceMaker) and p2.metadata()['tapname'] == name:
return p2.load()

... would not actually load any plugins, but work entirely from the cached 
metadata.  Since you wouldn't be loading the plugin except to actually invoke 
its dynamic behavior, we would no longer need ServiceMaker, just an instance of 
the actual IServiceMaker plugin, with no local imports or anything.

This would at least partially address one of your complaints, Stephen, in that 
it would mean that a plugin could be defined with 2 lines: import your class, 
and create an instance of it.  Of course you'd still need boilerplate 
somewhere, but it would be possible to put a

Re: [Twisted-Python] Twisted Plugins - Implementation Discussion

[David]

On 04/07/2011 02:08 PM, Tim Allen wrote:

>
> If you need a non-Turing-complete config language and rule out .ini and
> XML, I'm not sure what's left. JSON, perhaps.

Having had experience with JSON for configuration: it is a terrible 
format for configuration, if only because it does not support comments.

The syntax is also a bit too strict: enough to be annoying in something 
you want to edit all the time and easily in my experience.

cheers,

David


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Plugins - Implementation Discussion

[Tim Allen]

On Thu, Apr 07, 2011 at 03:24:57PM +0900, David wrote:
> On 04/07/2011 02:08 PM, Tim Allen wrote:
> > If you need a non-Turing-complete config language and rule out .ini and
> > XML, I'm not sure what's left. JSON, perhaps.
> 
> Having had experience with JSON for configuration: it is a terrible 
> format for configuration, if only because it does not support comments.
> 
> The syntax is also a bit too strict: enough to be annoying in something 
> you want to edit all the time and easily in my experience.

Well, that's pretty depressing. The only other candidate I can even
think of is YAML, and that's not in the standard library (as far as
I know).

Who'd have guessed it'd be so complicated to associate keys with values?

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Plugins - Implementation Discussion

[David]

On 04/07/2011 03:34 PM, Tim Allen wrote:

>
> Who'd have guessed it'd be so complicated to associate keys with values?

If that's the only thing you need, .ini would work fine. Another 
solution would be python files with only literals, parsed through the 
ast module for safety.

cheers,

David

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Instrumenting Reactors

[Michael Thompson]

On 6 April 2011 10:55, Paul Thomas  wrote:
> Should I just hack into the reactor somewhere? Or is there something sitting 
> in a library I haven't seen that will help with this?

You can time blocking calls by instrumenting
twisted.python.log.callWithContext and you could try writing the
timing info to something fast, like Redis.

Michael

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python