Re: [Twisted-Python] Fork/Spawn children to accept connections on the same port.
On Fri, 10 Apr 2009 10:16:06 -0700, Eric York wrote: From: Jean-Paul Calderone Date: April 10, 2009 8:57:19 AM PDT To: Twisted general discussion Subject: Re: [Twisted-Python] Fork/Spawn children to accept connections on the same port. Reply-To: Twisted general discussion On Mon, 6 Apr 2009 15:25:39 -0700, Eric York wrote: I am trying to get the highest level of performance using all of the processors cores on a server. In the past, a unix app would bind/listen to a socket and then fork or spawn children to accept connections on that socket. I can’t see how to do that in Twisted. Can someone point me in the right direction? The solution of using a single process to accept connections and then farm out work to child processes, while a workable solution, isn’t at the same level of performance as children processes that are doing their own accepts. It's also possible to bind a port in one process and then send it over a UNIX socket to another process; this comes closer to the bind/ fork approach, but accomplishes the resource sharing explicitly via fd passing rather than through fork. This is the path that I would like to follow. I can see how to spawn a child process and pass the fd in Twisted. What I can't see how to do in Twisted is to have a parent process just bind and listen to a socket and have a child process accept on that socket. In twisted/ internet/tcp.py in the class Port, there is a function startListening which does bind/listen/startReading all in this one function. It seems that a small refactoring would allow the parent to bind and a child to do startReading, if the call to startReading was moved out of startListening. The reactor calls would also need a small refactoring to allow this type of setup. How does that sound? The main problem that comes up with this approach is that little, or possibly none, of Twisted is written to be fork-safe. As with most programs, though, this is /usually/ not a problem. However, there are some areas where it can be. For example, if you use epoll reactor, the internal epoll file descriptor may end up shared between the two processes. This results in weird behavior like thread wakeup messages in one process waking up the reactor thread in the other process. It's probably possible to fix all the problems like this, but rather than try, I'd just not try forking a Twisted process without using exec in one of the resulting processes shortly afterwards. If you create a port in one process, you can just grab its fileno and pass that to another process. Then you can create a new Port object with that fileno. This may involve subclassing Port and overriding createInternetSocket to just return the fileno you have already. These APIs aren't really intended for direct use by applications, but there's not really another way to do what you want right now. You may want to help us work out a better API for creating Ports and such from pre-existing sockets if you go this route so that there's less chance of some deep-down implementation change disturbing your app. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Plugin startup and shutdown actions
On Sat, 11 Apr 2009 00:24:37 +0200, luper rouch wrote: I would like to port my Application API based services to the twistd plugin interface, to be able to retrieve command line options. In the existing services, I reimplemented t.a.s.MultiService startService() and stopService() to do some actions at startup and shutdown. How can I do this in a plugin ? The top-level API for plugins is mostly a function that returns an IService provider. So if you have your own version of MultiService, you can just return an instance of that, just like the one you were creating in your .tac file, from a makeService function which is registered as a plugin of the suitable type. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Two-way telnet client problem
On Fri, 10 Apr 2009 13:10:28 -0400, Greg Taylor wrote: I guess what I'm getting confused about is how to get my listen server (MUD server) talking to this IMC client connection. Let's say one of my player wants to send a message over the IMC (client) connection. This is done by telnetting into the MUD server and entering a command like: imcmessage "hello". So I've got a reactor serving up my MUD server process, and I'm not sure how this IMC client plugs in. This sounds a lot like this FAQ: http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#HowdoImakeinputononeconnectionresultinoutputonanother I guess I need to create some kind of queue that the IMC ConnectionMade checks periodically and executes if I can't directly get the MUD server talking to the IMCProtocol object. Fortunately, you can make them talk directly. :) No need for a queue. Also, do I need to do anything special to prevent blocking? As long as you're using Twisted APIs for dealing with the networking, no - those won't block. Do I use two separate reactors or should the IMC connection be using the same reactor or process group as the MUD server (listen server)? There's always exactly one reactor in a Twisted app. Thanks for your help. I'm sure this will be really simple to do once I figure out how to do this. Hope this helps, Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Plugin startup and shutdown actions
2009/4/11 Jean-Paul Calderone : > On Sat, 11 Apr 2009 00:24:37 +0200, luper rouch > wrote: >> >> I would like to port my Application API based services to the twistd >> plugin interface, to be able to retrieve command line options. >> >> In the existing services, I reimplemented t.a.s.MultiService >> startService() and stopService() to do some actions at startup and >> shutdown. >> >> How can I do this in a plugin ? >> > > The top-level API for plugins is mostly a function that returns an IService > provider. So if you have your own version of MultiService, you can just > return an instance of that, just like the one you were creating in your .tac > file, from a makeService function which is registered as a plugin of the > suitable type. > Thanks I didn't think it was so simple ! I have a last question, I put my plugins in a 'twisted/plugins' subfolder of my project, and running them from the command line works fine. How can I invoke them in unit tests (I need to be able to start and stop them), since the plugin files are not in a package ? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Plugin startup and shutdown actions
On Sat, 11 Apr 2009 19:29:28 +0200, luper rouch wrote: 2009/4/11 Jean-Paul Calderone : On Sat, 11 Apr 2009 00:24:37 +0200, luper rouch wrote: I would like to port my Application API based services to the twistd plugin interface, to be able to retrieve command line options. In the existing services, I reimplemented t.a.s.MultiService startService() and stopService() to do some actions at startup and shutdown. How can I do this in a plugin ? The top-level API for plugins is mostly a function that returns an IService provider. So if you have your own version of MultiService, you can just return an instance of that, just like the one you were creating in your .tac file, from a makeService function which is registered as a plugin of the suitable type. Thanks I didn't think it was so simple ! I have a last question, I put my plugins in a 'twisted/plugins' subfolder of my project, and running them from the command line works fine. How can I invoke them in unit tests (I need to be able to start and stop them), since the plugin files are not in a package ? They can be imported from "twisted.plugins". For example, if you name your "dropin" file foo_plugins.py, then "from twisted.plugins import foo_plugins" should work and let you test any code that is part of your plugin definitions. Note that you should try to keep the amount of code and dependencies in a dropin file to a minimum, since this must all be loaded and executed whenever a search for any plugin is performed. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Plugin startup and shutdown actions
Hi Luper > "luper" == luper rouch writes: luper> I have a last question, I put my plugins in a 'twisted/plugins' luper> subfolder of my project, and running them from the command line luper> works fine. How can I invoke them in unit tests (I need to be able luper> to start and stop them), since the plugin files are not in a luper> package? I suspect some will answer "Uh, you probably shouldn't be doing that"... But something like this will work: from twisted.plugin import getPlugins from twisted.application.service import IServiceMaker def getPlugin(name): for plugin in getPlugins(IServiceMaker): if hasattr(plugin, 'tapname'): if plugin.tapname == name: return plugin return None Terry ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Plugin startup and shutdown actions
On Saturday 11 April 2009 19:42:31 Jean-Paul Calderone wrote: > They can be imported from "twisted.plugins". For example, if you name your > "dropin" file foo_plugins.py, then "from twisted.plugins import > foo_plugins" should work and let you test any code that is part of your > plugin definitions. Note that you should try to keep the amount of code and > dependencies in a dropin file to a minimum, since this must all be loaded > and executed whenever a search for any plugin is performed. Shouldn't twisted and twisted/plugins be non-importable? I thought twisted and twisted/plugins must not have __init__.py files in order for the plugin system to be able to find them (through getPlugins). Or did that behavior change in recent versions? Cheers. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Plugin startup and shutdown actions
On Sat, 11 Apr 2009 21:54:08 +0200, Esteve Fernandez wrote: On Saturday 11 April 2009 19:42:31 Jean-Paul Calderone wrote: They can be imported from "twisted.plugins". For example, if you name your "dropin" file foo_plugins.py, then "from twisted.plugins import foo_plugins" should work and let you test any code that is part of your plugin definitions. Note that you should try to keep the amount of code and dependencies in a dropin file to a minimum, since this must all be loaded and executed whenever a search for any plugin is performed. Shouldn't twisted and twisted/plugins be non-importable? I thought twisted and twisted/plugins must not have __init__.py files in order for the plugin system to be able to find them (through getPlugins). Or did that behavior change in recent versions? They must not have __init__.py files, indeed. However, the files in twisted/plugins/ still define sub-modules of the twisted.plugins package due to the code in Twisted's twisted/plugins/__init__.py. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] adapt from more than one interface
Hi all. I have a bunch of services, which implement a set of interfaces, and I use a bunch of adapters to turn them into factories, making those services protocol-independent. The interface/adapter stuff works great and I'm able to expose those services to several protocols, without changing a single line in them. However, some of the protocols need some extra information to be passed, so I have to do this: factory = IFooFactory(myService) factory.bar = "some value" it works, but I'd rather use this: factory = IFooFactory(myService, configurationObject) which would take a second argument, an object implementing another interface (e.g. IConfigurationDescriptor), containing all the necessary information for that adapter. I don't want to put that information in the service, as it's only useful to the protocol and would tie the service to a particular protocol. In order to implement this, registerAdapter would have to be able to take a tuple of interfaces, like this: components.registerAdapter( AdaptToFactory, (IFooService, IConfigurationDescriptor), IFooFactory) would that feature make sense? If so, I'll file an issue right now :-) Cheers. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Plugin startup and shutdown actions
2009/4/11 Esteve Fernandez : > On Saturday 11 April 2009 19:42:31 Jean-Paul Calderone wrote: >> They can be imported from "twisted.plugins". For example, if you name your >> "dropin" file foo_plugins.py, then "from twisted.plugins import >> foo_plugins" should work and let you test any code that is part of your >> plugin definitions. Note that you should try to keep the amount of code and >> dependencies in a dropin file to a minimum, since this must all be loaded >> and executed whenever a search for any plugin is performed. > > Shouldn't twisted and twisted/plugins be non-importable? I thought twisted and > twisted/plugins must not have __init__.py files in order for the plugin > system to be able to find them (through getPlugins). Or did that behavior > change in recent versions? > > Cheers. It's working on 8.2.0 release, I have been able to port all my tests to the plugins API (they even seem less hackish now...) Thanks ! ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Giving inlineCallbacks-wrapped functions access to the deferred their wrapper will return
This will probably fall into the too-weird or too-infrequent to be worth implementing category, but I think it's worth mentioning. (BTW, none of the following code has been run.) When you write a normal function that returns a deferred, it (of course) has access to the deferred it's going to return: def func(): d = defer.Deferred() d.addCallback(...).addErrback(...) callSomethingElse(d) return d Apart from operating on the deferred itself, func can pass the deferred to some other function that might add more callbacks (as in the call to callSomethingElse). That's all very nice and convenient. But if I write this: @defer.inlineCallbacks def func(): # do some stuff to calculate a result. defer.returnValue(result) func never has its hands on the deferred that will be returned (by _inlineCallbacks) to the caller of its wrapper. I can get around this in ways that are less elegant: @defer.inlineCallbacks def func(): # do some stuff d = defer.Deferred() callSomethingElse(d) d.callback(result) newResult = yield d defer.returnValue(newResult) But I don't really like doing it that way: it's a slightly different coding style, it's much more manual, the above is just one simple example - actual usage could be more involved, and it doesn't feel as natural as the kind of code I can write when I'm not using inlineCallbacks. A way around this (which maybe will generate howls of protest) would be to have _inlineCallbacks do something this: def unwindGenerator(*args, **kwargs): assert '_inlineCallbacksDeferred' not in kwargs d = Deferred() return _inlineCallbacks(None, f(*args, **kwargs, _inlineCallbacksDeferred=d), d) return mergeFunctionMetadata(f, unwindGenerator) While I don't like this implementation much, it does manage to simply put the deferred that _inlineCallbacks will eventually return, into the hands of f (i.e., func above). func can then pass the deferred around, add callbacks to it etc. exactly as a normal (non-inlineCallbacks) function would. Maybe there's a cleaner way to implement this? I ran into this asymmetry today, when I wanted a function to pass its deferred off to another function but realized that because mine was an inlineCallbacks-decorated function I couldn't do it in the same way I would otherwise. Comments? Terry ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Plugin startup and shutdown actions
On 11 Apr, 08:39 pm, exar...@divmod.com wrote: On Sat, 11 Apr 2009 21:54:08 +0200, Esteve Fernandez wrote: Shouldn't twisted and twisted/plugins be non-importable? I thought twisted and twisted/plugins must not have __init__.py files in order for the plugin system to be able to find them (through getPlugins). Or did that behavior change in recent versions? They must not have __init__.py files, indeed. However, the files in twisted/plugins/ still define sub-modules of the twisted.plugins package due to the code in Twisted's twisted/plugins/__init__.py. They must not have __init__.py files specifically to avoid *conflicting* with this Twisted's twisted/plugins/__init__.py. Since the idea is that, during installation, your foo_plugin.py will go into the site's twisted/plugins directory, we forbid projects to make twisted/plugins a package because otherwise one package would stomp over the other's __init__.py and break things. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] adapt from more than one interface
On 11 Apr, 08:41 pm, est...@sindominio.net wrote:
factory = IFooFactory(myService)
factory.bar = "some value"
it works, but I'd rather use this:
factory = IFooFactory(myService, configurationObject)
This is a long and terrible road you're about to start down. It's hard
for me to explain exactly why it's so painful, but trust me, it is.
What you're doing now, passing out-of-band arguments to the IFooFactory
adapter, whose concrete type you are *not supposed to know* at that
point in the control flow, is bad. But trying to force adaptation to
become full-fledged multimethods is worse. If you want multimethods, I
believe PEAK provides a package that implements them.
If you want to keep doing what you're doing, you should do this:
factory = MyServiceFooFactory(myService)
factory.configureBar("some value")
i.e. don't ask for an adaptation which might give you an object without
a "bar" attribute, just create the concrete type that does what you need
based on the type of myService and configure it with explicitly
documented methods. Although if I misunderstand the spec of
"IFooFactory" and it *does* include a bar attribute, nevermind.
If you want something more magical and adapter-ish, them maybe you want
this:
IFooFactory(ConfigurationObject(myService, bar="some value"))
i.e. by the time you are getting around to the IFooFactory adaptation,
you should have an object that fully provides enough information that
the concrete FooFactory adapter type can be fully initialized when it is
created.
In order to implement this, registerAdapter would have to be able to
take a
tuple of interfaces, like this:
components.registerAdapter(
AdaptToFactory,
(IFooService, IConfigurationDescriptor),
IFooFactory)
would that feature make sense? If so, I'll file an issue right now :-)
Unfortunately, much as I've been encouraging people to file tickets
lately, no :). I don't think it makes sense.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] adapt from more than one interface
On Sun, Apr 12, 2009 at 10:31 AM, wrote: > > On 11 Apr, 08:41 pm, est...@sindominio.net wrote: >> >> factory = IFooFactory(myService) >> factory.bar = "some value" >> >> it works, but I'd rather use this: >> >> factory = IFooFactory(myService, configurationObject) > > This is a long and terrible road you're about to start down. It's hard for > me to explain exactly why it's so painful, but trust me, it is. > > What you're doing now, passing out-of-band arguments to the IFooFactory > adapter, whose concrete type you are *not supposed to know* at that point in > the control flow, is bad. But trying to force adaptation to become > full-fledged multimethods is worse. If you want multimethods, I believe > PEAK provides a package that implements them. > Also, Zope has getMultiAdapter(). The canonical example is adapting a model object and a request object to a view object. I don't think it's as bad as glyph says. I probably wouldn't use it in the case you describe -- calling methods on objects is almost always better. jml ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Giving inlineCallbacks-wrapped functions access to the deferred their wrapper will return
On 11 Apr, 10:55 pm, te...@jon.es wrote: I can get around this in ways that are less elegant: @defer.inlineCallbacks def func(): # do some stuff d = defer.Deferred() callSomethingElse(d) d.callback(result) newResult = yield d defer.returnValue(newResult) When I write functions that take a Deferred I typically have them return a Deferred as well, to avoid this, and other syntactic nuisances. Would you consider the following more elegant? @inlineCallbacks def func(): # do some stuff returnValue(yield callSomethingElse(succeed(result))) ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Weekly Bug Summary
Bug summary __ Summary for 2009-04-05 through 2009-04-12 Bugs opened: 22Bugs closed: 10 Total open bugs: 1076 (+12) |== Type Changes |== Priority Changes |== Component Changes |Defect: +1 |Highest: +1 |Core:-1 |Enhancement: +11 |Normal: +12 |Mail:+1 |Lowest: -1 |Names: -1 |Pb: -1 |Release Management: +0 |Web:+13 |Website: +0 |Words: +1 Total Tickets Open Tickets New / Reopened Bugs __ = Highest = [#3768] coding standard should explain the motivation for our choice of shebang line (opened by glyph) enhancement core http://twistedmatrix.com/trac/ticket/3768 = Normal = [#3751] line of code stranded after exception explicitly raised in IMAP4.client.__cbContinueAuth (opened by whysean) enhancement mail http://twistedmatrix.com/trac/ticket/3751 [#3752] twistd web options should be composable (opened by jack) enhancement webhttp://twistedmatrix.com/trac/ticket/3752 [#3753] overlay resources for twisted.web (opened by jack) enhancement webhttp://twistedmatrix.com/trac/ticket/3753 [#3754] Create benchmark runner with standalone examples for Twisted core (opened by fijal) enhancement core http://twistedmatrix.com/trac/ticket/3754 [#3755] twistd web plugin needs proxy overlays (opened by jack) enhancement webhttp://twistedmatrix.com/trac/ticket/3755 [#3756] easy configuration file support in twistd web (opened by jack) enhancement webhttp://twistedmatrix.com/trac/ticket/3756 [#3757] HTTPFactory logging should use a FilePath like object (opened by jack) enhancement webhttp://twistedmatrix.com/trac/ticket/3757 [#3758] HTTPFactory _openLogFile is overridable but private (opened by jack) defect webhttp://twistedmatrix.com/trac/ticket/3758 [#3759] twisted.python.log should use FilePath like objects (opened by jack) enhancement core http://twistedmatrix.com/trac/ticket/3759 [#3760] log file formatting logic should be separated from log writing logic (opened by jack) enhancement webhttp://twistedmatrix.com/trac/ticket/3760 [#3761] static.File's __init__ parameters missing documentation (opened by jack) defect webhttp://twistedmatrix.com/trac/ticket/3761 [#3762] static.File createSimilarFile is undocumented (opened by jack) enhancement webhttp://twistedmatrix.com/trac/ticket/3762 [#3763] twisted.web needs better resource lookup (opened by jack) enhancement webhttp://twistedmatrix.com/trac/ticket/3763 [#3764] ReverseProxyResource returns no data when / is omitted (opened by jack) defect webhttp://twistedmatrix.com/trac/ticket/3764 [#3765] twisted.web.static.File should compose a FilePath-like object rather than inheriting from FilePath (opened by glyph) enhancement webhttp://twistedmatrix.com/trac/ticket/3765 [#3766] tksupport does not allow application quit on MacOS X (opened by rowen) (CLOSED, fixed) defect core http://twistedmatrix.com/trac/ticket/3766 [#3767] quick way to go from string to a domish element (opened by twonds) enhancement words http://twistedmatrix.com/trac/ticket/3767 [#3769] problems with twisted.python.log module (opened by netskay) (CLOSED, invalid) defect core http://twistedmatrix.com/trac/ticket/3769 [#3770] twisted.web has no protection against HTTP response-splitting attacks (opened by ivank) defect webhttp://twistedmatrix.com/trac/ticket/3770 [#3771] Link to "coding standard" is broken on http://twistedmatrix.com/trac/wiki/ReviewProcess (opened by ssteiner) (CLOSED, fixed) defect websitehttp://twistedmatrix.com/trac/ticket/3771 = Lowest = [#1696] RPMs fail to build on RHEL 4 (opened by shieldszero) defect release management http://twistedmatrix.com/trac/ticket/1696 Closed Bugs __ = Normal = [#3713] Remove references to split code (opened by thijs, closed by thijs, fixed) defect core http://twistedmatrix.com/trac/ticket/3713 [#3243] Invalid Syntax while installation of Twisted 8.0.1 (opened by jprakashonnet, closed by jafo, duplicate) defect core http://twistedmatrix.com/trac/ticket/3243 [#3750] use of #!/usr/bin/python as shebang line makes Twisted unfriendly to virtualenv without installation (opened b
