[Twisted-Python] How to use ampoule?

[Chris]

Hi everyone,
I am using twisted to build a server,and the computing request maybe
costs lot of cpu resources.I have checked the maillist,it seems I can
use ampoule plugin to create another process ,I have checked the
website,and downloaded the source code.It seems there is no any document
for it.I did find a example dir,after checking the code,I was
confused.There should be two files,one for client and one for server,am
I right?the client will send the request to the server(could be in
another machine?),and the server responses.but I can't find anything
which matches what I thought.Can anyone explain a little bit to me?Or if
there is some code that would be better.
And the twisted server will receive lot of binary data from the
client,if I use ampoule,I have to send the same data to the ampoule
server again(now the twisted server acts as the ampoule client).Is the
ampoule suitable for such a kind of task? Or I should just use twisted
process module?
I don't know my understanding is correct or not,please correct me.

Best Wishes
Chris


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] twisted.web2 only can serve one connection at the same time when using scgi?

[Chris]

Hi everyone,
I am trying to use apache2 + scgi to write a application,I found
twisted.web2 supports the scgi module.I began to use this library to
write some code to test how to use it,just like this:

from twisted.web2 import server, channel, resource,http,http_headers
from twisted.application import service, strports
from twisted.internet import reactor, defer

class Test(resource.Resource):
def getResponse(self, result = None):
rep = "The output which shall be transferred to the browser."
res = http.Response( 200,
{'content-type': http_headers.MimeType('text','plain')},
rep)
print " getResponse returning: ", res
return res

def render(self,request):
print 'a new request is coming'
d = defer.Deferred()
import time
d.addCallback(lambda x:x).addCallback(self.getResponse)
reactor.callLater(10,d.callback,0)
return d

toplevel = Test()
site = server.Site(toplevel)

application = service.Application("demoserver")
s = strports.service('tcp:', channel.SCGIFactory(site))
s.setServiceParent(application)

in the render function,I used a defer to simulate a time-consuming
operation which takes ten seconds.IF in twisted.web,I can first return a
server.NOT_DONE_YET,and use request.write() and request.finish() when
the time-consuming operation is done(still use defer to do this).but
twisted.web2 only supports returning a iweb.IResponse.it seems returning
a defer is the only way.
_|| <http://python.net/crew/mwh/apidocs/twisted.web2.iweb.IResponse.html>_
here is the output:
2009-06-23 15:45:11+0800 [-] Log opened.
2009-06-23 15:45:11+0800 [-] twistd 8.2.0 (/usr/bin/python 2.5.2)
starting up.
2009-06-23 15:45:11+0800 [-] reactor class:
twisted.internet.selectreactor.SelectReactor.
2009-06-23 15:45:11+0800 [-] twisted.web2.channel.scgi.SCGIFactory
starting on 
2009-06-23 15:45:11+0800 [-] Starting factory

2009-06-23 15:45:26+0800 [SCGIChannelRequest,0,192.168.0.138] a new
request is coming
2009-06-23 15:45:36+0800 [-]  getResponse returning:

2009-06-23 15:45:36+0800 [SCGIChannelRequest,1,192.168.0.138] a new
request is coming
2009-06-23 15:45:46+0800 [-]  getResponse returning:

2009-06-23 15:45:46+0800 [SCGIChannelRequest,2,192.168.0.138] a new
request is coming
2009-06-23 15:45:56+0800 [-]  getResponse returning:


I use the 3 browsers to access the URL within 1 second,but as we all see
in the log,the server accepts the requests one by one,that is really not
acceptable.I checked a little about the twisted.web2 source code, this
render function is in a defer's callbacks chain,when I return a defer in
a defer's callbacks chain,sure,the rest of the callbacks will wait for
my defer to finish ,even I run it with reactor.callLater.so far,I feel I
can do nothing about this,anyone can help me about this?

Regards
Chris


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] problem on running apache2 + scgi

[Chris]

Hi,
I was working on a project which use twisted.web2 scgi.I chose apache2
as the front end web server.the apache2 server will receive a http POST
message which carry about 200k-1M data.I found that sometimes I get the
correct answer but sometimes there is a exception:
Traceback (most recent call last):
File "//usr/lib64/python2.5/site-packages/twisted/python/log.py", line
84, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File "//usr/lib64/python2.5/site-packages/twisted/python/log.py", line
69, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/usr/lib64/python2.5/site-packages/twisted/python/context.py",
line 59, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib64/python2.5/site-packages/twisted/python/context.py",
line 37, in callWithContext
return func(*args,**kw)
---  ---
File
"/usr/lib64/python2.5/site-packages/twisted/internet/selectreactor.py",
line 146, in _doReadOrWrite
why = getattr(selectable, method)()
File "/usr/lib64/python2.5/site-packages/twisted/internet/tcp.py", line
463, in doRead
return self.protocol.dataReceived(data)
File "/usr/lib64/python2.5/site-packages/twisted/web2/channel/scgi.py",
line 72, in dataReceived
self.request.handleContentChunk(data)
File "/usr/lib64/python2.5/site-packages/twisted/web2/http.py", line
393, in handleContentChunk
self.stream.write(data)
File "/usr/lib64/python2.5/site-packages/twisted/web2/http.py", line
331, in write
stream.ProducerStream.write(self, data)
File "/usr/lib64/python2.5/site-packages/twisted/web2/stream.py", line
667, in write
self.producer.pauseProducing()
exceptions.AttributeError: SCGIChannelRequest instance has no attribute
'pauseProducing'

it seems the error is from the twisted.web2 module,usually when the data
is very big,I will get this exception.how to avoid this exception? I
have to transfer the data in one request anyway.
Any suggestion is helpful.Thank you!

Regards
Chris


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] revisiting onboarding

[Chris]

Glyph,

I support this idea. I like the fact that it gives a clear goal (5 submitted 
patches, 5 reviews, and 2 accepted submissions) towards which people interested 
in committing can work.

-Chris (herrwolfe)

> On Feb 2, 2015, at 1:18 PM, Glyph Lefkowitz  wrote:
> 
> A few months ago, the question of an official process for obtaining commit 
> access was raised.
> 
> The discussion lead to this proposal - 
> https://twistedmatrix.com/trac/wiki/Proposal/ContributorAdvancementPath 
> <https://twistedmatrix.com/trac/wiki/Proposal/ContributorAdvancementPath>.  I 
> like that proposal, but it is still incomplete: to wit, it defines a template 
> for how roles might be defined but does not actually define any roles.  It 
> also seems to be a bit overly ambitious, just from the observed reaction of 
> nobody having the time to implement it :).
> 
> So I have a proposal for a scaled back process that nevertheless would give 
> us something official-ish:
> 
> Not all committers are actively involved in the project at all times, so we 
> should form a "committer committee" of people who are interested in 
> evaluating candidates.  If you would like to do that and you're already a 
> committer, please contact me and I'll add you to the list.  I want to do this 
> so there's somewhere to apply to which isn't just a public mailing list or 
> discussion, since public discussions can create social pressure to grant 
> someone commit just to be nice, and rejections might be perceived as mean.
> 
> Candidates should submit an application to this new list, 
> com...@twistedmatrix.com <mailto:com...@twistedmatrix.com> which is a list of 
> links to at least 10 tickets, at least 5 of which are patches they've 
> submitted, and at least 5 of which are code reviews they've done which have 
> been accepted by a committer.  At least 2 of their authored patches should 
> have gone all the way through the process to be landed.
> 
> As with the other parts of our process, if there is at least one sponsor, and 
> no objections from anyone on the committee within 7 days, any member of the 
> committee may add the committer.
> 
> New committers should then be announced on the mailing list.
> 
> This is not really an ideal process - particularly, it lacks a good way to 
> give contributors something specific and useful to do - but it's something at 
> least.  If there is general assent that this is an improvement, I'll go make 
> a wiki page and a mailing list.
> 
> -glyph
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] doWrite on twisted.internet.tcp.Port

[chris]

Hey guys,

 

Last year I hit a condition discussed in this ticket:
https://twistedmatrix.com/trac/ticket/4759 for doWrite called on a
twisted.internet.tcp.Port.  

 

I ignored it at the time since it was just on Linux, and my main platform
was Windows.  Now I'm coming back to it.  I'll add context on the problem
below, but first I want to ask a high-level, design-type question with
multiprocessing and Twisted:

 

Referencing Jean-Paul's comment at the end of ticket 4759, I read you
shouldn't fork a process (multiprocessing module) that already has a Twisted
reactor.  Understood.  But what about a parent process (not doing anything
Twisted) forking child processes, where each child process starts their own
Twisted reactor?  Is that intended to work from the Twisted perspective?

 

 

Context:

I only hit this problem on Linux, not Windows.

 

The software project (github.com/opencontentplatform/ocp) has been a lot of
fun, especially with walking the tight rope in using multi-processing,
multi-threading, and Twisted reactors.  The main controller process kicks
off about 10 child processes, each doing different types of work.  In
general though, the child processes individually start a Twisted reactor,
connect to Kafka, connect to a database, use a shared REST API, and some
listen for connecting clients to accomplish work.

 

I test on Linux about once a year, so too many changes to rollback and
figure out that way.  It was working on Linux 2 years ago, but last year's
testing and current testing, receive the doWrite error.  It continues
running fine on Windows.  I've gone back about 2 years of versions with
Python3, Twisted, and dependent libs. on both Windows and Linux.  Every
version change yields the same result - continues to work on Windows and
continues to hit the error on Linux.  So something I added has caused Linux
to throw that error.  

 

I'm not explicitly sharing much between the main process and the sub
processes.  They are spun up with (1) a shared multiprocessing.Event() - to
have the main process shut the children down, (2) their own unique
multiprocessing.Event() - to have the child processes notify back to the
parent, and (3) a deep copy of a dictionary (containing a bunch of settings
that remain constant).  The main process uses twisted.logger, but for
testing I strip that out to remove any twisted imports in the main process.
So I'm not importing anything Twisted in the main process, and I don't
believe I'm explicitly sharing something I shouldn't.  Seems like something
is implicitly being exposed/shared across Linux child processes, that aren't
on Windows.

 

The tracebacks come through on Linux (sometimes randomly), on the console of
the parent controller process.  No need to paste here, since it's the same
as the ticket shows.  I can't reliably reproduce the problem, but I know if
I stop/start client connections
(ServerFactory/twisted.internet.interfaces.IReactorTCP and
twisted.internet.protocol.ReconnectingClientFactory) then it will eventually
happen.  I need to devote time at whittling down the code and attempting to
create a reliable test case. if even possible.

 

The error is slightly different when running HTTP vs HTTPS, but the story is
the same.  It cripples whatever child process that hits it, from doing much
of anything thereafter.  Not much luck with troubleshooting.  The tracebacks
do not include a calling function from my code, to tell me where to start.
And it happens across different child process types, so not the same one
each time.  When I throw debuggers on the child processes, the problem seems
to mask itself.  Well, at least I didn't hit the problem over the last 3
days using pudb and stepping through code at breakpoints.

 

I'm absolutely open to suggestions for troubleshooting, but first wanted to
take a HUGE step back and ask a design question regarding Twisted and
multiprocessing.

 

Thanks!

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] doWrite on twisted.internet.tcp.Port

[chris]

Hi Jean-Paul,

 

Thank you very much for the detailed answer.  And my appologies for not 
providing OS details; I’ve tested on CentOS and RedHat EL variants, not FreeBSD 
as the ticket discussed.  Looks like Red Hat (EL 7.6) is using epoll reactor, 
and the Windows side is using the select reactor.

 

Thanks for the direction on checking out sys.modules.  To avoid the reactor 
being loaded in the parent process, I can presumably move twisted imports 
within the multiprocessing child modules (from top, down into the run() 
functions).  I will see how far I need to go (e.g. if I can continue using 
Twisted’s JSON logging or if absolutely everything should be isolated until 
after child process startup).  But knowing I need to head that direction for 
epoll or other potential reactor conflicts - is very helpful.

 

Reminds me of the GI Joe cartoon in the early 1980’s that would end with, 
“knowing is half the battle.”

 

-Chris

 

 

From: Twisted-Python  On Behalf Of 
Jean-Paul Calderone
Sent: Friday, September 11, 2020 1:28 PM
To: Twisted general discussion 
Subject: Re: [Twisted-Python] doWrite on twisted.internet.tcp.Port

 

On Fri, Sep 11, 2020 at 1:34 PM mailto:ch...@cmsconstruct.com> > wrote:

Hey guys,

 

Last year I hit a condition discussed in this ticket: 
https://twistedmatrix.com/trac/ticket/4759 for doWrite called on a 
twisted.internet.tcp.Port.  

 

I ignored it at the time since it was just on Linux, and my main platform was 
Windows.  Now I’m coming back to it.  I’ll add context on the problem below, 
but first I want to ask a high-level, design-type question with multiprocessing 
and Twisted:

 

Referencing Jean-Paul’s comment at the end of ticket 4759, I read you shouldn’t 
fork a process (multiprocessing module) that already has a Twisted reactor.  
Understood.  But what about a parent process (not doing anything Twisted) 
forking child processes, where each child process starts their own Twisted 
reactor?  Is that intended to work from the Twisted perspective?

 

To answer the asked question, I don't think there is rigorous (or even casual) 
testing of very much of Twisted in the context of "some Twisted code has been 
loaded into memory and then the process forked".  So while it seems like a 
reasonable thing, I wouldn't say there's currently much effort being put 
towards making it a supported usage of Twisted.  Of course this can change at 
almost any moment if someone decides to commit the effort.

 

To dig a bit further into the specific problem, even if you only import the 
reactor in the parent process and then fork a child and try to start the 
reactor in the child, I strongly suspect epollreactor will break.  This is 
because the epoll object is created by reactor instantiation (as opposed to 
being delayed until the reactor is run).  epoll objects have a lot of weird 
behavior.  See the Questions and Answers section of the epoll(7) man page.

 

I don't know if this is the cause of your particular expression of these 
symptoms (it certainly doesn't apply to the original bug report which is on 
FreeBSD where there is no epoll) but it's at least a possible cause.

 

This could probably be fixed in Twisted by only creating the epoll object when 
run is called.  There's nothing particularly difficult about that change but it 
does involve touching a lot of the book-keeping logic since that all assumes it 
can register file descriptors before the reactor is started (think 
reactor.listenTCP(...); reactor.run()).

 

I'm not sure but it may also be the case that only delaying creation of the 
waker until the reactor starts would also fix this.  This is because as long as 
the epoll object remains empty a lot of the weird behavior is avoided and the 
waker is probably the only thing that actually gets added to it if you're just 
importing the reactor but not running it before forking.

 

Alternatively, your application should be able to fix it by studiously avoiding 
the import of twisted.internet.reactor (directly or transitively, of course).  
You could add some kind of assertion about the state of sys.modules immediately 
before your forking code to develop some confidence you've managed this.

 

And if this is really an epoll problem then switching to poll or select reactor 
would also presumably get rid of the issue.

 

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] doWrite on twisted.internet.tcp.Port

[chris]

[Closing the loop for future contextual searches]

 

This solved the problem.  Moving all Twisted reactor-related imports inside 
overloaded multiprocessing.Process.run() functions, allows a single controller 
process to manage many Twisted reactors running in child processes.  

 

Thanks again.

-Chris

 

 

From: Twisted-Python  On Behalf Of 
ch...@cmsconstruct.com
Sent: Saturday, September 12, 2020 12:07 PM
To: 'Twisted general discussion' 
Subject: Re: [Twisted-Python] doWrite on twisted.internet.tcp.Port

 

Hi Jean-Paul,

 

Thank you very much for the detailed answer.  And my appologies for not 
providing OS details; I’ve tested on CentOS and RedHat EL variants, not FreeBSD 
as the ticket discussed.  Looks like Red Hat (EL 7.6) is using epoll reactor, 
and the Windows side is using the select reactor.

 

Thanks for the direction on checking out sys.modules.  To avoid the reactor 
being loaded in the parent process, I can presumably move twisted imports 
within the multiprocessing child modules (from top, down into the run() 
functions).  I will see how far I need to go (e.g. if I can continue using 
Twisted’s JSON logging or if absolutely everything should be isolated until 
after child process startup).  But knowing I need to head that direction for 
epoll or other potential reactor conflicts - is very helpful.

 

Reminds me of the GI Joe cartoon in the early 1980’s that would end with, 
“knowing is half the battle.”

 

-Chris

 

 

From: Twisted-Python mailto:twisted-python-boun...@twistedmatrix.com> > On Behalf Of Jean-Paul 
Calderone
Sent: Friday, September 11, 2020 1:28 PM
To: Twisted general discussion mailto:twisted-python@twistedmatrix.com> >
Subject: Re: [Twisted-Python] doWrite on twisted.internet.tcp.Port

 

On Fri, Sep 11, 2020 at 1:34 PM mailto:ch...@cmsconstruct.com> > wrote:

Hey guys,

 

Last year I hit a condition discussed in this ticket: 
https://twistedmatrix.com/trac/ticket/4759 for doWrite called on a 
twisted.internet.tcp.Port.  

 

I ignored it at the time since it was just on Linux, and my main platform was 
Windows.  Now I’m coming back to it.  I’ll add context on the problem below, 
but first I want to ask a high-level, design-type question with multiprocessing 
and Twisted:

 

Referencing Jean-Paul’s comment at the end of ticket 4759, I read you shouldn’t 
fork a process (multiprocessing module) that already has a Twisted reactor.  
Understood.  But what about a parent process (not doing anything Twisted) 
forking child processes, where each child process starts their own Twisted 
reactor?  Is that intended to work from the Twisted perspective?

 

To answer the asked question, I don't think there is rigorous (or even casual) 
testing of very much of Twisted in the context of "some Twisted code has been 
loaded into memory and then the process forked".  So while it seems like a 
reasonable thing, I wouldn't say there's currently much effort being put 
towards making it a supported usage of Twisted.  Of course this can change at 
almost any moment if someone decides to commit the effort.

 

To dig a bit further into the specific problem, even if you only import the 
reactor in the parent process and then fork a child and try to start the 
reactor in the child, I strongly suspect epollreactor will break.  This is 
because the epoll object is created by reactor instantiation (as opposed to 
being delayed until the reactor is run).  epoll objects have a lot of weird 
behavior.  See the Questions and Answers section of the epoll(7) man page.

 

I don't know if this is the cause of your particular expression of these 
symptoms (it certainly doesn't apply to the original bug report which is on 
FreeBSD where there is no epoll) but it's at least a possible cause.

 

This could probably be fixed in Twisted by only creating the epoll object when 
run is called.  There's nothing particularly difficult about that change but it 
does involve touching a lot of the book-keeping logic since that all assumes it 
can register file descriptors before the reactor is started (think 
reactor.listenTCP(...); reactor.run()).

 

I'm not sure but it may also be the case that only delaying creation of the 
waker until the reactor starts would also fix this.  This is because as long as 
the epoll object remains empty a lot of the weird behavior is avoided and the 
waker is probably the only thing that actually gets added to it if you're just 
importing the reactor but not running it before forking.

 

Alternatively, your application should be able to fix it by studiously avoiding 
the import of twisted.internet.reactor (directly or transitively, of course).  
You could add some kind of assertion about the state of sys.modules immediately 
before your forking code to develop some confidence you've managed this.

 

And if this is really an epoll problem then switch

Re: [Twisted-Python] SURVEY: Have you submitted a patch to Twisted and it never got in?

[chris]

Hi all,

On 01.07.2011 18:36, Phil Mayers wrote:
> However, more constructively (less whiney!) some tickets languished in
> "make these tiny cleanups" and that's just incredibly painful in the
> current setup, with SVN and Trac mediating things.
>
> I've got absolutely no interest in pulling SVN head, writing a patch,
> submitting it as an attachment via Trac and *then* being told "ok, I've
> created this branch. Go off and learn how to do branches in a crappy old
> centralised VCS, and in a way compatible with UQDS, re-do your patch in
> a branch, then send another diff in as a file"

I absolutely agree with Phil here.
The twisted code and contribution standards are so high that patches 
from new contributors (like myself) are bound to be rejected/resubmitted 
at least once, maybe more. Don't get me wrong, I believe high standards 
are a good thing, but doing continuous development based on tools like 
svn and trac is really painful and it's really difficult to motivate 
yourself to work on a once rejected ticket.

That being said, I believe that the move to a DVCS is a smart move for 
any project looking for continuous community contribution, because IMHO 
they simply allow for a more developer friendly process for all 
involved, thus making the whole review process a more friendly and less 
discouraging thing.

Cheers,
Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] naming question before we end up committed...

[Chris]

+1 for twisted.logger.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Software that uses Twisted

[Chris Hallman]

I don't mean to be picky, but whomever ever wrote that site used poor
English grammar. To be taken more serious, you need to correct the
grammatical mistakes.



chris



On Tue, Jan 13, 2009 at 4:32 PM,  wrote:

> Description: A control panel for Windows visible form any part of the
> world through web browsers
> URL: www.winctp.com
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Software that uses Twisted

[Chris Hallman]

I apologize to all on the mailing. I meant to reply only to the original
sender, not all.



On Wed, Jan 14, 2009 at 6:33 PM, Vasil Vangelovski
wrote:

> Would you be interested in getting internet providings that is better
> than your current providings?
> Seriously
>
> On Wed, Jan 14, 2009 at 6:35 PM, s s 
> wrote:
> > With seriously, taken you will be more.
> >
> > S
> >
> > On Jan 14, 2009, at 11:42 AM, Giles Antonio Radford wrote:
> >
> >> Surely "seriously"?
> >>
> >> HTH, HAND,
> >>
> >> Moof
> >>
> >> On Wed, Jan 14, 2009 at 4:30 PM, Chris Hallman 
> wrote:
> >>>
> >>> I don't mean to be picky, but whomever ever wrote that site used poor
> >>> English grammar. To be taken more serious, you need to correct the
> >>> grammatical mistakes.
> >>>
> >>>
> >>>
> >>> chris
> >>>
> >>>
> >>>
> >>> On Tue, Jan 13, 2009 at 4:32 PM, 
> wrote:
> >>>>
> >>>> Description: A control panel for Windows visible form any part of the
> >>>> world through web browsers
> >>>> URL: www.winctp.com
> >>>>
> >>>> ___
> >>>> Twisted-Python mailing list
> >>>> Twisted-Python@twistedmatrix.com
> >>>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> >>>>
> >>>
> >>>
> >>> ___
> >>> Twisted-Python mailing list
> >>> Twisted-Python@twistedmatrix.com
> >>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> >>>
> >>>
> >>
> >> ___
> >> Twisted-Python mailing list
> >> Twisted-Python@twistedmatrix.com
> >> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> >
> >
> > ___
> > Twisted-Python mailing list
> > Twisted-Python@twistedmatrix.com
> > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> >
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Using sqlalchemy in twisted.

[Chris Foster]

I think SQLAlchemy's ORM might work fine with Twisted.  Check out http://foss.eepatents.com/sAsync/ 
 .  sAsync doesn't appear to be widely used, but I got the examples  
to run with some minor changes to the sqlite connection.  I'm hoping  
to try something useful in the next week or two.


On Mar 4, 2009, at 11:04 AM, Valentino Volonghi wrote:


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1


On Mar 4, 2009, at 3:28 AM, Peter Cai wrote:


The code doesn't work.   When I limit the thread numbers to 1

  reactor.suggestThreadPoolSize(1)

Everything goes fine.  Other wise the server would be blocked and  
must

be killed by "kill 9 ...".

The result conflicts with my understanding of sqlalchemy.  Since I
don't share any object between threads, there should be no problem!


I'm pretty sure you can't say this with full certainty and actually  
you are

just wrong based on the code you showed. deferToThread doesn't use
the same thread every time you call it, there's absolutely no  
guarantee
in that and sqlalchemy keeps state around in that thread related to  
that
object it returned. If you want to do any operations you need either  
to
detach the object from the session before returning it and do any  
modification
on the same object in another thread after reattaching it (pretty  
cumbersome).


Or write your own threadpool that gives names to threads so that you  
can
have a guarantee that you always call the same thread when working  
with

a bunch of objects.

Nonetheless though sqlalchemy is a huge project and I'm pretty sure  
it has

some deadlocks and race conditions around which means that even taking
care of these issues you'll have other problems when dealing with  
the orm.


If you want to use sqlalchemy don't use its orm but just the query  
builder,
it's the only sane way to integrate with twisted. Which anyway IMHO  
is the
best way to use it anyway because it uses a lot less memory since it  
doesn't
have to always cache objects because you control everything and can  
make

that call when you really need it.


Ah  It always have risk to use something you haven't tried
before 
I think I have no choice but always set thread pool size to 1 ...



Not entirely true.

- --
Valentino Volonghi aka Dialtone
Now running MacOS X 10.5
Home Page: http://www.twisted.it
http://www.adroll.com

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (Darwin)

iEYEARECAAYFAkmu0NoACgkQ9Llz28widGXBawCg32svBsLqsIRLzvzOThgR4sA0
5UkAoIgNfyUDPl9c0nwSud0sem3aKjz5
=2XIX
-END PGP SIGNATURE-

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Twisted mirror down?

[Chris Foster]

Who/where/how do we report a problem with http://tmrc.mit.edu/mirror/twisted/Twisted/8.2/Twisted-8.2.0.tar.bz2 
 ? 


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] How to do basic authentication on twisted web

[Chris Adams]

Hi there,

I'm new to twisted, so please excuse the noobish question, but:

I'm using twisted-web on an openwrt computer, using busybox, and after
looking through the docs, I'm a little unclear about how to implement
something like apache's htpasswd style authentication.

Using apache, I'd do something like:

htpasswd -c passwrods admin topSecretPassword

What's the simplest way to add a simple check for a username and
password like this using twisted web?

Where possible, I'd like to keep all of this inside twisted rather
than rely on installing other binaries, that might not work within the
confined of a small openwrt install.

C

-- 
---
I'm currently only checking my email at 9am, midday and at 4pm.
If you need a response from me urgently, please call or text my
mobile, or contact me via Skype (chris.d.adams).
---
Chris Adams
Stemcel Studios
The Hub
5 Torrens Street
London
EC1V 1NQ

email: ch...@stemcel.co.uk
web:  www.stemcel.co.uk
twitter:chris_d_adams
skype: chris.d.adams
mob: 07974 368 229
tel: 0207 558 8971

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How to do basic authentication on twisted web

[Chris Adams]

Wow, thanks fo the quick replies guys.

I've tried to combine this with the example earlier, with some
annotations as to what I think is going on, but I'm on somewhat shaky
ground here.

Do these annotations sum up roughly what is going on here?

My annotated example is also up on github - http://gist.github.com/169102


The original on twisted matrix
http://twistedmatrix.com/projects/web/documentation/examples/webguard.py




2009/8/17 Reza Lotun :
> Hi Colin,
>
>> Where is Portal and SimpleRealm in your example? Those are actually quite
>> large constructs which are crucial to guard doing anything useful at all.
>
> Sorry, this snippet of code I wrote tracked the webguard.py example
> that I linked to.
>
> from twisted.cred.portal import IRealm, Portal
>
> class SimpleRealm(object):
>    """
>    A realm which gives out L{GuardedResource} instances for authenticated
>    users.
>    """
>    implements(IRealm)
>
>    def requestAvatar(self, avatarId, mind, *interfaces):
>        if resource.IResource in interfaces:
>            return resource.IResource, GuardedResource(), lambda: None
>        raise NotImplementedError()
>
> Does that make sense now?
>
>> Since there are numerous circumstances that people need Htauth, it is
>> implemented as follows
>
> Ok, of course you can do that if you want. But you still need to be
> able to *verify* credentials.
>
> Anyway, to each their own.
>
> Reza
>
>
>
> --
> Reza Lotun
> mobile: +44 (0)7521 310 763
> email:  rlo...@gmail.com
> work:   r...@tweetdeck.com
> twitter: @rlotun
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>



-- 
---
I'm currently only checking my email at 9am, midday and at 4pm.
If you need a response from me urgently, please call or text my
mobile, or contact me via Skype (chris.d.adams).
---
Chris Adams
Stemcel Studios
The Hub
5 Torrens Street
London
EC1V 1NQ

email: ch...@stemcel.co.uk
web:  www.stemcel.co.uk
twitter:chris_d_adams
skype: chris.d.adams
mob: 07974 368 229
tel: 0207 558 8971

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How to do basic authentication on twisted web

[Chris Adams]

Thanks Reza,

> Generally it looks fine - but I should mention these random snippets
> of code were given with the assumption that a .tac file is being used.
> A .tac file is basically a "runner" script or driver for your twisted
> program - it works in conjunction with the twistd command-line utility
> and does fun stuff like automatic daemonization and reactor selection.
>



Automatic daemonization and monitoring? This sounds like exactly what I'm after.

I've been calling twisted like so for a project: explicitly setting
the process id, logger and source file here. And relying on a separate
cron script to check if all is well:

twistd --pidfile=$PIDFILE --syslog --prefix=program_name --python
program_name.py

Would using a .tac file make some of these flags redundant?

All I could found about .tac files is here -
http://twistedmatrix.com/projects/core/documentation/howto/application.html

Are there any other resources where I can found out about the
advantages of using .tac files instead the way I've bee doing stuff?

-- 
---
I'm currently only checking my email at 9am, midday and at 4pm.
If you need a response from me urgently, please call or text my
mobile, or contact me via Skype (chris.d.adams).
---
Chris Adams
Stemcel Studios
The Hub
5 Torrens Street
London
EC1V 1NQ

email: ch...@stemcel.co.uk
web:  www.stemcel.co.uk
twitter:chris_d_adams
skype: chris.d.adams
mob: 07974 368 229
tel: 0207 558 8971

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] deciding to use twisted or not

[Chris Adams]

If you're using a linux based system, you may have some luck setting
up syslogger to forward logging packets to the remote ip address, and
running the twisted daemon on the other box, and sending notifications
if the heartbeat from the monitored machine stops.

I was working on a project recently to set up a monitor a RFID based
access control system for building, that runs on
twisted/openwrt/linux, we used an approach like this.

The link below shows a sample twisted python file that runs on a
monitoring machine.

http://github.com/derfred/doord/blob/cd300a1cde930c07cd13d98be3e45cb89df79809/log_watcher.py

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] deferring result to PB a callRemote method

[Chris Laws]

I am writing a tool for work that will run various software development aids
such as message capturing and diagnostic control of system processes.
My current design strategy is to implement these aids as plugins (not
twisted plugins) to a generic plugin runner rather than stand alone
applications.
The plugins are spawned as separate processes by the plugin-runner and
communicate using Perspective Broker during the plugin installation and
shutdown phases.

Some plugins may take a while to shutdown as they need to close
sockets, finalize files, move large files, etc.
I would like to instruct the plugin to shutdown and be told when it is
finally ready to be shut down.

To date all my callRemote methods have effectively returned immediately. For
example:

def remote_get_name(self):
return self.name

My question to the list was going to be: Is there a pattern/example I could
follow where I could call plugin.callRemote("shutdown") on a plugin and not
return a result to the plugin-runner until the plugin has completed all
it's, potentially, long running activities? However, while I was trying to
write a small code snippet that would demonstrate what I was wanting, I
think I got it working.
I think the simple answer to my question is to just return a deferred as the
result to the callRemote("shutdown") method and trigger it as normal.
Google is my friend but I could not find examples of this usage. Is there
any references to this usage in the twisted docs?

Below (and attached in case the formatting goes screwy) is a short example
which emulates a long running shutdown activity performed by the plugin
prior to shutdown.
It seems to delay the processing of the callRemote("shutdown") result until
the plugin has completed it's long running activity.
I have omitted the separate process stuff as it didn't seem relevant for
this snippet.

Is the following code snippet the standard/normal way to defer the return
result of a callRemote method call?
If this is the normal way, how does triggering the deferred on the plugin
(client) side also trigger the same/copy deferred returned to the
plugin-runner (server)?
Is this PB magic, somehow managing deferreds across the PB interface?

Regards,
Chris



from twisted.internet import reactor, defer
from twisted.spread import pb
import datetime

class PluginClient(pb.Referenceable):
""" Plugin client interface exposed to PluginServer's """

def __init__(self, shutdownCallback):
self.shutdownHandler = shutdownCallback

def remote_shutdown(self):
""" Instruct Plugin to shutdown """
print "plugin instructed to shutdown"
d = defer.Deferred()
self.shutdownHandler(d)
return d

class PluginServer(pb.Root):
""" Plugin server interface exposed to PluginClient's """

def remote_set_client_perspective(self, pluginRef):
print "plugin-runner got client reference"
reactor.callLater(1.0, self.shutdown_plugin, pluginRef)

def shutdown_plugin(self, pluginRef):

def pluginShutdownCompleted(result, startTime):
endTime = datetime.datetime.now()
print "Plugin shutdown took %s to complete." % (endTime -
startTime)
return result

print "plugin-runner asking plugin to shutdown"
d = pluginRef.callRemote("shutdown")
d.addCallback(pluginShutdownCompleted, datetime.datetime.now())
d.addCallback(self.shutdown)

def shutdown(self, _):
reactor.stop()


def startPluginServer(port):
""" Start a plugin communications server """
print "starting server"
reactor.listenTCP(port=port,
  factory=pb.PBServerFactory(PluginServer()),
  interface='localhost')


def startPluginClient(port, shutdownHandler):
""" Start a plugin communications client """

def gotServerPerspective(serverPerspective, pluginPerspective):
""" Give the plugin-runner this client's perspective """
serverPerspective.callRemote("set_client_perspective",
pluginPerspective)
return serverPerspective

print "starting plugin"
client = PluginClient(shutdownHandler)
factory = pb.PBClientFactory()
reactor.connectTCP(host='localhost', port=port, factory=factory)
return factory.getRootObject().addCallback(gotServerPerspective, client)

if __name__ == "__main__":
port = 42155

def longRunningAction(d, countDown=10):
""" Emulate long running shutdown activities """
print "shuting down in %i seconds" % countDown
if countDown == 0:
d.callback(True)
else

[Twisted-Python] deferring result to PB a callRemote method

[Chris Laws]

> >* >Google is my friend but I could not find examples of this usage.
Is
*> >* there
*> >* >any references to this usage in the twisted docs?
*> >*
*> >*
*> >* Huh, surprisingly not in the primary PB docs.  The Twisted “finger”
*> >* tutorial
*> >* does do this, though, if you read it carefully enough.  See
finger21.tac in
*> >* <
http://twistedmatrix.com/documents/current/core/howto/tutorial/pb.html>;
*> >* the
*> >* remote_* methods of PerspectiveFingerFromService delegate to
self.service,
*> >* which
*> >* is FingerService, which returns Deferreds from its getUser and
getUsers
*> >* methods.
*> >*
*> >*
*> >*
*> It sems like this should be discussed in the PB docs.  Maybe someone
should
> file a ticket..
>
> (hint, hint)

Done, #4228.

Thanks for the informative response to my question.

Regards,
Chris
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 10.0.0 released

[Chris Withers]

Jonathan Lange wrote:
>  * A new Windows installer that ships without zope.interface

Interesting, why?
Is zope.interface not used/required anymore?
If so, why is it shipping in the non-windows version?

FWIW, zope.interface:
- has binaries available for windows
- has pure python fall-backs for when no binaries are available, 
regardless of platform (ie: GAE)

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 9.0.0?

[Chris Withers]

exar...@twistedmatrix.com wrote:
> Perhaps easy_install isn't the best way to manage deployments to your 
> production server?

No, but buildout is...

However, this process of discovering packages (hit pypi, scrape web pags 
if the distro isn't on pypi) is the only solution the python community 
(be it pip, easy_install, virtualenv or buildout) is using to download 
packages...

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
 - http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 9.0.0?

[Chris Withers]

exar...@twistedmatrix.com wrote:
> On 04:53 pm, kevin.h...@gmail.com wrote:
>> Is there any particular reason Twisted doesn't upload packages to PyPI
>> itself?
> 
> Messing about with web forms is no fun and "setup.py register" has 
> issues.

What issues? I've never had problems with either the register or upload 
commands...

Having the distributions on PyPI makes the OP's problem go away, and 
reduces the maintenance burned for you guys who release twisted...

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
 - http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 9.0.0?

[Chris Withers]

Jonathan Lange wrote:
> I don't know how the PyPI announcement was done (although I'd love
> to!), but I think you can announce new versions without
> decommissioning old ones.

PyPI already does the right thing w.r.t. "hide previous versions" 
(either way works, it doesn't affect tools like easy_install) but ONLY 
if you put your distros on PyPI...

...obviously PyPI has not knowledge of your website and so can't do 
anything about it.

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
 - http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 9.0.0?

[Chris Withers]

Tim Allen wrote:
> On Thu, Mar 04, 2010 at 01:55:50PM -0500, Glyph Lefkowitz wrote:
>> Now that our release process is better documented (THANK YOU JML), you
>> might be able to have a look at
>> <http://twistedmatrix.com/trac/wiki/ReleaseProcess> and figure out
>> where 'setup.py upload' could fit in without uploading a tarball that
>> is _not_ actually our official release.
> 
> If "setup.py upload" involves "setup.py sdist" then I guess this is
> ticket #4138 again.

No, it doesn't... see the release process I use for my own packages here:

http://packages.python.org/errorhandler/development.html#making-a-release

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
 - http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 9.0.0?

[Chris Withers]

Tim Allen wrote:
> ...so presumably sdist *is* a prerequisite for upload. Or are you saying
> that you can store any old file as dist/$PACKAGE-$VERSION.tar.gz and
> "setup.py upload" will ship it to PyPI?

I think that's true, but I admit, I haven't checked myself ;-)

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
 - http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 10.0.0 released

[Chris Withers]

exar...@twistedmatrix.com wrote:
> Also, I'm pretty sure that the 9.0 Windows installer _also_ didn't 
> include zope.interface.  So this isn't a new situation with 10.0.
>> Is zope.interface not used/required anymore?
>> If so, why is it shipping in the non-windows version?
> 
> It doesn't ship in any of the packages for other platforms, either (nor 
> do any of our other external dependencies).

Ah, okay, I guess my question becomes: why was it ever bundles in the 
windows distribution then?

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
 - http://www.simplistix.co.uk


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 10.0.0 released

[Chris Withers]

Thomas Hervé wrote:
> Le jeudi 11 mars 2010 à 10:08 +0000, Chris Withers a écrit :
>> exar...@twistedmatrix.com wrote:
>>> Also, I'm pretty sure that the 9.0 Windows installer _also_ didn't 
>>> include zope.interface.  So this isn't a new situation with 10.0.
>>>> Is zope.interface not used/required anymore?
>>>> If so, why is it shipping in the non-windows version?
>>> It doesn't ship in any of the packages for other platforms, either (nor 
>>> do any of our other external dependencies).
>> Ah, okay, I guess my question becomes: why was it ever bundles in the 
>> windows distribution then?
> 
> Because Zope didn't provide a binary version of zope.interface for
> Windows at that time,

Really? To my knowledge, Zope has provided windows binaries for 
zope.interface prettymuch since it came into existence...

(I know, I'm sure I used to compile them for a time ;-) )

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
 - http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] SQLAlchemy, Twisted, and sAsync

[Chris Withers]

Matthew Williams wrote:
>  From previous posts to this and other lists, it seems that ORMs and  
> threads don't get along too well...

What makes you think that?

> and, as far as I can tell, there's  
> no way to get away from threads if you don't want longish queries to  
> block your entire application.

Right, SQLAlchemy doesn't play nicely with *non-threaded* environments, 
from my understanding, which may well be wrong ;-)

> It took me quite some time to piece together everything I could find  
> related to sAsync (which seems to be a dead but functional project),  
> so I threw up a quick Trac page for it at http://sasync.org.

Cool. What is it you're doing that needs to mix Twisted and SQLAlchemy?

cheers,

Chris

PS: I've CC'ed the SQLAlchemy list in as well as I think people there 
may want to add to this discussion...

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [sqlalchemy] Re: SQLAlchemy, Twisted, and sAsync

[Chris Withers]

Matthew Williams wrote:
> 
> "It's much trickier if you want to use the ORM, unless you are very
> careful to fully eager load every thing in any possible database
> operation if you have need of the information subsequently in your
> twisted code. Otherwise you may block unexpectedly simply when
> accessing your objects, and end up with database operations from the
> main twisted thread."
> 
> So perhaps I should have said "SQL Alchemy's ORM and threads don't get 
> along too well"... that's not to say it's impossible, you just have to 
> be exceedingly careful how you use it.

I think you have the wrong end of the stick.
SQLAlchemy and threads play fine, SQLAlchemy and Twisted's asynchronous 
model, which refuses to use threads on principle, do not, for the 
reasons you describe.

>>> It took me quite some time to piece together everything I could find  
>>> related to sAsync (which seems to be a dead but functional project),  
>>> so I threw up a quick Trac page for it at http://sasync.org.
>>
>> Cool. What is it you're doing that needs to mix Twisted and SQLAlchemy?
> 
> The project (an internal project) doesn't really *need* to mix them... I 
> could just use mysqldb.

Heh, wrong end of the stick again; my question was why you needed to use 
Twisted ;-)

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
 - http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [sqlalchemy] Re: SQLAlchemy, Twisted, and sAsync

[Chris Withers]

King Simon-NFHD78 wrote:
> The solution to this is either to eager-load all the attributes you
> think you are going to need before handing the instance off to another
> thread (difficult), or (probably better) to detach (expunge) the
> instance from thread A's session.

Are there any recommended code examples around for doing this?

Once detatched, are these objects (I'm guessing for most people they'll 
be instances of declaratively mapped models) pickleable?

> Thread B should then merge the object
> into its own session (using the load=False flag so that it doesn't
> needlessly requery the database).

Good code examples of this around too?

cheers

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
 - http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] twisted vs wsgi

[Chris Withers]

Hi All,

I've managed to cobble this together as a way of starting a BFG app 
under Twisted's wsgi server:

"""
from twisted.web.wsgi import WSGIResource
from twisted.web.server import Site
from twisted.internet import reactor

from webob import Response
from repoze.bfg.configuration import Configurator

def hello_world(request):
 return Response('Hello world!')

if __name__ == '__main__':
 config = Configurator()
 config.begin()
 config.add_view(hello_world)
 config.end()

 resource = WSGIResource(reactor, reactor.getThreadPool(),
 config.make_wsgi_app())
 factory = Site(resource, 'server.log')
 reactor.listenTCP(8080, factory)
 reactor.run()
"""

I struggled to find good narrative docs on using Twisted as a WSGI 
server. Where should I have been looking and have I dropped any clangers 
in the above?

cheers,

Chris


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] data update to multiple clients

[Chris Cusack]

Probably another newbie question but after much effort I am not  
progressing on the following.

I am trying to design a system where I have a microprocessor  
periodically feeding data on a serial connection to a server script. I  
would then like the server script to notify one or more client scripts  
of the changed data across a local network. I have tried a few simple  
tcp server and client script examples which communicate well for a  
single call but do not seem to suit my application which is not client  
event driven. I considered having each client connect to the server  
every ?? seconds to query for new data but there may be a better way.

Can anyone suggest a suitable design approach using twisted? Is there  
any example code available that may be similar.

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] getting "connection per thread" behaviour

[Chris Withers]

Hi All,

I was asking about this on irc but had to leave in a hurry so thought 
I'd mail here...

So, I have a pool of wsgi threads being run by Twisted's wsgi server 
that want to send stomp messages. The stomp client is Twisted-based 
(http://code.google.com/p/broadwick/source/browse/trunk/broadwick/messaging/twistedstomp.py).
 
I want to have as few stomp connections as possible, maxing out at one 
per thread.

Bonus complications:

- each wsgi thread has a logical transaction, that needs to marry up 
with the stomp logical transaction, so I can't just have one stomp 
connection per thread

- I don't know how much overhead there is in creating the stomp 
connection, so I'd like to open as few as possible but keep them open 
and re-use them across wsgi requests.

I found:

http://pypi.python.org/pypi/txconnpool

Anyone know how good that is and if it's useful here?

All help gratefully received!

Chris


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] SQLAlchemy and Twisted

[Chris Withers]

Doug Farrell wrote:
> 
> I’ve been doing some searching about how to get SQLAlchemy and Twisted 
> working together in a Twisted application. 

Short version: to be safe, anything that touches any SQLAlchemy-mapped 
object needs to be run in its own thread. Any query or access of an 
attribute of a mapped object may result in a blocking sql query. (aka: 
twisted doesn't play nice with orms)

> definitive answer. The most promising one I’ve run across concerns 
> running the SQLAlchemy queries in a separate process (rather than a 
> separate thread) and communicating the queries between the Twisted 
> application in one process and the SQLAlchemy application in another. 

That seems a little odd.
What would be the IPC?
How would the "sqlachemy application" be run?

> 1)  Would the SQLAlchemy process also be a Twisted application with 
> all the queries running as deferreds in the main thread, and blocking?

What do you men by "all the queries"?

> Thanks in advance for any help!

In my case, since most of the app I'm working on is "web requested" 
(either xmlrpc or http), I just agve up and used a good wsgi stack 
(repoze.bfg in my case) and munge other incoming requests into wsgi 
requests.

Twisted's wsgi server runs each request in its own thread, so be it.

cheers,

Chris


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] How do I get xUnit output from trial for Hudson?

[Chris Withers]

Hi All,

How can I get trial to emit output that makes Hudson happy?

cheers,

Chris


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How do I get xUnit output from trial for Hudson?

[Chris Withers]

Chris Withers wrote:
> How can I get trial to emit output that makes Hudson happy?

Er, shoot me, forgot I'd asked this before.

No-one written a junitxml reporter for trial?
Any guesses how hard that'd be?

cheers,

Chris


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How do I get xUnit output from trial for Hudson?

[Chris Withers]

Jonathan Lange wrote:
> Not very.
> 
> "trial --reporter=subunit  | subunit2junitxml" works well for me,
> although I am running on an OS that supports pipes :)

...and on which subunit compiles ;-)

*cough*notwindows*cough

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
 - http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] any change of the .exe builds on PyPI?

[Chris Withers]

ie: http://pypi.python.org/pypi/Twisted/10.1.0

It's great that they're being done now, but easy_install and friends on 
Windows still fail to find them unless they're on PyPI :-(

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] any change of the .exe builds on PyPI?

[Chris Withers]

On 15/09/2010 18:12, Chris Withers wrote:
> ie: http://pypi.python.org/pypi/Twisted/10.1.0
>
> It's great that they're being done now, but easy_install and friends on
> Windows still fail to find them unless they're on PyPI :-(

Oh, and I'm afraid the 10.1.0 ones are useless since they're not named 
correctly, from the package tools point of view :-(

We have: Twisted-10.1.0.winxp32-py2.5.exe
We need: Twisted-10.1.0.win32.exe

(and no, I don't know you create a .exe usign that format for both Py2.5 
and Py2.6 :-( )

Ideal would be a binary egg distro, but I know you guys have no love for 
them :-(

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] any change of the .exe builds on PyPI?

[Chris Withers]

On 16/09/2010 10:28, Jonathan Lange wrote:
> Two questions:
>* Where are these "correct names" documented?

*snivvles* probably in the setuptools source.
I would have assumed http://www.python.org/dev/peps/pep-0376, but I 
can't see anything on it in there...

>* Would you be at all interested in updating our build to generate
> exes with these names?

Probably :-) Where are the docs on how to duplicate your current windows 
build process?

>> Ideal would be a binary egg distro, but I know you guys have no love for
>> them :-(
>
> If you can figure out how to make our Windows buildbots build them
> together with the .msis and .exes, that would be great.

Indeed!

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] how to write a safe catch-all

[Chris Withers]

Hi All,

The attached .py file demonstrates a problem I'm having with a little 
scheduler I need to maintain.


The loop() function is supposed to be a "catch all and report" error 
handler. However, when the async code initiated by doStuff throws the 
AttributeError, what actually gets logged is:


2010-09-28 14:41:15,706 ERROR   : log (14757|7f691ba676e0): 
Unhandled Error

Traceback (most recent call last):
  File "test_looping.py", line 41, in 
reactor.run()
  File "Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/base.py", 
line 1128, in run

self.mainLoop()
  File "Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/base.py", 
line 1137, in mainLoop

self.runUntilCurrent()
---  ---
  File "Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/base.py", 
line 757, in runUntilCurrent

call.func(*call.args, **call.kw)
  File "test_looping.py", line 24, in __call__
del self.connector
exceptions.AttributeError: Break instance has no attribute 'connector'

2010-09-28 14:41:15,707 ERROR   : log (14757|7f691ba676e0): 
Unhandled scheduled exception

Traceback (most recent call last):
  File "test_looping.py", line 41, in 
reactor.run()
  File "Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/base.py", 
line 1128, in run

self.mainLoop()
  File "Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/base.py", 
line 1137, in mainLoop

self.runUntilCurrent()
---  ---
  File "test_looping.py", line 35, in loop
yield doStuff()
exceptions.GeneratorExit:

But, more crucially, the looping call then appears to stop.
What can I do to get the exception logged and then everything handled 
sanely such that the looping call can continue and my lopp function will 
keep getting called once every second rather than stopping?


cheers,

Chris
from datetime import datetime
from twisted.internet.defer import inlineCallbacks, maybeDeferred, fail, 
Deferred
from twisted.internet import task, reactor
from twisted.python import log
from twisted.python.log import PythonLoggingObserver, defaultObserver

import logging

defaultObserver.stop()
logging.basicConfig(level=0,
format=(
"%(asctime)s %(levelname)-8s: %(module)-11s"
" (%(process)d|%(thread)x): %(message)s"
))
observer = PythonLoggingObserver()
observer.start()

class Break:

def __init__(self):
self.deferred = Deferred()

def __call__(self):
del self.connector

def doStuff():
b = Break()
reactor.callLater(2,b)
return b.deferred

@inlineCallbacks
def loop():
print datetime.now()
try:
yield doStuff()
except Exception,e:
log.err(None,'Unhandled scheduled exception')

looper = task.LoopingCall(loop)
looper.start(1.0)
reactor.run()
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

On 28/09/2010 15:21, exar...@twistedmatrix.com wrote:
>> But, more crucially, the looping call then appears to stop.
>
> The function you're looping over returns a Deferred that never fires.
> The LoopingCall isn't stopped, it's waiting for the Deferred.

So, inlineCallbacks/generator things will only process errbacks, not 
actual exceptions raised inside asyncronous code called from them?!

>> What can I do to get the exception logged and then everything handled
>> sanely such that the looping call can continue and my lopp function
>> will keep getting called once every second rather than stopping?
>
> When you do reactor.callLater(n, f), you put f into an execution context
> where the only thing that will happen to exceptions it raises is that
> they will be logged.

Okay, the script was a simplification of the "real problem" to try and 
give the list a "smallest failing example to try and help with".

The real logging looks like this:

2010-09-27 15:30:16,340 ERROR   : log (24331|7f2e47b4d6e0): 
Unhandled exception sending schedule transmission
Traceback (most recent call last):
   File 
"Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/python/context.py", line 
37, in callWithContext
 return func(*args,**kw)
   File 
"Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/selectreactor.py", line 
146, in _doReadOrWrite
 why = getattr(selectable, method)()
   File "Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/tcp.py", 
line 631, in doConnect
 self.failIfNotConnected(error.getConnectError((err, strerror(err
   File "Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/tcp.py", 
line 588, in failIfNotConnected
 del self.connector
---  ---
   File "ourcode.py", line 180, in checkSchedule
 yield self.sendTransmissions(...)
exceptions.GeneratorExit:

2010-09-27 15:30:28,428 ERROR   : log (24331|7f2e47b4d6e0): 
Unhandled error in Deferred:
2010-09-27 15:30:28,584 ERROR   : log (24331|7f2e47b4d6e0): 
Unhandled Error
Traceback (most recent call last):
Failure: twisted.protocols.ftp.FTPError: ('Connection Failed', 
>)

I don't quite follow what the above is trying to tell me, other than an 
FTP connection failed. However, I don't understand why that results in a 
GeneratorExit rather than an errback of the original exception being 
caught by the top level handler in the loop() function (switching back 
to the example terminology for simplicity). I also don't understand why 
an unhandled deferred is being logged rather than fed back into the 
handler I built for it!

> exception, then you have to arrange for that.  You can do this by
> wrapping f with another function that handles the exception and sends it
> where you want.

Well, as far as I can tell, that's what I'm trying to do. However, the 
thing failing in the real world is in code I don't "own" (looks like 
twisted's inards...) and I'd like to be able to cater for any failure, 
unforseen or not (barring say a SyntaxError ;-)) and still have the 
loop() call keep doing its thing.

How can I do that?

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

On 29/09/2010 17:16, Phil Mayers wrote:
> No.
>
> The problem is that your example is malformed.

Well, it's not, it's the reality of the situation and one I'd like to 
protect against; "the scheduler must not die" is the rule I need to make 
work...

> You do this:
>
>1. Create a deferred on the "Break" class instance

The "Break" class, in reality, is the bowels of 
twisted.protocols.ftp.FTPClient, copying from my previous mail:

  self.failIfNotConnected(error.getConnectError((err, strerror(err
File "Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/tcp.py",
line 588, in failIfNotConnected
  del self.connector
---  ---
File "ourcode.py", line 180, in checkSchedule
  yield self.sendTransmissions(...)
exceptions.GeneratorExit:

How can I protect my scheduler against code which doesn't catch an 
exception when it should?

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

On 29/09/2010 18:31, exar...@twistedmatrix.com wrote:
> Then you're talking about an API in Twisted which returns a Deferred
> that sometimes doesn't errback when the implementation encounters an
> error.
>
> Also, `failIfNotConnected` should never raise an exception.
>
> These sound like bugs.
>
> File a couple tickets.  With a unit tests please. :)

That's one side of things, sure, but how can I write a scheduler which 
handles the current situation?

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

On 29/09/2010 22:06, exar...@twistedmatrix.com wrote:
>> That's one side of things, sure, but how can I write a scheduler which
>> handles the current situation?
>
> Beats me.

So, if some buggy code that should be doing a deferred callback/errback 
instead raises an exception, you're basically screwed?

There's really no way to write a "safety belt" handler that will record 
the problem but then keep going?!

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

On 30/09/2010 00:31, Glyph Lefkowitz wrote:
>
> There are so many ways.

Yes, that's what I thought, which is why I was confused to keep on 
getting "fix the code" responses when I'd already pointed out it wasn't 
my code to fix, and I'd like to build a solution that caters even for 
this eventually...

> You can add a log observer that handles isError=True log messages.

Okay, and this would mean that my `loop` function from the example I 
posted would keep getting called by LoopingCall?

> You can write a 'safetyBelt' function that catches the exception and does 
> something with it.

 From my original example, I'm pretty sure that's what my `loop` 
function is; I'm confused as to why it catches a GeneratorExit when the 
attribute error is indirectly raised in the `doStuff` function, and more 
confused that even thought the GeneratorExit has been caught, the calls 
to `loop` from LoopingCall cease...

> You can always invoke your code with maybeDeferred, which will turn 
> exceptions into failures for you.

In "the real app", we have another layer or two between doStuff and 
Break, one of which does call down with maybeDeferred, doesn't make any 
difference...

> You can use inlineCallbacks, where in 'x = yield y()', y() raising an 
> exception and y() returning a failed Deferred are basically indistinguishable 
> to the caller.

Yes, as you can see, this is what `loop` does. In "the real app", 
doStuff is actually @inlineCallbacks as well, the level below that calls 
with a maybeDeferred and so yes, doing this, same result...

> Why would you say there's no way?

Because I haven't found any permutation that doesn't result in the 
LoopingCall's calls to `loop` from stopping after the exception.

I would be more than ecstatic to be proved wrong ;-)

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

On 30/09/2010 04:45, exar...@twistedmatrix.com wrote:
> On 12:52 am, ch...@simplistix.co.uk wrote:
>>
>> Because I haven't found any permutation that doesn't result in the
>> LoopingCall's calls to `loop` from stopping after the exception.
>>
>> I would be more than ecstatic to be proved wrong ;-)
>
> You keep saying the LoopingCall stops.  It does not stop.  It is waiting
> for a result.

What result is it waiting for and how do I pass that from the `loop` 
function in my first example when the exception is caught?

> If you provide it with a result, it will proceed.

See above: how do I do that?

> Glyph
> suggested a number of (hackfully) mechanisms you might use to provide it
> with a result.

Why do you say hackfully? I'm trying to build a scheduler that is 
resilient to failure in a scheduled task. Yes, of course I want to fix 
that failure, but I don't want a failure in one scheduled task to 
prevent any further scheduled tasks from being executed...

> I suggested another way (fix the broken code, or at
> least supply enough information for someone else to fix it).

For all I know, it's already fixed in a newer version of Twisted 
(unfortunately, I can't move this project to a newer version of Twisted) 
but once I have a robust scheduler, I'll certainly do my best to report 
the underlying failure to you guys properly...

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
 - http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

On 30/09/2010 14:03, Phil Mayers wrote:

On 30/09/10 13:44, Phil Mayers wrote:


This is because (as you've discovered) there are circumstances when a
function is called in Twisted where the exception can't propagate "up"
to the caller, because the caller is no longer higher in the call stack.


Indeed, but, as I keep saying, I need to build a scheduler that's 
resilient to errors in the tasks its running ;-)



Most prominent is reactor.callLater.


...which I'm not actually using, it was just a small-as-possible way I 
could simulate the failure mode (rather than the specific failure) I 
need to protect against.



@inlineCallbacks
def loop():
try:
  yield doStuff()
except Exception, e:
  log.err(e)

...will then work, and "see" the exception.


Actually, what appears to work is simply changing `loop` to not be an 
async fuction:


def loop():
try:
doStuff()
except Exception,e:
log.err(None,'Unhandled scheduled exception')

looper = task.LoopingCall(loop)
looper.start(1.0)
reactor.run()

This appears to solve most of the problems:
- the schedule now keeps running regardless
- exceptions in doStuff and below get logged

However, it now has the problem that if doStuff *does* return a deferred 
and it errbacks, the I get the ugly:


2010-09-30 14:31:12,478 ERROR   : log (22159|7fc6c5c356e0): 
Unhandled error in Deferred:
2010-09-30 14:31:12,479 ERROR   : log (22159|7fc6c5c356e0): 
Unhandled Error

Traceback (most recent call last):
Failure: twisted.python.failure.DefaultException: Break!

...rather than having my exception handler log with a meaningful message.

I've attached the full example as I have it again...

Is there any way I can get both errbacks *and* exceptions handled nicely 
in my `loop` call?


cheers,

Chris
from datetime import datetime
from twisted.internet.defer import inlineCallbacks, maybeDeferred, fail, 
Deferred, succeed
from twisted.internet import task, reactor
from twisted.python import log
from twisted.python.log import PythonLoggingObserver, defaultObserver

import logging

defaultObserver.stop()
logging.basicConfig(level=0,
format=(
"%(asctime)s %(levelname)-8s: %(module)-11s"
" (%(process)d|%(thread)x): %(message)s"
))
observer = PythonLoggingObserver()
observer.start()

class Break:

called = 0

def __init__(self):
self.deferred = Deferred()

def __call__(self):
self.__class__.called += 1
print "called ",self.called
if self.called==3:
del self.connector
if self.called==5:
self.deferred.errback('Break!')
else:
self.deferred.callback('happy!')

def doStuff():
b = Break()
reactor.callLater(2,b)
return b.deferred

def loop():
try:
doStuff()
except Exception,e:
log.err(None,'Unhandled scheduled exception')

looper = task.LoopingCall(loop)
looper.start(1.0)
reactor.run()
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

On 30/09/2010 14:46, Phil Mayers wrote:
> On 30/09/10 14:36, Chris Withers wrote:
>>
>> Indeed, but, as I keep saying, I need to build a scheduler that's
>> resilient to errors in the tasks its running ;-)
>
> Sure.
>
>>
>>>> Most prominent is reactor.callLater.
>>
>> ...which I'm not actually using, it was just a small-as-possible way I
>> could simulate the failure mode (rather than the specific failure) I
>> need to protect against.
>
> Ah.
>
>> Actually, what appears to work is simply changing `loop` to not be an
>> async fuction:
>>
>> def loop():
>> try:
>> doStuff()
>> except Exception,e:
>> log.err(None,'Unhandled scheduled exception')
>>
>> looper = task.LoopingCall(loop)
>> looper.start(1.0)
>> reactor.run()
>>
>> This appears to solve most of the problems:
>> - the schedule now keeps running regardless
>> - exceptions in doStuff and below get logged
>>
>> However, it now has the problem that if doStuff *does* return a deferred
>> and it errbacks, the I get the ugly:
>
> Well, you could do this, which is more Twist-y, and Glyph suggested:
>
> def loop():
> d = defer.maybeDeferred(doStuff, *args)
> d.addErrback(log.err)

Okay, but further down in the call stack of the "real app", I already have:

 @inlineCallbacks
 def sendTransmission(self...):
 ...
 try:
 yield maybeDeferred(feed.initiateTransmission,
 ...)
 except Exception, ex:
 if isinstance(ex,GeneratorExit):
 raise
 ...

Is that not doing what you're talking about?
If it is, it didn't help...

> FWIW, you say "if doStuff does return a deferred"; I presume you don't
> really have a single call returning both normal and deferred values,

Correct, doStuff *should* always return a deferred, the problem is that 
the deferred returned never fires (errback or callback) when an 
exception is raised in the code that should be firing it.

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

On 30/09/2010 14:39, Jonathan Lange wrote:
> On Thu, Sep 30, 2010 at 2:36 PM, Chris Withers  wrote:
> ...
>>
>> Is there any way I can get both errbacks *and* exceptions handled nicely in
>> my `loop` call?
>
> You know about defer.maybeDeferred, right?

Yep, the problem is with `loop` implemented like so:

def loop():
 d = maybeDeferred(doStuff)
 d.addErrback(partial(log.err,_why='Unhandled scheduled exception'))

...the logging is odd:

2010-09-30 15:07:03,161 ERROR   : log (22194|7f41910b26e0): 
Unhandled Error
Traceback (most recent call last):
   File "test_looping.py", line 47, in 
 reactor.run()
   File "/twisted/internet/base.py", line 1166, in run
 self.mainLoop()
   File "/twisted/internet/base.py", line 1175, in mainLoop
 self.runUntilCurrent()
---  ---
   File "/twisted/internet/base.py", line 779, in runUntilCurrent
 call.func(*call.args, **call.kw)
   File "test_looping.py", line 30, in __call__
 del self.connector
exceptions.AttributeError: Break instance has no attribute 'connector'

called  4
called  5
/twisted/internet/defer.py:262: DeprecationWarning: Don't pass strings 
(like 'Break!') to failure.Failure (replacing with a DefaultException).
   fail = failure.Failure(fail)
2010-09-30 15:07:05,167 ERROR   : log (22194|7f41910b26e0): 
Unhandled scheduled exception
Traceback (most recent call last):
Failure: twisted.python.failure.DefaultException: Break!

So, how come my log.err doesn't get used for the AttributeError on 
connector?

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] The Real Error (tm) [was Re: how to write a safe catch-all]

[Chris Withers]

On 30/09/2010 15:23, Phil Mayers wrote:
>> def loop():
>> d = maybeDeferred(doStuff)
>> d.addErrback(partial(log.err,_why='Unhandled scheduled exception'))
>
>> So, how come my log.err doesn't get used for the AttributeError on
>> connector?
>
> If you mean in your most recent "test_looping.py" example, it still uses
> reactor.callLater. The call stack has gone away by the time the call is
> made, so the exception just propagates up to the top level, where it's
> logged.

So, I appear to be back to the case where I can either gracefully handle 
the exception *or* gracefully handle the errback, but not both?

> I honestly think a more complete example showing the real Twisted API
> that's causing you unhandled errors would help here.

Unfortunately, you neither want to see, nor am I allowed to publish to a 
mailing list, that several hundred lines of proprietary code I'd need to 
post...

As far as the original error goes, we hit the problem using 
twisted.protocols.ftp.FTPClient to ftp a file up to a remote host.
That remote host only accepts active ftp transfers. As a result of 
either of both of our firewall or FTPClient not handling active ftp and 
only handling passive ftp, we end up seeing the following logging as the 
scheduler dies:

2010-09-27 15:30:16,340 ERROR   : log (24331|7f2e47b4d6e0):
Unhandled exception sending schedule transmission
Traceback (most recent call last):
File
"Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/python/context.py", line
37, in callWithContext
  return func(*args,**kw)
File
"Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/selectreactor.py", line 

146, in _doReadOrWrite
  why = getattr(selectable, method)()
File "Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/tcp.py",
line 631, in doConnect
  self.failIfNotConnected(error.getConnectError((err, strerror(err
File "Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/tcp.py",
line 588, in failIfNotConnected
  del self.connector
---  ---
File "ourcode.py", line 180, in checkSchedule
  yield self.sendTransmissions(...)
exceptions.GeneratorExit:

2010-09-27 15:30:28,428 ERROR   : log (24331|7f2e47b4d6e0):
Unhandled error in Deferred:
2010-09-27 15:30:28,584 ERROR   : log (24331|7f2e47b4d6e0):
Unhandled Error
Traceback (most recent call last):
Failure: twisted.protocols.ftp.FTPError: ('Connection Failed',
>)

I can't quite make sense of the above, which is why I distilled it down 
to as-small-as-possible a piece of code that shows the type of 
exceptions and errbacks I need to deal with...

cheers,

Chris


-- 
Simplistix - Content Management, Batch Processing & Python Consulting
 - http://www.simplistix.co.uk

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] The Real Error (tm) [was Re: how to write a safe catch-all]

[Chris Withers]

On 30/09/2010 15:53, Phil Mayers wrote:
>> So, I appear to be back to the case where I can either gracefully handle
>> the exception *or* gracefully handle the errback, but not both?
>
> It should be possible if using Twisted APIs correctly to reliably
> capture errors.
>
> If it is not, that is a Twisted bug (or possibly API limitation)
>
> However, your "test_looping.py" is not using the API correctly; the
> reactor.callLater throws away the call stack, so the error has nowhere
> to go.

Right, but that's what appears to be happening with the "real code", and 
I get the mysterious GeneratorExit...

> Since you're not using reactor.callLater in your real code, the example
> is not valid. Hopefully I'm being more clear here!

Yes, but I do feel a bit like I'm banging my head against a wall too. 
The effect is the same as the real live observed problem, only I can't 
replicate the real problem without an ftp server that only happens to 
accept active connections. I don't have one of those around, and *even* 
if I did, I want to make the scheduler bulletproof, not *just* go and 
fix the one error that has cropped up so far...

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

On 30/09/2010 15:33, exar...@twistedmatrix.com wrote:
>> 2010-09-30 15:07:03,161 ERROR   : log (22194|7f41910b26e0):
>> Unhandled Error
>> Traceback (most recent call last):
>>File "test_looping.py", line 47, in
>>  reactor.run()
>>File "/twisted/internet/base.py", line 1166, in run
>>  self.mainLoop()
>>File "/twisted/internet/base.py", line 1175, in mainLoop
>>  self.runUntilCurrent()
>> ---  ---
>>File "/twisted/internet/base.py", line 779, in runUntilCurrent
>>  call.func(*call.args, **call.kw)
>>File "test_looping.py", line 30, in __call__
>>  del self.connector
>> exceptions.AttributeError: Break instance has no attribute 'connector'
>
> This is not logged by your code.  Do you recognize that?

Yes, this is what I'm complaining about ;-)

>> /twisted/internet/defer.py:262: DeprecationWarning: Don't pass strings
>> (like 'Break!') to failure.Failure (replacing with a DefaultException).
>>fail = failure.Failure(fail)
>> 2010-09-30 15:07:05,167 ERROR   : log (22194|7f41910b26e0):
>> Unhandled scheduled exception
>> Traceback (most recent call last):
>> Failure: twisted.python.failure.DefaultException: Break!
>
> This comes from some code not included in the code you posted.

Sure it is, it was attached to the message I sent at 14:36.

> It looks
> like you're using Failure wrong though.

Sure, but that's hardly the issue at hand here...
What should I be passing to errback?

>> So, how come my log.err doesn't get used for the AttributeError on
>> connector?
 >
> Your Deferred *never* fires with a Failure corresponding to that
> AttributeError.  This is the most important thing.  If you don't
> understand this, say so and we can talk about it some more.  Everything
> else is just confusing particulars.

Yes, I understand this, and this is what I'm talking about when I say "I 
cannot gracefully handle the exception." Reading back, yes, it appears I 
was mistaken at some stage that my `loop` function was handling the 
exception, but I do understand now that it was not... which is 
frustrating...

> The AttributeError never becomes an error result on any Deferred.  It is
> caught inside the reactor implementation, logged there by Twisted
> itself, and then thrown away forever.

:-(

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] The Real Error (tm) [was Re: how to write a safe catch-all]

[Chris Withers]

On 30/09/2010 18:01, Phil Mayers wrote:
> It is more than a little confusing, and I'm sure frustrating.
>
> I've tried to produce something like this locally, but cannot.

The very first message in this thread (28th Sept, 14:48) and my email of 
30th Sept, 14:36 both had scripts attached which do exactly this..

> Let me see if I understand the problem in full.
>
> You have an @inlineCallbacks-decorated generator which is the target of
> a LoopingCall, like so:
>
> @inlineCallbacks
> def loop():
> try:
>   somework()
> except:
>   log.err()
>
> lc = task.LoopingCall(loop)

Almost:

 @inlineCallbacks
 def loop(self):
 # Check for files to send on schedule
 yield self.checkSchedule()

 @inlineCallbacks
 def checkSchedule(self):
try:
yield somework()
except Exception:
log.err(None,'Unhandled exception ...')

...although I'm currently changing the loop function to be:

 def loop(self):
self.checkSchedule()

...as this appears to give me what I want, until something proves 
otherwise...

> You want this loop function to catch&  log all exceptions resulting from
> work it initiates.

Yep, errbacks, internal twisted bugs, whatever. As long as they're 
logged by something, preferabyl the try-except above, I don't mind.
What absolutely must not cannot ever happen is for the scheduler to die ;-)

> Your "somework" function calls, amongst other things, an
> @inlineCallbacks-decorated worker function:
>
> @inlineCallbacks
> def sendTransmission(...):
> try:
>   yield maybeDeferred(feed.initiateTransmission)
> except:
>   ...some handling

Yep.

> You are seeing two errors:
>
>
>1. A GeneratorExit exception. This appears (if I'm reading your
> logging right) to be caught by your logging?

Yep.

>2. A ConnectionLost exception. This is not caught by your logging, and
> is propagating up to the reactor, and giving "Unhandled Error"

Correct.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

Hi Glyph,

On 30/09/2010 19:52, Glyph Lefkowitz wrote:
>> File "Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/base.py",
>> line 757, in runUntilCurrent
>> call.func(*call.args, **call.kw)
>> File "test_looping.py", line 24, in __call__
>> del self.connector
>> exceptions.AttributeError: Break instance has no attribute 'connector'
>
> This traceback indicates a bug in Twisted.
>
> This is an error that other people have very occasionally spotted, and
> we have never been able to reliably reproduce it.

While I'm able to reproduce this, I can't really as it involves a 
customer ftp server also generating error messages and they've already 
had enough of them ;-)

> Over the years we have
> tried to diagnose it in various ways and we have always failed.
>
> It would be really valuable to everyone if you could write up a bug and

Is there a bug already tracking this?

> provide detailed reproduction instructions, ideally with some python
> code that triggers this error, so that we can address the bug. It would
> be super useful if you could write an example that reproduces the same
> bug on a recent Twisted version (8.2 is pretty old), especially the 10.2
> prerelease. But, if your example reproduces on 8.2 and not 10.0, that
> tells us something too.

Well, unfortunately I can't do any of the above :-( I've only seen this, 
as I said previously, when we use twisted.protocols.ftp.FTPClient to try 
to send a file to an ftp server that was only able to handle active ftp 
transfers when our setup only allowed passive ftp transfers.

It was happening in a steady, reproducible fashion (ie: every time we 
tried) but unfortunately we've had to bail and now send via sftp using a 
pexpect-based wrapper around the real sftp command client (we had buggy 
behaviour with twisted's sftp client elsewhere)

I hope this helps, do let me know if I should add it to an issue 
somewhere...

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to write a safe catch-all

[Chris Withers]

On 01/10/2010 15:00, exar...@twistedmatrix.com wrote:
>> I hope this helps, do let me know if I should add it to an issue
>> somewhere...
>
> Can you at least provide a traffic capture recording an instance of this
> happening?

I'm afraid not, I didn't have the forsight to do this at the time, and 
as I've said, the customer has had enough of us causing their ftp server 
to generate error messages :-S

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 15.1 feedback

[Chris Wolfe]

Hawkowl,

The new functionality for installing optional dependencies does not on
Python3 (ticket: https://twistedmatrix.com/trac/ticket/7807).

This is due to two things:
1. some of the optional dependencies do not support python3
2. setup3.py isn't using the optional extras built in the main setup.py
file.

Thanks!
Chris (herrwolfe)

On Sat, Mar 21, 2015 at 10:29 AM, HawkOwl  wrote:

> Hi everyone,
>
> It's been about two weeks now -- does anyone have any feedback on whether
> 15.1 worked/didn't work/caused a resonance cascade?
>
> If you missed the announcement, tarballs can be found at
> http://twistedmatrix.com/Releases/pre/15.1.0pre1/ , and the full NEWS
> file can be found at
> http://twistedmatrix.com/Releases/pre/15.1.0pre1/NEWS.txt .
>
> Once I'm happy that it's not completely broken, I'll push 15.1 out the
> door proper, and get 15.2 on the way... :)
>
> - Hawkie
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>


-- 
Chris Wolfe
chriswwo...@gmail.com
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 15.1 feedback

[Chris Wolfe]

Jean-Paul,

You are correct. No, this is not a regression compared to to 15.0. I missed
the words 'resonance cascade'.

Thanks,
Chris

On Sat, Mar 21, 2015 at 11:53 AM,  wrote:

> On 03:41 pm, chriswwo...@gmail.com wrote:
>
>> Hawkowl,
>>
>> The new functionality for installing optional dependencies does not on
>> Python3 (ticket: https://twistedmatrix.com/trac/ticket/7807).
>>
>> This is due to two things:
>> 1. some of the optional dependencies do not support python3
>> 2. setup3.py isn't using the optional extras built in the main setup.py
>> file.
>>
>
> Are these regressions compared to 15.0?  I don't think they are because
> these features are brand new (on Python 2) in 15.0.  #7807 should be fixed
> (and hopefully for 15.2) but it doesn't sound like it's a blocking issue
> for completing the 15.1 release because it doesn't remove something that
> was included in 15.0.
>
> Jean-Paul
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>



-- 
Chris Wolfe
chriswwo...@gmail.com
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [Twisted-web] Twisted 15.1 Release Announcement

[Chris Wolfe]

Hey Hawkie!

I gave Glyph all of the stickers that I had, as I thought he'd come into
contact with more Twisted people than I do. As such, you might want ping
him to get a hold of one.

Thanks for all your hard work!

Best,
Chris // herrwolfe

On Tue, Apr 14, 2015 at 12:28 AM, HawkOwl  wrote:

> zomg twisted stickers!!!
>
> Chris, let me know how much it is for you to send me some -- my laptop has
> Django stickers, but no Twisted. FOR SHAME
>
> - hawkie
>
> > On 14 Apr 2015, at 13:23, Glyph  wrote:
> >
> >
> >> On Apr 13, 2015, at 04:17, HawkOwl  wrote:
> >>
> >> On behalf of Twisted Matrix Laboratories, I am honoured to announce the
> release of Twisted 15.1.0 -- just in time for the PyCon sprints!
> >>
> >> This is not a big release, but does have some nice-to-haves:
> >>
> >> - You can now install Twisted's optional dependencies easier -- for
> example, `pip install twisted[tls]` installs Twisted with TLS support.
> >> - twisted.web.static.File allows defining a custom resource for
> rendering forbidden pages.
> >> - Twisted's MSN support is now deprecated.
> >> - More documentation has been added on how Trial finds tests.
> >> - ...and 26 other closed tickets containing bug fixes, feature
> enhancements, and documentation.
> >>
> >> For more information, check the NEWS file (link provided below).
> >>
> >> You can find the downloads at <https://pypi.python.org/pypi/Twisted>
> (or alternatively <http://twistedmatrix.com/trac/wiki/Downloads>) . The
> NEWS file is also available at
> >> <
> https://github.com/twisted/twisted/blob/releases/release-15.1.0-7758/NEWS
> >.
> >>
> >> Many thanks to everyone who had a part in this release - the supporters
> of the Twisted Software Foundation, the developers who contributed code as
> well as documentation, and all the people building great things with
> Twisted!
> >>
> >> Twisted Regards,
> >> Hawkie Owl
> >
> > This is very exciting.  I am SUPER PUMPED to start telling people to
> `pip install twisted[tls]´ instead of the whole mess it used to be.  Thanks
> in particular to you, Hawkie, and to Chris Wolfe who landed that patch (and
> brought twisted stickers to PyCon!)
> >
> > -glyph
> >
> > ___
> > Twisted-Python mailing list
> > Twisted-Python@twistedmatrix.com
> > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>


-- 
Chris Wolfe
chriswwo...@gmail.com
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] revisiting onboarding

[Chris Wolfe]

Hi,

Has this proposal been accepted? Is there anything I can do to help
implement it?

-Chris

On Sun, Feb 8, 2015 at 9:01 AM, Tom Prince  wrote:

> Glyph Lefkowitz  writes:
>
> > So I have a proposal for a scaled back process that nevertheless would
> give us something official-ish:
> > <..details...>
>
> I support this proposal.
>
>   Tom
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>



-- 
Chris Wolfe
chriswwo...@gmail.com
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] revisiting onboarding

[Chris Wolfe]

Sure! I can only think of three things that need to be done:

1. Add a wiki page detailing the process you proposed. It may be helpful to
display an example email showing what should be included in a new
contributor request. The following pages should have links to the new
policy:
  - https://twistedmatrix.com/trac/wiki/TwistedDevelopment#Policies
  - https://twistedmatrix.com/trac/wiki/ContributingToTwistedLabs

I can create the wiki page and the example email. To do so, I'll need to
get wiki permissions added to my trac account.

2. Activate the email address com...@twistedmatrix.com and compile a list
of people to whom new commit requests should be sent for review. I can't do
this.

3. Send an email to the general mailing list once the new pages are up to
announce the new advancement path. I can do this.

Is there anything I'm missing?

- Chris // herrwolfe

On Tue, Apr 28, 2015 at 3:25 PM, Glyph Lefkowitz 
wrote:

> I think we can consider it tacitly accepted by the community (nobody
> seemed to object) but we still don't have anyone to implement it. Do you
> want to step up to do that? :)
>
> -g
>
> On Apr 27, 2015, at 6:04 PM, Chris Wolfe  wrote:
>
> Hi,
>
> Has this proposal been accepted? Is there anything I can do to help
> implement it?
>
> -Chris
>
> On Sun, Feb 8, 2015 at 9:01 AM, Tom Prince 
> wrote:
>
>> Glyph Lefkowitz  writes:
>>
>> > So I have a proposal for a scaled back process that nevertheless would
>> give us something official-ish:
>> > <..details...>
>>
>> I support this proposal.
>>
>>   Tom
>>
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
>
>
>
> --
> Chris Wolfe
> chriswwo...@gmail.com
>
>
>


-- 
Chris Wolfe
chriswwo...@gmail.com
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] revisiting onboarding

[Chris Wolfe]

On Tue, Apr 28, 2015 at 5:08 PM, Glyph Lefkowitz 
wrote:

> Can you send me an iCalendar invite of some kind to give you all the
> relevant permissions?  I would definitely like to volunteer to do it, but
> if we don't have a specific time, I will 100% guarantee you that I will
> forget :).
>
> -glyph
>
> P.S.: I've been setting a bad example, but we should generally be
> bottom-posting on this list, it makes the conversation easier to follow :).
>
> On Apr 28, 2015, at 2:54 PM, Chris Wolfe  wrote:
>
> Sure! I can only think of three things that need to be done:
>
> 1. Add a wiki page detailing the process you proposed. It may be helpful
> to display an example email showing what should be included in a new
> contributor request. The following pages should have links to the new
> policy:
>   - https://twistedmatrix.com/trac/wiki/TwistedDevelopment#Policies
>   - https://twistedmatrix.com/trac/wiki/ContributingToTwistedLabs
>
> I can create the wiki page and the example email. To do so, I'll need to
> get wiki permissions added to my trac account.
>
> 2. Activate the email address com...@twistedmatrix.com and compile a list
> of people to whom new commit requests should be sent for review. I can't do
> this.
>
> 3. Send an email to the general mailing list once the new pages are up to
> announce the new advancement path. I can do this.
>
> Is there anything I'm missing?
>
> - Chris // herrwolfe
>
> On Tue, Apr 28, 2015 at 3:25 PM, Glyph Lefkowitz 
> wrote:
>
>> I think we can consider it tacitly accepted by the community (nobody
>> seemed to object) but we still don't have anyone to implement it. Do you
>> want to step up to do that? :)
>>
>> -g
>>
>> On Apr 27, 2015, at 6:04 PM, Chris Wolfe  wrote:
>>
>> Hi,
>>
>> Has this proposal been accepted? Is there anything I can do to help
>> implement it?
>>
>> -Chris
>>
>> On Sun, Feb 8, 2015 at 9:01 AM, Tom Prince 
>> wrote:
>>
>>> Glyph Lefkowitz  writes:
>>>
>>> > So I have a proposal for a scaled back process that nevertheless would
>>> give us something official-ish:
>>> > <..details...>
>>>
>>> I support this proposal.
>>>
>>>   Tom
>>>
>>> ___
>>> Twisted-Python mailing list
>>> Twisted-Python@twistedmatrix.com
>>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>>
>>
>>
>>
>> --
>> Chris Wolfe
>> chriswwo...@gmail.com
>>
>>
>>
>
>
> --
> Chris Wolfe
> chriswwo...@gmail.com
>
>
>
Hi,

I've added a draft wiki page detailing the contributor advancement path.
The page is located at
https://twistedmatrix.com/trac/wiki/Drafts/ContributorAdvancementPath.

If anyone has any feedback on the document, please feel free to either edit
the wiki or send me an email through the mailing list. If there aren't any
objections to what I've written by June 5, I will move it out of the drafts
section and link it up to the other documentation.

Thanks!
Chris

-- 
Chris Wolfe
chriswwo...@gmail.com
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Blacklisting hosts

[Chris Norman]

Hi all,
I am using Twisted to make a game server. I want to be able to ban IP 
addresses. Currently I check if the host is in a blacklist, and if it 
is, call abortConnection on the transport. It works fine, but I'm 
thinking there should be a better way, to actively refuse the connection 
in the first place?


Cheers,

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Blacklisting hosts

[Chris Norman]

Hello,
> On 9 Aug 2015, at 17:07, Cory Benfield  wrote:
> 
> 
>> On 8 Aug 2015, at 08:07, Chris Norman  wrote:
>> 
>> Hi all,
>> I am using Twisted to make a game server. I want to be able to ban IP 
>> addresses. Currently I check if the host is in a blacklist, and if it is, 
>> call abortConnection on the transport. It works fine, but I'm thinking there 
>> should be a better way, to actively refuse the connection in the first place?
> 
> I am not aware of any hook in the BSD socket API that lets you refuse a 
> connection entirely. Generally, you put a socket into ‘listen’ mode 
> (indicating to the OS that you’ll accept new connections), and then you call 
> accept() to get the new connection. In fact, the OS will accept the 
> connection even before you call accept(): it’ll do it asynchronously, and you 
> will just get the FD for the connection. IIRC Windows has a winsock specific 
> thing that might do what you want, but that’s pretty platform specific and 
> probably doesn’t actually prevent the connection getting established anyway.
> 
> If you really want to never allow the connection at all, you’ll probably want 
> to program iptables (or some other firewall if you aren’t on Linux) to do the 
> packet filtering for you. A combination of iptables and ipsets will get you a 
> high-performance IP address blacklist that will drop all packets before they 
> ever reach your application.

Thanks for that. I was sort of hoping for a Pythonic solution that doesn't rely 
on SubProcess ETC, particularly as I want this server to run on any OS you 
throw at it. Thanks for the idea though, I'll certainly use that if I get 
something that little Python can't handle.
> 
> Cory
> 
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Blacklisting hosts

[Chris Norman]

Hello,

> On 10 Aug 2015, at 03:32, Glyph  wrote:
> 
>> 
>> On Aug 9, 2015, at 9:07 AM, Cory Benfield > <mailto:c...@lukasa.co.uk>> wrote:
>> 
>> 
>>> On 8 Aug 2015, at 08:07, Chris Norman >> <mailto:chris.norm...@googlemail.com>> wrote:
>>> 
>>> Hi all,
>>> I am using Twisted to make a game server. I want to be able to ban IP 
>>> addresses. Currently I check if the host is in a blacklist, and if it is, 
>>> call abortConnection on the transport. It works fine, but I'm thinking 
>>> there should be a better way, to actively refuse the connection in the 
>>> first place?
>> 
>> I am not aware of any hook in the BSD socket API that lets you refuse a 
>> connection entirely. Generally, you put a socket into ‘listen’ mode 
>> (indicating to the OS that you’ll accept new connections), and then you call 
>> accept() to get the new connection. In fact, the OS will accept the 
>> connection even before you call accept(): it’ll do it asynchronously, and 
>> you will just get the FD for the connection. IIRC Windows has a winsock 
>> specific thing that might do what you want, but that’s pretty platform 
>> specific and probably doesn’t actually prevent the connection getting 
>> established anyway.
>> 
>> If you really want to never allow the connection at all, you’ll probably 
>> want to program iptables (or some other firewall if you aren’t on Linux) to 
>> do the packet filtering for you. A combination of iptables and ipsets will 
>> get you a high-performance IP address blacklist that will drop all packets 
>> before they ever reach your application.
> 
> 
> There is a shortcut in Twisted, at least, although it does not actually 
> refuse the initial connection for the reasons listed above; you can examine 
> the "addr" passed to IProtocolFactory.buildProtocol and return None.

This is perfect, thanks. It would have been better to refuse the connection 
entirely, but as Corey said, I can use iptables if I get desperate.

> 
> -glyph
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com <mailto:Twisted-Python@twistedmatrix.com>
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python 
> <http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] revisiting onboarding

[Chris Wolfe]

On 2 February 2015 at 19:18, Glyph Lefkowitz 
wrote:
[snip]
> Not all committers are actively involved in the project at all times, so
we should form a "committer committee"
> of people who are interested in evaluating candidates.  If you would like
to do that and you're already a committer,
> please contact me and I'll add you to the list.

It looks like this new process has stalled on the formation of the
committee to review new contributor requests. As of right now, the
committee has no members. :-)

This email is a call for volunteers. If you would like to serve on the
committee, please reply to this thread and state that you are interested in
serving on the committee. Once enough volunteers have signed up, new
contributor requests will be sent to the committee's mailing list.

Thanks!
-- 
Chris Wolfe / herrwolfe
chriswwo...@gmail.com
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Possible bug

[Chris Norman]

Hi all,
Using windows 10 with Python 3.5, importing pretty much anything gives 
me the following error. Not sure if there's something wrong with my 
configuration or if it's a bug... I seem to remember something a while 
back, should I file a ticket?



Traceback (most recent call last):
  File "", line 1, in 
  File "C:\python35\lib\site-packages\twisted\internet\reactor.py", 
line 38, in 

from twisted.internet import default
  File "C:\python35\lib\site-packages\twisted\internet\default.py", 
line 56, in 

install = _getInstallFunction(platform)
  File "C:\python35\lib\site-packages\twisted\internet\default.py", 
line 50, in _getInstallFunction

from twisted.internet.selectreactor import install
  File 
"C:\python35\lib\site-packages\twisted\internet\selectreactor.py", line 
18, in 

from twisted.internet import posixbase
  File "C:\python35\lib\site-packages\twisted\internet\posixbase.py", 
line 18, in 

from twisted.internet import error, udp, tcp
  File "C:\python35\lib\site-packages\twisted\internet\udp.py", line 
53, in 

from twisted.internet import base, defer, address
  File "C:\python35\lib\site-packages\twisted\internet\base.py", line 
23, in 
from twisted.internet import fdesc, main, error, abstract, defer, 
threads
  File "C:\python35\lib\site-packages\twisted\internet\defer.py", line 
29, in 

from twisted.python import lockfile, failure
  File "C:\python35\lib\site-packages\twisted\python\lockfile.py", line 
52, in 

_open = file
NameError: name 'file' is not defined

Cheers,


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Possible bug

[Chris Norman]

Hi Amber

On 31/10/2015 06:45, Amber "Hawkie" Brown wrote:

Hi Chris,

Twisted does not yet support Python 3 on Windows.


That's a shame. I've never contributed to an open source project before, 
but I'd be glad to try and help out. I feel I __may__ know enough Python 
now to certainly have a shot at it.


There's a ticket at https://twistedmatrix.com/trac/ticket/8025#ticket which 
makes all the tests pass on the platform; I just need to work on it some more. 
I expect Twisted 16.0 to have base support for Python 3 on Windows.


As above, if I can help, please tell me how!

Cheers,


- Amber


On 31 Oct 2015, at 14:41, Chris Norman  wrote:

Hi all,
Using windows 10 with Python 3.5, importing pretty much anything gives me the 
following error. Not sure if there's something wrong with my configuration or 
if it's a bug... I seem to remember something a while back, should I file a 
ticket?


Traceback (most recent call last):
  File "", line 1, in 
  File "C:\python35\lib\site-packages\twisted\internet\reactor.py", line 38, in 

from twisted.internet import default
  File "C:\python35\lib\site-packages\twisted\internet\default.py", line 56, in 

install = _getInstallFunction(platform)
  File "C:\python35\lib\site-packages\twisted\internet\default.py", line 50, in 
_getInstallFunction
from twisted.internet.selectreactor import install
  File "C:\python35\lib\site-packages\twisted\internet\selectreactor.py", line 18, in 

from twisted.internet import posixbase
  File "C:\python35\lib\site-packages\twisted\internet\posixbase.py", line 18, in 

from twisted.internet import error, udp, tcp
  File "C:\python35\lib\site-packages\twisted\internet\udp.py", line 53, in 

from twisted.internet import base, defer, address
  File "C:\python35\lib\site-packages\twisted\internet\base.py", line 23, in 

from twisted.internet import fdesc, main, error, abstract, defer, threads
  File "C:\python35\lib\site-packages\twisted\internet\defer.py", line 29, in 

from twisted.python import lockfile, failure
  File "C:\python35\lib\site-packages\twisted\python\lockfile.py", line 52, in 

_open = file
NameError: name 'file' is not defined

Cheers,


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Possible bug

[Chris Norman]

Hi,
I've used git extensively, and I can soon read up on how to contribute 
on git. So what's the next step? Is there a howto somewhere on the 
Twisted site?


Cheers,


On 31/10/2015 07:39, Glyph Lefkowitz wrote:


On Oct 30, 2015, at 11:49 PM, Chris Norman 
mailto:chris.norm...@googlemail.com>> 
wrote:


Hi Amber

On 31/10/2015 06:45, Amber "Hawkie" Brown wrote:

Hi Chris,

Twisted does not yet support Python 3 on Windows.


That's a shame. I've never contributed to an open source project 
before, but I'd be glad to try and help out. I feel I __may__ know 
enough Python now to certainly have a shot at it.


If you feel like you might, you definitely do :-).  Often people feel 
like they need to be advanced rocket surgeons to work on Twisted, but 
in fact, our robust code-review process means that you don't have to 
worry about making mistakes at all.  Just submit your best effort, and 
you'll get a code review telling you how to improve it so that it can 
be integrated (and you _will_ get at least one round of review: I 
created Twisted, and even _I_ can't usually land a patch without one 
or two rounds of review ;-)).


Thanks for offering to help out!  If everybody who wanted Python 3 
support just signed on to work on a ticket like this, the port would 
have been finished a year ago ;-).

There's a ticket athttps://twistedmatrix.com/trac/ticket/8025#ticket  which 
makes all the tests pass on the platform; I just need to work on it some more. 
I expect Twisted 16.0 to have base support for Python 3 on Windows.

As above, if I can help, please tell me how!


Our toolchain is a little clunky (sorry about that) for hysterical 
raisins.


Before I launch into an explanation though, are you already familiar 
with Git and/or how to contribute on GitHub?


-glyph


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Running Trial [wav]: Re: Possible bug

[Chris Norman]

Hi,
So I went on the net on the off chance that the info I was looking for 
was easy to obtain, and I found out about trial.


I cloned the git and did:
python bin/trial twisted

I got the following:

Unhandled Error
Traceback (most recent call last):
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\scripts\trial.py", 
line 473, in postOptions

_BasicOptions.postOptions(self)
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\scripts\trial.py", 
line 383, in postOptions

self['reporter'] = self._loadReporterByName(self['reporter'])
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\scripts\trial.py", 
line 370, in _loadReporterByName

for p in plugin.getPlugins(itrial.IReporter):
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\plugin.py", line 
213, in getPlugins

    allDropins = getCache(package)
---  ---
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\plugin.py", line 
171, in getCache

provider = pluginModule.load()
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\python\modules.py", 
line 389, in load

    return self.pathEntry.pythonPath.moduleLoader(self.name)
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\python\reflect.py", 
line 303, in namedAny

topLevelPackage = _importAndCheckStack(trialname)
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\python\reflect.py", 
line 242, in _importAndCheckStack

return __import__(importName)
  File 
"C:\Users\chris\Dropbox\SRC\twisted\twisted\plugins\twisted_core.py", 
line 5, in 
from twisted.internet.endpoints import _SystemdParser, 
_TCP6ServerParser, _StandardIOParser
  File 
"C:\Users\chris\Dropbox\SRC\twisted\twisted\internet\endpoints.py", line 
34, in 

from twisted.internet.stdio import StandardIO, PipeAddress
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\internet\stdio.py", 
line 30, in 

from twisted.internet import _win32stdio
  File 
"C:\Users\chris\Dropbox\SRC\twisted\twisted\internet\_win32stdio.py", 
line 15, in 

from twisted.internet import _pollingfile, main
  File 
"C:\Users\chris\Dropbox\SRC\twisted\twisted\internet\_pollingfile.py", 
line 106, in 

class _PollableReadPipe(_PollableResource):
  File 
"C:\Users\chris\Dropbox\SRC\twisted\twisted\internet\_pollingfile.py", 
line 108, in _PollableReadPipe

implements(IPushProducer)
  File "C:\python35\lib\site-packages\zope\interface\declarations.py", 
line 412, in implements

raise TypeError(_ADVICE_ERROR % 'implementer')
builtins.TypeError: Class advice impossible in Python3.  Use the 
@implementer class decorator instead.


Unexpected error while writing cache file
Traceback (most recent call last):
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\scripts\trial.py", 
line 473, in postOptions

_BasicOptions.postOptions(self)
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\scripts\trial.py", 
line 383, in postOptions

self['reporter'] = self._loadReporterByName(self['reporter'])
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\scripts\trial.py", 
line 370, in _loadReporterByName

for p in plugin.getPlugins(itrial.IReporter):
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\plugin.py", line 
213, in getPlugins

allDropins = getCache(package)
---  ---
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\plugin.py", line 
185, in getCache

dropinPath.setContent(pickle.dumps(dropinDotCache))
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\python\filepath.py", 
line 1532, in setContent

os.rename(sib.path, self.path)
builtins.ValueError: rename: src and dst must be the same type

Traceback (most recent call last):
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\python\lockfile.py", 
line 91, in symlink

rename(newlinkname, filename)
FileExistsError: [WinError 183] Cannot create a file when that file 
already exists: 
'C:\\Users\\chris\\Dropbox\\SRC\\twisted\\_trial_temp.lock.1446280185217.newlink' 
-> 'C:\\Users\\chris\\Dropbox\\SRC\\twisted\\_trial_temp.lock'


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bin/trial", line 22, in 
run()
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\scripts\trial.py", 
line 616, in run

test_result = trialRunner.run(suite)
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\trial\runner.py", 
line 959, in run

return self._runWithoutDecoration(test, self._forceGarbageCollection)
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\trial\runner.py", 
line 983, in _runWithoutDecoration

oldDir = self._setUpTestdir()
  File "C:\Users\chris\Dropbox\SRC\twisted\twisted\trial\runner.py", 
line 886, in _setUpTestdir

    testdir, self._

[Twisted-Python] Waiting for transports to close

[Chris Norman]

Hi all,
I'm writing a MUD server, and I want a way for transports to be notified 
ofa shutdown before being disconnected, and the reactor being stopped.


I've tried:

for t in transports:
 t.write('Shutting down.\r\n')
 t.loseConnection()
reactor.stop()

This doesn't seem to notify the transports.

I also tried:
for t in transports:
 t.write('Shutting down.\r\n')
 t.loseConnection()
 while t.connected:
  pass
reactor.stop()

That just blocked and did nothing, presumably something do with my while 
loop.


Is there a stopWhenEmpty function on the somewhere? I did look over the 
methods, and I couldn't find anything promising.


I'm just using the standard from twisted.internet import reactor 
reactor, so no special cases here. In case it matters the transports I'm 
using are twisted.protocols.basic.LineReceiver, and everything else 
works with them.


Cheers in advance for the help.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Waiting for transports to close

[Chris Norman]

Hi,
It's a MUD server, so players type in commands and receive textual 
responses.


One of the admin commands is the ability to shutdown the server (or 
CTRL-C might be pressed on the console). I'd like this action to notify 
all connected transports that the server is going down for shutdown, so 
they're not rudely disconnected, then once the notifications have all 
gone through, then the server is free to shutdown.


I hope all this makes sense.

Cheers,

On 12/17/2015 11:49 AM, Glyph Lefkowitz wrote:


On Dec 16, 2015, at 9:25 AM, Chris Norman 
mailto:chris.norm...@googlemail.com>> 
wrote:


Hi all,
I'm writing a MUD server, and I want a way for transports to be 
notified ofa shutdown before being disconnected, and the reactor 
being stopped.


I've tried:

for t in transports:
t.write('Shutting down.\r\n')
t.loseConnection()
reactor.stop()

This doesn't seem to notify the transports.

I also tried:
for t in transports:
t.write('Shutting down.\r\n')
t.loseConnection()
while t.connected:
 pass
reactor.stop()

That just blocked and did nothing, presumably something do with my 
while loop.


Is there a stopWhenEmpty function on the somewhere? I did look over 
the methods, and I couldn't find anything promising.


I'm just using the standard from twisted.internet import reactor 
reactor, so no special cases here. In case it matters the transports 
I'm using are twisted.protocols.basic.LineReceiver, and everything 
else works with them.


Cheers in advance for the help.


This is definitely doable, but before I explain it would help to know 
/why/ you want to do this.


The reason I ask is: servers crash; hardware fails.  The falcon cannot 
hear the falconer; things fall apart; the centre cannot hold.


When those servers /do/ crash (and they will), you don't get a clean 
notification of disconnects.  So if you're writing your application to 
rely very heavily on the ability to do a clean shutdown and get 
notifications of every disconnect at the time you expect 
reactor.stop() to be running, you are probably designing your system 
in a way that will be very fragile and prone to data loss.  I know, I 
have made this mistake more than once myself :).


So, before you continue: do you actually need to do this?  Could you 
just ignore the notification of the connection drop and exit 
gracefully, perhaps properly cleaning up whatever state is left over 
at next startup?  If you really need this, understanding why you need 
it would also help in determining which implementation technique to 
suggest (there are a few).


-glyph



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Waiting for transports to close

[Chris Norman]

Hello,

On 17/12/2015 13:03, Glyph Lefkowitz wrote:


On Dec 17, 2015, at 4:56 AM, Chris Norman 
mailto:chris.norm...@googlemail.com>> 
wrote:


Hi,
It's a MUD server, so players type in commands and receive textual 
responses.


One of the admin commands is the ability to shutdown the server (or 
CTRL-C might be pressed on the console). I'd like this action to 
notify all connected transports that the server is going down for 
shutdown, so they're not rudely disconnected, then once the 
notifications have all gone through, then the server is free to shutdown.


Gotcha.  So you don't need to necessarily wait for all the messages to 
be delivered if there are slow clients waiting around; you just want 
to send everyone a farewell message and if they haven't responded 
within a reasonable timeout, go ahead and shut down anyway.


If your MUD server is already a Service 
<https://twistedmatrix.com/documents/15.5.0/api/twisted.application.service.IService.html> 
being launched by twistd, you just need to add a stopService 
<https://twistedmatrix.com/documents/15.5.0/api/twisted.application.service.IService.html#stopService> 
method that returns a Deferred.  When CTRL-C is hit (or anything else 
causes reactor.stop to be called), it will call this stopService 
method, and won't exit until a Deferred fires.


In your case, a simple deferLater 
<https://twistedmatrix.com/documents/15.5.0/api/twisted.internet.task.html#deferLater> 
will probably do the trick.  You can also speed things up when there 
are no connected clients left by cancelling that Deferred to make it 
finish firing immediately.


Will that work for you?


I hope all this makes sense.


It's not a service no... Should it be? I wasn't planning to use twistd, 
mainly because I don't know how to, and running

python main.py
is working fine, accepting command line arguments - the works.

It could be converted though, if there is an advantage with services?

Also, I've read quite a lot about Deferreds. I thought initially they 
were for multithreading your application, but I realise that's wrong, so 
I don't understand what the point in them is?


This isn't to say there isn't one mind you, I think I'm just majorly 
missing the point.




P.S.: For future reference, on this list the preferred style of reply 
is interleaved https://en.wikipedia.org 
<https://en.wikipedia.org/wiki/Posting_style#Interleaved_style>


Sorry, I'll do that in the future.

/wiki/Posting_style#Interleaved_style 
<https://en.wikipedia.org/wiki/Posting_style#Interleaved_style> or 
bottom-posting: https://en.wikipedia.org/wiki/Posting_style#Bottom-posting


-glyph



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Twisted serialport

[Chris West]

I'm currently using the SerialPort connection type in twisted 15.5.0 with 
python 2.7.7 on a windows machine.

Twisted serial communications with the serial port fail when pyserial is 
upgraded from 2.7 to 3.0.1 and fails with the following error.

Traceback (most recent call last):
  File "C:\Anaconda\Scripts\fmv3dbg-script.py", line 9, in 
load_entry_point('fmv3tools==0.2.5', 'console_scripts', 'fmv3dbg')()
  File 
"C:\Anaconda\lib\site-packages\fmv3tools-0.2.5-py2.7.egg\fmv3tools\fmv3dbg.py", 
line 338, in main
server = create_system(args)
  File 
"C:\Anaconda\lib\site-packages\fmv3tools-0.2.5-py2.7.egg\fmv3tools\fmv3dbg.py", 
line 310, in create_system
server = SerialServer(com_port, outputs=outputs)
  File 
"C:\Anaconda\lib\site-packages\fmv3tools-0.2.5-py2.7.egg\fmv3tools\fmv3dbg.py", 
line 57, in __init__
FMv3SerialGateway.__init__(self, com_port, *args, **kwargs)
  File 
"C:\Anaconda\lib\site-packages\flatmesh-0.2.8-py2.7.egg\flatmesh\fmv3_datacoms.py",
 line 403, in __init
__
baudrate=baudrate)
  File 
"C:\Anaconda\lib\site-packages\twisted-15.5.0-py2.7-win32.egg\twisted\internet\_win32serialport.py",
 line 56, in __init__
self._finishPortSetup()
  File 
"C:\Anaconda\lib\site-packages\twisted-15.5.0-py2.7-win32.egg\twisted\internet\_win32serialport.py",
 line 65, in _finishPortSetup
flags, comstat = win32file.ClearCommError(self._serial.hComPort)
AttributeError: 'Serial' object has no attribute 'hComPort'

I've removed pyserial 3.0.1 from my system for the moment but I was wondering 
if there is a way to have both pyserial 2.7 and 3.0.1 on my system when using 
twisted?

Thanks,
Chris
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted serialport

[Chris West]

Adi,

Thanks for the reply, I've applied the patch and that has sorted out my problem.

Many thanks,
Chris

Date: Wed, 9 Mar 2016 09:59:21 +
From: Adi Roiban 
To: Twisted general discussion 
Subject: Re: [Twisted-Python] Twisted serialport
Message-ID:

Content-Type: text/plain; charset="utf-8"

On 9 March 2016 at 09:53, Chris West  wrote:

> I?m currently using the SerialPort connection type in twisted 15.5.0 
> with python 2.7.7 on a windows machine.
>
>
>
> Twisted serial communications with the serial port fail when pyserial 
> is upgraded from 2.7 to 3.0.1 and fails with the following error.
>
>
>

Hi,

I think that this is a know issue.

Please see https://twistedmatrix.com/trac/ticket/8159

It would help if you could try the patch and report that all is ok for your use 
case.

If not the same issue, please report a new ticket.

Regards
--
Adi Roiban

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Sending disconnect messages to clients

[Chris Norman]

Hi all,
I'm sure I asked this question before, but I can't find any answers on 
google, and I've changed my work flow a bit now, so thought it was worth 
re-asking.


So on my server's protocol, I have a send method. This allows me to pass 
arguments which get sent off to the client in the right format. The send 
method looks something like this:


 def send(self, command, **kwargs):
  """Cause the client to issue getattr(connection, command)(**kwargs). 
If disconnect evaluates to True, disconnect the client after the message 
is sent."""

  disconnect = kwargs.get('disconnect', False)
  try:
   del kwargs['disconnect']
  except KeyError:
   pass # Argument not found.
  d = defer.Deferred()
  d.addCallback(self.prepare_command) Convert the command and kwargs to 
json.

  d.addCallback(self.deferred_write) # Write the json to the transport.
  if disconnect:
   d.addCallback(self.deferred_disconnect) # Issue 
self.transport.loseConnection().
  reactor.callFromThread(d.callback, [command, kwargs]) # Call the 
deferred's callback chain.

  return d # Return the deferred.

If I try something like:
protocol.send('alert', message = '*** Disconnected. ***', disconnect = True)
the client gets disconnected, but never sees the "*** Disconnected. ***" 
message.


I guess I could do a reactor.callLater and just wait for the transport 
to get the message, but that seems sloppy, and I can't help thinking 
there must be something I'm missing.


Any ideas welcome!

Cheers all,


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Sending disconnect messages to clients

[Chris Norman]

Hi Daniel,

On 22/03/2016 19:49, L. Daniel Burr wrote:

Hi Chris,

On March 22, 2016 at 2:42:14 PM, Chris Norman 
(chris.norm...@googlemail.com <mailto:chris.norm...@googlemail.com>) 
wrote:



Hi all,
I'm sure I asked this question before, but I can't find any answers on
google, and I've changed my work flow a bit now, so thought it was worth
re-asking.

So on my server's protocol, I have a send method. This allows me to pass
arguments which get sent off to the client in the right format. The send
method looks something like this:

def send(self, command, **kwargs):
"""Cause the client to issue getattr(connection, command)(**kwargs).
If disconnect evaluates to True, disconnect the client after the message
is sent."""
disconnect = kwargs.get('disconnect', False)
try:
del kwargs['disconnect']
except KeyError:
pass # Argument not found.
d = defer.Deferred()
d.addCallback(self.prepare_command) Convert the command and kwargs to
json.
d.addCallback(self.deferred_write) # Write the json to the transport.
if disconnect:
d.addCallback(self.deferred_disconnect) # Issue
self.transport.loseConnection().


You are disconnecting right here, without waiting for your Deferred to 
fire.  It might make better sense to add the call the 
self.transport.loseConnection() to your Deferred’s callback chain.


Hope this helps,



Actually it's been added to the callback. I think my mail client wrapped 
the lines, but issue self.transport.LoseConnection() is the comment, so 
self.deferred_disconnect is added to the callback chain.


Sorry for the confusion.

Anything else which can cause this?

Cheers,


Daniel
--
L. Daniel Burr
ldanielb...@me.com <mailto:ldanielb...@me.com>
(312) 656-8387


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Sending disconnect messages to clients

[Chris Norman]

OK... ignore me.

I made your test program and it worked fine. So I tried connecting to my 
server with telnet and saw the expected message... So I checked the 
error function which should have been called on the client with the 
particular message to see it was as good as pass...


Sorry for all the messages, everything works fine, panic over!

Thank you for your help with this!


On 22/03/2016 20:37, L. Daniel Burr wrote:

Hi Chris,

On March 22, 2016 at 3:09:08 PM, Chris Norman 
(chris.norm...@googlemail.com <mailto:chris.norm...@googlemail.com>) 
wrote:



Hi Daniel,



[SNIP]

Actually it's been added to the callback. I think my mail client 
wrapped the lines, but issue self.transport.LoseConnection() is the 
comment, so self.deferred_disconnect is added to the callback chain.


Sorry for the confusion.

Anything else which can cause this?



Without seeing the rest of your code, I can’t suggest anything 
concrete.  You say the client never “sees” the message, but your call 
to loseConnection() still occurs, so you’re not encountering an 
exception during the callback chain.  Can you come up with an minimal 
example the demonstrates the problem?


Daniel
—
L. Daniel Burr
ldanielb...@me.com <mailto:ldanielb...@me.com>
(312) 656-8387



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] 16.0.0 32-bit wheels for Windows

[Chris Norman]

Hi,

On 30/03/2016 19:20, anatoly techtonik wrote:

Hi,

I am having the same problem as this guy:
https://stackoverflow.com/questions/36279141/pip-doesnt-install-twisted-on-windows
Absence of 32-bit wheels breaks Python
projects that added Twisted>10 as a
dependency, for example.

Is that intentional change?
https://pypi.python.org/pypi/Twisted/16.0.0

No clue about any change, but which Python version are you using?

I have happily `pip install`ed twisted on Windows 10 (64 bit) with 32 
bit Python 3.5.1, and 2.7.11.


Only place where I had trouble was Debian 64 bit where a simple python 
setup.py installed did the trick.


Sorry you're having trouble.



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Problem with Deferreds

[Chris Norman]

Hi all,
I recently got over myself and forced myself to read about Deferreds 
rather than using threading opperations.


I've used them successfully in a couple of places, but this one has me 
flummoxed:


Here's the code for the Deferred and it's sub-commands:

 def do_command(self, cmd, **kwargs):
  """Process a command in true Deferred style."""
  if cmd.permissions(self.connection):
   cmd(self.connection, **kwargs)
  else:
   logger.warning('Blocked from running command %s which is secured 
with %s.', cmd.__name__, cmd.permissions.__name__)
   raise CommandError('You have insufficient privileges to perform this 
action. The staff have been notified.')


 def handle_error(self, err):
  """Handle an error from do_command."""
  if isinstance(err, CommandError):
   return self.send_error(e.message, disconnect = e.disconnect)
  else:
   self.log(e, level = 'exception')
   self.send_error('Sorry, but a problem with the server means your 
command was not executed. The staff have been notified.')


 def lineReceived(self, line):
  """Parse an incoming command."""
  global lines
  lines += 1 # Increment the line count.
  data = line.decode(settings.ENCODING)
  try:
   command, kwargs = json.loads(data)
   if not isinstance(command, string_types) or not isinstance(kwargs, 
dict):
raise TypeError('Expecting [str, dict]. Got [%s, %s] instead.' % 
(type(command), type(kwargs)))

  except (TypeError, ValueError) as e:
   self.log('Invalid command string: %s', data, level = 'error')
   self.log(e, level = 'exception')
   return self.send_error('Invalid command.', disconnect = True)
  cmd = commands.commands.get(command, None)
  if cmd  is None:
   self.log('Unrecognised command: %s.', command, level = 'warning')
  elif self.connection.player or not cmd.login_required:
   d = defer.Deferred()
   print('Adding callback.')
   d.addCallback(self.do_command, **kwargs)
   print('Adding errback.')
   d.addErrback(self.handle_error)
   print('Calling callback.')
   d.callback(cmd)
   print('Called.') # Never gets this far.
   return d
  else:
   return self.send_error('Not authenticated.', disconnect = True)

Here's the traceback I get when the callback gets called:

Unhandled Error
Traceback (most recent call last):
  File "server/functions.py", line 88, in server_start
reactor.run()
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/base.py", 
line 1194, in run

self.mainLoop()
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/base.py", 
line 1206, in mainLoop

self.doIteration(t)
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/epollreactor.py", 
line 396, in doPoll

log.callWithLogger(selectable, _drdw, selectable, fd, event)
---  ---
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/log.py", 
line 101, in callWithLogger

return callWithContext({"system": lp}, func, *args, **kw)
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/log.py", 
line 84, in callWithContext

return context.call({ILogContext: newCtx}, func, *args, **kw)
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/context.py", 
line 118, in callWithContext

return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/context.py", 
line 81, in callWithContext

return func(*args,**kw)
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/posixbase.py", 
line 610, in _doReadOrWrite

self._disconnectSelectable(selectable, why, inRead)
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/posixbase.py", 
line 258, in _disconnectSelectable

selectable.connectionLost(failure.Failure(why))
  File 
"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/failure.py", 
line 232, in __init__

tb = self.value.__traceback__
builtins.AttributeError: 'Deferred' object has no attribute '__traceback__'

Anyone have any ideas?

Cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Problem with Deferreds

[Chris Norman]

Hi,
Yes, that sorted the problem out no worries.

Thank you so much.



On 02/04/2016 15:00, Kevin Conway wrote:

Hi Chris,

tl;dr: Returning a value from 'dataReceived', or any of its extensions 
such as 'lineReceived' in the 'LineReceiver' Protocol subclass, 
triggers a disconnect and uses the returned value as the 'reason'. A 
'reason' must be an Exception or t.p.Failure object as other values 
will trigger this error.


Are you quite certain that your last line is not getting printed? I'm 
not sure exactly where this feature is documented, but returning any 
non-None value from a Protocol's 'dataReceived' method can result in 
this behaviour. The t.protocols.basic.LineReceiver calls 
'lineReceived' from 'dataReceived' and returns any value it gets from 
your implementation. The value returned from 'dataReceived' is passed 
along to the transport's 'doRead' which, again, returns it to the 
portion of the reactor handling selectables. The reactor assumes that 
anything returned from a transport during a read or write operation is 
a bad thing and disconnects the transport. During the disconnect 
process the reactor is generating a t.p.failure.Failure object and 
passing in your returned value as the 'why' which is expected to be an 
Exception or Failure and not a Deferred. Try returning None instead of 
your Deferred. That should resolve this particular issue.


On Sat, Apr 2, 2016 at 4:39 AM Chris Norman 
mailto:chris.norm...@googlemail.com>> 
wrote:


Hi all,
I recently got over myself and forced myself to read about Deferreds
rather than using threading opperations.

I've used them successfully in a couple of places, but this one has me
flummoxed:

Here's the code for the Deferred and it's sub-commands:

  def do_command(self, cmd, **kwargs):
   """Process a command in true Deferred style."""
   if cmd.permissions(self.connection):
cmd(self.connection, **kwargs)
   else:
logger.warning('Blocked from running command %s which is secured
with %s.', cmd.__name__, cmd.permissions.__name__)
raise CommandError('You have insufficient privileges to
perform this
action. The staff have been notified.')

  def handle_error(self, err):
   """Handle an error from do_command."""
   if isinstance(err, CommandError):
return self.send_error(e.message, disconnect = e.disconnect)
   else:
self.log(e, level = 'exception')
self.send_error('Sorry, but a problem with the server means your
command was not executed. The staff have been notified.')

  def lineReceived(self, line):
   """Parse an incoming command."""
   global lines
   lines += 1 # Increment the line count.
   data = line.decode(settings.ENCODING)
   try:
command, kwargs = json.loads(data)
if not isinstance(command, string_types) or not isinstance(kwargs,
dict):
 raise TypeError('Expecting [str, dict]. Got [%s, %s] instead.' %
(type(command), type(kwargs)))
   except (TypeError, ValueError) as e:
self.log('Invalid command string: %s', data, level = 'error')
self.log(e, level = 'exception')
return self.send_error('Invalid command.', disconnect = True)
   cmd = commands.commands.get(command, None)
   if cmd  is None:
self.log('Unrecognised command: %s.', command, level = 'warning')
   elif self.connection.player or not cmd.login_required:
d = defer.Deferred()
print('Adding callback.')
d.addCallback(self.do_command, **kwargs)
print('Adding errback.')
d.addErrback(self.handle_error)
print('Calling callback.')
d.callback(cmd)
print('Called.') # Never gets this far.
return d
   else:
return self.send_error('Not authenticated.', disconnect = True)

Here's the traceback I get when the callback gets called:

Unhandled Error
Traceback (most recent call last):
   File "server/functions.py", line 88, in server_start
 reactor.run()
   File

"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/base.py",
line 1194, in run
 self.mainLoop()
   File

"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/base.py",
line 1206, in mainLoop
 self.doIteration(t)
   File

"/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/epollreactor.py",
line 396, in doPoll
 log.callWithLogge

Re: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.dist renamed to twisted.python._setup

[Chris Wolfe]

+1 from me as well. Depending directly on how twisted installs itself seems
problematic. In my opinion, the more we discourage this practice the better.

On Sun, Aug 7, 2016 at 8:00 PM Tristan Seligmann 
wrote:

> +1 from me
>
> On Mon, 8 Aug 2016 at 02:35 Craig Rodrigues 
> wrote:
>
>>
>> I understand the motivation, but the time to have done this was when
>> dist.py
>> was originally reviewed.
>>
>> There are real problems in the Twisted code, and in the Twisted
>> infrastructure.
>> Focusing on something like this instead of those problems seems pointless,
>> gratuitous and of little benefit.
>>
>
> The problem is that if we defer this change until later, it becomes a lot
> more work then. If we do it now, while nobody is depending on the module,
> then we save ourselves that work in future by doing a trivial amount of
> work now.
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-- 
- Chris
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] in case there's downtime

[Chris Norman]

Hi,

What needs taking over on the mailing list side? I'd love to help out if 
possible.



Cheers,


On 25/03/2017 04:14, Glyph Lefkowitz wrote:
We've been very conservative about upgrading our operating system on 
twistedmatrix.com , since we don't want to 
disrupt anything.  Since our OS is one release too old to support 
Docker, that unfortunately means we haven't been able to migrate 
incrementally to containers, which creates a bit of a chicken-and-egg 
problem.


However, as we are days away from the end-of-life cliff for the 
operating system as a whole, and as I discovered a few minutes ago, 
dozens of packages are already unsupported and not receiving security 
updates, I'm going to upgrade the host OS manually, even if it results 
in some disruption.  As with the Twisted compatibility policy, 
security is a good enough reason to break stuff.


Maybe we'll even get a new version of Mailman that will let me turn 
automatic subscriptions back on while the star-crossed 
https://lists.twistedmatrix.com project waits for someone else to help 
out :-).


-glyph


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Beginner

[Chris Norman]

On 12/04/2017 14:21, catch me wrote:
Hey! I don't know I should ask this question or not but as a beginner 
I have to ask.
I have just started learn python network programming and fortunately 
found such an amazing Twisted Framework.


Welcome! Glad you found Twisted, it really is awesome!


As a beginner I could not be able to decide where to start I mean what 
should I make first using twisted framework. I would be very thankful 
to you for guiding me.


I personally like telnet-like servers. They are very easy to make with 
the LineReceiver protocol (hint hint), and you can do a lot with them.


I guess the most basic one is some kind of echo server / client that 
simply echos back whatever you type.


For more help, don't forget the Twisted documentation and the examples 
therein on http://www.twistedmatrix.com



Pardon me for bad english!


Not to worry at all, everyone is welcome.



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Beginner

[Chris Norman]

On 12/04/2017 14:41, Jean-Paul Calderone wrote:
On Wed, Apr 12, 2017 at 9:37 AM, Chris Norman 
mailto:chris.norm...@googlemail.com>> 
wrote:




On 12/04/2017 14:21, catch me wrote:

Hey! I don't know I should ask this question or not but as a
beginner I have to ask.
I have just started learn python network programming and
fortunately found such an amazing Twisted Framework.


Welcome! Glad you found Twisted, it really is awesome!



As a beginner I could not be able to decide where to start I mean
what should I make first using twisted framework. I would be very
thankful to you for guiding me.


I personally like telnet-like servers. They are very easy to make
with the LineReceiver protocol (hint hint), and you can do a lot
with them.


If you want /telnet/ then you should probably use 
/twisted.conch.telnet/.  It's a little bit more complicated than 
/LineReceiver/ but it will actually speak telnet for you.  
LineReceiver will get you some simple line-oriented interactions, though.




Ah, than you. I could never figure out how to make the proper 
TelnetProtocol to do anything useful, and line-based is all I've ever 
really needed.


Thank you for the tip though.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Forcing Twisted / Klein to use HTTPS

[Chris Norman]

Hi all,

I'm trying to force Klein to use HTTPS, and so far I have a custom error 
handler which redirects the user to the HTTPs version of the page via a 
check_secure function which takes the request object and raises the 
right error if request.isSecure() is False.



Is there a better global way to enforce HTTPS with any part of Klein or 
Twisted?



Cheers,


Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] automated tests for a server application

[Chris Withers]

Hi All,

(copying in Moshe as he expressed an interest!)

I've inherited a twisted app for which I'd like to add some changes but 
want to improve the automated test coverage before I do.


The app itself listens on a web socket (autobahn), an rpc port (RPyC) 
and also connects to a mysql database.
So, for the automated tests, I want to check what happens when I send 
various messages to the websocket or rpc port, and then check what ends 
up in the db and what other state changes happen in the app itself.


What's the best/correct way to write automated tests for this kind of 
situation? I'm using pytest and would happily use pytest-twisted too. 
Not keen to use trial...


Any help is good help, pointers to nice existing examples would be 
fantastic!


cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] "blocking" on a response to a client

[Chris Withers]

Hi Again,

So, separate question: I have a client that's making a connection to a 
websocket from a server. The client needs to send an "auth" message once 
the websocket is up, and the client code shouldn't progress until it 
receives an affirmative response. How do I do this in a non-blocking way 
when using twisted? (inside an inlineDeferred method in this case!)


Similarly, how do I get this dance to happen again if the websocket 
drops? How would I do some kind of backoff on he connection retries?


And, icing on the cake, where can I find good examples of how to write 
automated tests for all of the above?


cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] automated tests for a server application

[Chris Withers]

On 06/11/2018 05:43, Moshe Zadka wrote:


Some of the best advice depends on details of the application. One 
trick that is sometimes useful is passing in a "fake" reactor object. 
This, of course, is only useful if the application is structured in a 
way that functions/classes expect to get a reactor, instead of 
reaching for the global one. However, usually *that's* not a 
complicated refactoring to do.


Cool, do you have any example tests that do this?
Interesting, looks like pytest-twisted does away for the need for this 
by showing how to install a fake reactor globally:

https://github.com/pytest-dev/pytest-twisted/blob/master/pytest_twisted.py#L129-L142

You can look at 
https://twistedmatrix.com/documents/current/api/twisted.test.proto_helpers.MemoryReactor.html for 
inspiration although, as all code under `twisted.test`, it is not 
intended as end-user functionality, so using it directly is problematic. 


Not sure I fully understand this, why is the MemoryReactor bad to use? 
Where is it used?


It would be nice to have externally-useful reactors along the lines of 
Clock 
(https://twistedmatrix.com/documents/current/api/twisted.internet.task.Clock.html ) 
but, unfortunately, we do not have that yet.


So, clock is just a clock, right? How would that get worked into a 
"real" reactor for testing?


You can, of course, use a real reactor and a real client to pump data. 
However, in that case, you probably do want to switch to trial so that 
you can return a deferred from a test function and have the reactor 
run until the deferred fires. This is not great, but can work in a pinch.


pytest-twisted looks like it supports this pattern too, allowing test 
functions to return deferreds...


I guess I'm still really looking to see examples of all of this...

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] "blocking" on a response to a client

[Chris Withers]

On 05/11/2018 18:32, meejah wrote:

Chris Withers  writes:


So, separate question: I have a client that's making a connection to a
websocket from a server. The client needs to send an "auth" message
once the websocket is up, and the client code shouldn't progress until
it receives an affirmative response. How do I do this in a
non-blocking way when using twisted? (inside an inlineDeferred method
in this case!)


There's nothing really special about an @inlineCallbacks method -- it
just returns a Deferred so you can handle it like any other
Deferred-returning method.


Thanks, but the special bit I was asking about is having my 
application's logical flow pause until the websocket is successfully up 
and a successful response has been received, all without blocking the 
reactor.


How do I do that?

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] "blocking" on a response to a client

[Chris Withers]

On 06/11/2018 13:47, Jean-Paul Calderone wrote:

Thanks, but the special bit I was asking about is having my
application's logical flow pause until the websocket is successfully up
and a successful response has been received, all without blocking the
reactor.

How do I do that?


Put the rest of your code in a callback on a Deferred that fires when 
the event you're interested in happens.  When using inlineCallbacks, 
that's like:


     yield the_event_youre_interested_in
     # ... more code

At least, this is the extremely-localized solution to the problem.  
There are many possible ways to structure such a thing.  It's easy to 
describe this one because you said you're using inlineCallbacks and 
didn't say anything else about how your code is organized.


Right, but I'm still too stupid to join the dots :-(

More concretely, I want to spin up an autobahn websocket client, have it 
connect, send a login message, confirm that worked, send a 'create' 
message, confirm that worked, and *then* let the rest of my code progress...


I'm surprised there don't appear to be any example of this floating 
around, and disappointed at the lack of examples of good tests for these 
things.


Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] probably stupid question about automated testing and reactor.run

[Chris Withers]

Hi All,

Sorry, the silly questions about testing keep coming...

Is it legit for a unit testable piece of an application's code to call 
reactor.run()?


How do you feed a reactor.stop() into such a test? Would that be a sane 
thing to try and do?


How do people deal with this kind of testing?

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] pytest-twisted questions

[Chris Withers]

On 13/11/2018 20:31, Kyle Altendorf wrote:
I would just open an issue on GitHub.  Feel free to copy this in as the 
first message to keep the initial context.


Well, there's no specific issue to log here, so let's keep on going with 
email for now :-)


(in fact, copying in the twisted mailing list, as that's probably the 
right place for this)



On 2018-11-13 12:35, Chris Withers wrote:

- What's the intention of the plugin? Should all tests still subclass
twisted.trial.unittest.TestCase or should they *never* do so if using
this plugin?


I don't know what _should_ be done, 


Victor, what was your intention with the project?

but I know that I mostly don't have 
test classes and, for the class I do have, I didn't inherit.  Mostly I 
just @pytest.inlineCallbacks (I still don't like the namespace squashing 
into pytest though :] ) and I suppose in the probably-not-too-distant 
future I'll instead be using more @pytest_twisted.async_await  (ala 
#31/#34).


Okay, but twisted.trial.unittest.TestCase does a bunch of reactor 
management stuff, most notable making you aware when you've left the 
reactor in a bad state. As far as I can see from the code, 
pytest-twisted does not do that, correct?



- What's with the greenlet mentions? Is this plugin okay to use when
I'm just using a normal Twisted epoll reactor?


https://github.com/pytest-dev/pytest-twisted/blob/master/pytest_twisted.py#L4
https://github.com/pytest-dev/pytest-twisted/blob/master/pytest_twisted.py#L36-L46
https://github.com/pytest-dev/pytest-twisted/blob/master/pytest_twisted.py#L68-L83
https://github.com/pytest-dev/pytest-twisted/blob/master/pytest_twisted.py#L123-L126

What does greenlet have to do with twisted in this context? Would be 
great to have it as an optional thing, unless I'm missing something?



- What does pytest provide *without* this plugin, when it comes to
testing with Twisted?


I am not aware of anything twisted/pytest specific that is outside 
pytest-twisted, 


https://docs.pytest.org/en/latest/faq.html#how-does-pytest-relate-to-twisted-s-trial

So, my guess is that twisted.trial.unittest.TestCase subclasses 
unittest.TestCase and so pytest treats it in the same way. That means 
you get the management and checking of the reactor, along with the handy 
methods it provides, when you put your tests in class-based suites that 
subclass twisted.trial.unittest.TestCase.


Ronny, does pytest do anything else that's twisted-specific?

If you are just trying to get started with something that works, I'd 
skip the classes and inheritance and just let the reactor take care of 
itself. 


My experience with Twisted over the last 10 years or so is that this is 
an exceedingly dangerous approach to take...


cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] "disconnecting properly" in tests still hangs on macOS

[Chris Withers]

Right, so, I've been trying to get the technique in 
https://jml.io/pages/how-to-disconnect-in-twisted-really.html to work 
for me.


No hating please, most of my testing in the past has involved hitting a 
relational database, so there's already a TCP connection flying around, 
one more won't make any difference.


jml's example, exactly as-is on that page, hangs around 30-40% of the 
time when running on my macOS laptop. From changing the teardown to look 
like this:


    def tearDown(self):
    ds = defer.maybeDeferred(self.serverPort.stopListening)
    dc = defer.maybeDeferred(self.clientConnection.disconnect)
    print()

    ds.addCallback(lambda _: print('serverPort.stopListening'))
    dc.addCallback(lambda _: print('self.clientConnection.disconnect'))
    self.clientDisconnected.addCallback(lambda _: 
print('self.clientDisconnected'))
    self.serverDisconnected.addCallback(lambda _: 
print('self.serverDisconnected'))
    self.serverDisconnected.addErrback(lambda _: 
print('self.serverDisconnected:', _))
    return defer.gatherResults([ds, dc, self.clientDisconnected, 
self.serverDisconnected])


...it appears that it's the serverDisconnected deferred that's failing 
to fire. I can't reproduce this on Linux as of yet, so I'm guessing this 
is a difference between the SelectReactor used on macOS and the 
EPollReactor used on Linux.


What's the best way to go about debugging a non-firing deferred like this?

Anyone know what this might be?

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] "disconnecting properly" in tests still hangs on macOS

[Chris Withers]

On 14/11/2018 13:41, exvito here wrote:


Chris,

I played with this for a bit and quickly reproduced the "server side disconnect 
never seems to happen" behaviour you described, on my system running macOS 10.12.6 
and Twisted 18.9.0 using the SelectReactor.

Suspecting of an eventual race condition between "server stop listen" and "server 
disconnect", I tried this variation of tearDown which seems to work reliably:

 @defer.inlineCallbacks
 def tearDown(self):
 self.clientConnection.disconnect()
 yield defer.gatherResults([self.clientDisconnected, 
self.serverDisconnected])
 yield defer.maybeDeferred(self.serverPort.stopListening)

Do I have any motive to suspect such race condition? No, but after ensuring 
"disconnect first, stop listening later", things work much better here.

Also tested the CFReactor and KQueueReactor: CFReactor seems to exhibit a 
similar behaviour (original code hangs every now and then on cleanup, my code 
works); KQueueReactor always hangs on cleanup with the original code, and works 
reliably with my code.


Thanks, this has worked flawlessly!

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] carly: a tool for testing twisted protocols with real networking

[Chris Withers]

Hi All,

Okay, so, I've started working jml's "really disconnecting" blog post 
into a more generic set of tooling for testing protocols talking to each 
other over real networking like this:


https://github.com/cjw296/carly/blob/master/tests/test_line_receiver_server.py

Very interested in peoples thoughts, and especially if you can spot any 
bugs in these two:


https://github.com/cjw296/carly/blob/master/carly/hook.py
https://github.com/cjw296/carly/blob/master/carly/context.py

Thanks for all the help so far, it's certainly helping me get my head 
around this all :-)


Chris

PS: Real networking is bad, yes, I know: slow, unreliable, etc ;-) 
However, in most of the testing I do, I end up having to hit a real 
database server to check my code works with it, so a few more tcp 
connections to localhost is not the end of the world!


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Thread Consumption Problem in Daemon?

[Chris Withers]

On 22/11/2018 02:30, Glyph wrote:


On Nov 19, 2018, at 6:16 AM, Darren Govoni <mailto:darrenus...@gmail.com>> wrote:


I tried to find out if there is a way to limit the thread pool size 
from command line for twisted web and found nothing. Does it exist?


The thread pool is limited to 10. While this is configurable via the 
API, no command line option is exposed to tune it.  (This would be a 
great contribution if you were so inclined!)


It seems likely to me that Flask is spawning background threads for some 
reason; given the way Twisted's threadpool works, leaks like this are 
not common.  However, anything is possible: you probably want to gather 
some information about what all those threads are doing.


Some ideas on this front:

- pstree/ps and strace will tell you at a low level

- http://pyrasite.com/ and then use Python's thread introspection stuff.

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] leakage between tests with trial when code under test uses deferToThread

[Chris Withers]

Hi All,

Does trial do anything to clean up stuff that's been passed to 
deferToThread?


I'm seeing what looks like leakage between tests where stuff that is 
deferred to a thread from a LoopingCall is resulting in a DelayedCall 
ending up in the reactor for the next test.

Does that ring any bells?

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] leakage between tests with trial when code under test uses deferToThread

[Chris Withers]

Forgot to include the list in my reply...

On 23/11/2018 22:27, Chris Withers wrote:

On 23/11/2018 22:22, Glyph wrote:




On Nov 23, 2018, at 7:58 AM, Chris Withers  wrote:

Hi All,

Does trial do anything to clean up stuff that's been passed to 
deferToThread?


Nope.  It does its best to clean up stuff that it knows is "in" the 
reactor (sockets, timers, and the like), but since threads can kind 
of do ~whatever~ there hasn't been support for that.


Actually, looks like there's stuff in trial's Janitor class, but it 
only kicks in after the suite is finished, and I'm seeing leakage 
between tests within a suite...


deferToThread is a bit of a special case and you make a good point 
here: there should probably be special support for it in trial.


...which I plan to add in carly in the meantime: basically block with 
a timeout on everything in the threadpool finishing it's work.


Speaking of which, I'm happy with how carly is turning out, but would 
still welcome feedback, particularly on how the tests suites feel:


https://github.com/cjw296/carly/tree/master/tests

Just don't read hook.py unless you like head-bendy code ;-)


Okay, so here's what I came up with 
<https://github.com/cjw296/carly/commit/71a1d2bfd501f5c561712c75253fc23c28c3bba7#diff-3670fb1f3b913f6c1ec584fa64302c2dR17>. 
I'm not a fan of the sleep / busy loop pattern, but twisted makes it 
prettymuch impossible to get hold of the ThreadWorkers queue, so that I 
could block on them in this method. Any ideas on improvements would be 
very welcome!


cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] code coverage when code executed in a twisted worker thread

[Chris Withers]

Hi All,

I've noticed that code coverage doesn't appear to be recorded when code 
is executed in something that has been deferToThread'ed.


Is this a known issue? Does anyone have an explanation?

cheers,

Chris

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python