Re: [Twisted-Python] Twisted developers: please donate your time!
On Tue, 2009-02-24 at 21:25 +1300, Michael Hudson wrote: > 2009/2/24 Itamar Shtull-Trauring : > > As part of the TSF's fundraising efforts, we are trying to get upfront > > donations of time you will spend developing Twisted. It will then get > > matched by a donor, if we are successful in getting this grant. So if > > you're planning on working on Twisted anyway this year, your work will > > count twice as much! > > > > So, if you're interested, please reply, saying something like "I will > > spend two weeks working on Gopher support over the next year." > > I don't entirely understand, but I will commit to spending the > equivalent of two working weeks as a general review monkey over the > next year. Is that the sort of thing you wanted to hear? Yes - thanks to you and all thee rest of the people who are volunteering! Anyone else willing to step up? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted developers: please donate your time!
On Mon, 02 Mar 2009 09:08:00 -0500, Itamar Shtull-Trauring wrote: On Tue, 2009-02-24 at 21:25 +1300, Michael Hudson wrote: 2009/2/24 Itamar Shtull-Trauring : > As part of the TSF's fundraising efforts, we are trying to get upfront > donations of time you will spend developing Twisted. It will then get > matched by a donor, if we are successful in getting this grant. So if > you're planning on working on Twisted anyway this year, your work will > count twice as much! > > So, if you're interested, please reply, saying something like "I will > spend two weeks working on Gopher support over the next year." I don't entirely understand, but I will commit to spending the equivalent of two working weeks as a general review monkey over the next year. Is that the sort of thing you wanted to hear? Yes - thanks to you and all thee rest of the people who are volunteering! Anyone else willing to step up? I will spend the equivalent of four weeks over the next year working on ticket triage, code reviews, Lore, and Web. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] newbie problem with SMTPClient
I love the idea of twisted but I think I must have a twisted learning
disability, as I have gotten nowhere in what ought to be a simple matter.
I need to send out emails to small groups from my apache server running a
python cgi using mod_python, but my hosting service doesn't have a MTA. Instead
of learning how to install and configure exim I thought I would use twisted to
make a simple mail client. I started with the tutorial example that appears at:
http://twistedmatrix.com/projects/mail/documentation/tutorial/smtpclient/smtpclient.html
( is indenting)
tutorial code
import StringIO
from twisted.application import service
application = service.Application("SMTP Client Tutorial")
from twisted.application import internet
from twisted.internet import protocol
from twisted.internet import defer
from twisted.mail import smtp, relaymanager
class SMTPTutorialClient(smtp.ESMTPClient):
mailFrom = "tutorial_sen...@example.com"
mailTo = "tutorial_recipi...@example.net"
mailData = '''\
Date: Fri, 6 Feb 2004 10:14:39 -0800
From: Tutorial Guy
To: Tutorial Gal
Subject: Tutorate!
Hello, how are you, goodbye.
'''
def getMailFrom(self):
result = self.mailFrom
self.mailFrom = None
return result
def getMailTo(self):
return [self.mailTo]
def getMailData(self):
return StringIO.StringIO(self.mailData)
def sentMail(self, code, resp, numOk, addresses, log):
print 'Sent', numOk, 'messages'
from twisted.internet import reactor
reactor.stop()
class SMTPClientFactory(protocol.ClientFactory):
protocol = SMTPTutorialClient
def buildProtocol(self, addr):
return self.protocol(secret=None, identity='example.com')
def getMailExchange(host):
def cbMX(mxRecord):
return str(mxRecord.exchange)
return relaymanager.MXCalculator().getMX(host).addCallback(cbMX)
def cbMailExchange(exchange):
smtpClientFactory = SMTPClientFactory()
smtpClientService = internet.TCPClient(exchange, 25, smtpClientFactory)
smtpClientService.setServiceParent(application)
getMailExchange('example.net').addCallback(cbMailExchange)
end tutorial code
This nicely looks up the right MX record and sends out an email, just what I
need. Now I want to expand it to allow me to give it a list of email addresses
to send the message to (not just call this same routine multple times, which
seems wasteful and slow and doesn't use twisted's power to process the multiple
emails in multiple threads), but I'm having terrible trouble figuring out how
to do that, which tells me I'm missing a paradigm somewhere, there's something
I'm not getting.
Trouble 1 is figuring out the right way to pass additional parameters to
callbacks. Is this right:
Dosomething(that-returns-a-deferred).addCallback(Then-do-the-next-thing,
extra-parameter1, extraparameter2)
The function Then-do-the-next-thing() will receive the deferred returned
results from Dosomething() as its first argument, and extra-parameter1 and
extraparameter2 as the next two. That is as if calling:
Then-do-the-next-thing(Result-returned-by-Dosomething(),extra-parameter1,
extraparameter2). Have I got this correct?
So, if this is right, then where do I want to put the additional argument that
contains the next email address to send, if I iterate through the list and hand
each one to the email sending process like this:
elist=['ad...@domain.com','ad...@nextdomain.com'...]
for e in elist:
e2={'mxhost':'','toaddr':e}
getMailExchange(e2).addCallback(cbMailExchange)
In the tutorial, getMailExchange() is passed just the domain of the addressee,
and the sending out of the email happens when the callback returns the MX
exchange. I changed that to split the email address, so now it returns both the
full address and the mx:
def getMailExchange(addr):
host=addr.split('@')[1]
def cbMX(mxRecord):
return [addr,str(mxRecord.name)]
return relaymanager.MXCalculator().getMX(host).addCallback(cbMX)
At this point I can't figure out how to get the email address passed to
wherever it needs to go. And I don't know really where it needs to go... yikes.
In the tutorial the actual email address is hard coded into the class
SMTPTutorialClient(smtp.ESMTPClient) as a class attribute, mailTo. I need to
change that to be variable.
How do I get this value (of mailTo) changed for each of the instances created
by smtpClientFactory = SMTPClientFactory()? I think I must be confused about
the roles of Factories and Protocols.
I can't seem to figure out a way that works to get the email address passed
into the system as a variable. Rather than waste people's time by describing my
various failures, I thought I'd just ask for suggestions about the right
twisted way to do it.
Thanks for any suggestions and directions!
-Dave
___
Twisted-Python mailing list
Twisted
Re: [Twisted-Python] newbie problem with SMTPClient
On Mon, 2 Mar 2009 15:57:17 -0500, Dave Britton wrote:
I love the idea of twisted but I think I must have a twisted learning
disability, as I have gotten nowhere in what ought to be a simple matter.
I need to send out emails to small groups from my apache server running a
python cgi using mod_python, but my hosting service doesn't have a MTA. Instead
of learning how to install and configure exim I thought I would use twisted to
make a simple mail client. I started with the tutorial example that appears at:
It may be that installing exim is actually a better idea. In order to get
reliable message deliver, you'll need to handle transient failures. That
means persisting state over time (as long as several days) and performing
redelivery attempts. However, if you don't mind losing outgoing messages
when there is a transient failure...
[snip]
This nicely looks up the right MX record and sends out an email, just what I
need. Now I want to expand it to allow me to give it a list of email addresses
to send the message to (not just call this same routine multple times, which
seems wasteful and slow and doesn't use twisted's power to process the multiple
emails in multiple threads), but I'm having terrible trouble figuring out how
to do that, which tells me I'm missing a paradigm somewhere, there's something
I'm not getting.
There is a reason that just calling the top-level function you've written
once for each email might be wasteful, but I don't think it's the reason
you're thinking.
If you just naively call getMailExchange(host).addCallback(cbMailExchange)
once for each email, then you will get parallel processing. Almost as soon
as you call most Twisted APIs, they'll start an operation (to be precise,
many of them start it immediately - before they even return - and others
start it when you allow program execution flow to return to the reactor).
So if you call getMailExchange in a loop, each iteration of the loop will
start a new operation and they'll all run in parallel. That seems like
just what you're after.
The reason this isn't the most efficient solution is that your list of
email addresses might contain two addresses which have the same mail
exchange host. In this case, you could connect to that mail exchange
once and address your message to both addresses and then deliver the
message body to the mail exchange just once. This is why getMailTo
returns a list.
So I think your other questions aren't directly relevant to this problem,
but I'll answer them anyway, since they're good questions.
Trouble 1 is figuring out the right way to pass additional parameters to
callbacks. Is this right:
Dosomething(that-returns-a-deferred).addCallback(Then-do-the-next-thing,
extra-parameter1, extraparameter2)
The function Then-do-the-next-thing() will receive the deferred returned
results from Dosomething() as its first argument, and extra-parameter1 and
extraparameter2 as the next two. That is as if calling:
Then-do-the-next-thing(Result-returned-by-Dosomething(),extra-parameter1,
extraparameter2). Have I got this correct?
Yes, this is right.
So, if this is right, then where do I want to put the additional argument that
contains the next email address to send, if I iterate through the list and hand
each one to the email sending process like this:
elist=['ad...@domain.com','ad...@nextdomain.com'...]
for e in elist:
e2={'mxhost':'','toaddr':e}
getMailExchange(e2).addCallback(cbMailExchange)
I'm not sure why you switched to a dictionary here. Ignoring that, the
biggest potential problem with this code snippet is that it creates a
Deferred (the one returned by getMailExchange) and then drops it on the
floor (albeit after adding a callback). You'll almost certainly want to
build up a list of Deferreds in cases like this, and then use something
like twisted.internet.defer.gatherResults to find out when they've all
fired. Otherwise, you don't really know when your list of operations has
completed.
As to where to put extra arguments for tracking which email addresses to
send the message to next, I don't see why you'd want that in this case.
Your for loop iterates over all the addresses, so none of your callbacks
should need to know about any more addresses. That is to say, by the end
of the for loop, there are no more addresses to which the message needs to
be sent - you've started sending to all of them already.
In the tutorial, getMailExchange() is passed just the domain of the addressee,
and the sending out of the email happens when the callback returns the MX
exchange. I changed that to split the email address, so now it returns both the
full address and the mx:
def getMailExchange(addr):
host=addr.split('@')[1]
def cbMX(mxRecord):
return [addr,str(mxRecord.name)]
return relaymanager.MXCalculator().getMX(host).addCallback(cbMX)
At this point I can't figure out how to get the email address passed to
wherever it needs to go. And
[Twisted-Python] Bug in twisted.names.client
Hi, It seems the resolv file in twisted.names.client#maybeParseConfig is never closed. This creates a exceptions.IOError: [Errno 24] Too many open files: '/etc/resolv.conf (FYI I am resolving a domain very often in my script). Am I correct? Would you like me to submit a patch to correct this? Best, Nicolas http://www.m--x--m.net ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Bug in twisted.names.client
On Mon, 2 Mar 2009 22:55:26 +0100, Nicolas Toper wrote: Hi, It seems the resolv file in twisted.names.client#maybeParseConfig is never closed. This creates a exceptions.IOError: [Errno 24] Too many open files: '/etc/resolv.conf (FYI I am resolving a domain very often in my script). Am I correct? Would you like me to submit a patch to correct this? It's true that it isn't explicitly closed and fixing this would be nice. However, unless you're using Jython or PyPy I don't think this is the cause of your IOError. As soon as maybeParseConfig returns, CPython will close the file, so it's not really a resource leak. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Bug in twisted.names.client
Good point, you are right. Thanks :) I can still commit a patch: really there just need a finally clause at the end of maybeParseConfig to close the file AFAI Best, Nicolas On Mon, Mar 2, 2009 at 11:06 PM, Jean-Paul Calderone wrote: > On Mon, 2 Mar 2009 22:55:26 +0100, Nicolas Toper wrote: > >> Hi, >> >> It seems the resolv file in twisted.names.client#maybeParseConfig is >> never >> closed. This creates a exceptions.IOError: [Errno 24] Too many open >> files: '/etc/resolv.conf >> >> (FYI I am resolving a domain very often in my script). >> >> Am I correct? Would you like me to submit a patch to correct this? >> > > It's true that it isn't explicitly closed and fixing this would be nice. > However, unless you're using Jython or PyPy I don't think this is the > cause of your IOError. As soon as maybeParseConfig returns, CPython will > close the file, so it's not really a resource leak. > > Jean-Paul > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Bug in twisted.names.client
On Mon, 2 Mar 2009 23:33:57 +0100, Nicolas Toper wrote: Good point, you are right. Thanks :) I can still commit a patch: really there just need a finally clause at the end of maybeParseConfig to close the file AFAI A patch (with unit test :) would be quite welcome. Please open a ticket at http://twistedmatrix.com/trac/ (you'll need to register an account first) and attach it there. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted developers: please donate your time!
On 2 Mar, 02:08 pm, ita...@itamarst.org wrote: On Tue, 2009-02-24 at 21:25 +1300, Michael Hudson wrote: 2009/2/24 Itamar Shtull-Trauring : > As part of the TSF's fundraising efforts, we are trying to get upfront > donations of time you will spend developing Twisted. It will then get > matched by a donor, if we are successful in getting this grant. So if > you're planning on working on Twisted anyway this year, your work will > count twice as much! > > So, if you're interested, please reply, saying something like "I will > spend two weeks working on Gopher support over the next year." I don't entirely understand, but I will commit to spending the equivalent of two working weeks as a general review monkey over the next year. Is that the sort of thing you wanted to hear? Yes - thanks to you and all thee rest of the people who are volunteering! Anyone else willing to step up? Sorry for the delay - I started writing a program to calculate my minimum probable contribution over the next year, and got stuck on that :). In addition to attending most of the sprints, which add up to roughly three weeks of work (including the PyCon sprint), I plan to spend at least the equivalent of one week on general reviews and one week on new development on the coming year. So let's call that 5 weeks of work. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
