Re: [Twisted-Python] Multiple reactors, connecting to self, or other solution?

[Glyph Lefkowitz]

> On Nov 15, 2015, at 9:54 AM, Itamar Turner-Trauring  
> wrote:
> 
> On 11/15/2015 10:19 AM, Oon-Ee Ng wrote:
>> Based on my reading/searching, multiple reactors in the same process
>> (even in multiple threads) is pretty much a no-go because
>> twisted.internet.reactor is a global singleton.
>> 
>> I'm also unable to find any information about connecting to self (for
>> example, to send messages from one reactor to itself).
>> 
>> 
> 
> You can just have a single reactor. E.g. if you do a listenTCP (e.g. on port 
> 8080) on the reactor you can in the same process do a connectTCP to localhost 
> in the same process on the same reactor; just connect to '127.0.0.1' or 
> 'localhost' on port 8080.

But of course you'd use 
, not 
'listenTCP' and 'connectTCP' directly, right? :)

-glyph


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Multiple reactors, connecting to self, or other solution?

[Pantelis Theodosiou]

On Wed, Nov 18, 2015 at 8:55 AM, Glyph Lefkowitz 
wrote:

>
> > On Nov 15, 2015, at 9:54 AM, Itamar Turner-Trauring 
> wrote:
> >
> > On 11/15/2015 10:19 AM, Oon-Ee Ng wrote:
> >> Based on my reading/searching, multiple reactors in the same process
> >> (even in multiple threads) is pretty much a no-go because
> >> twisted.internet.reactor is a global singleton.
> >>
> >> I'm also unable to find any information about connecting to self (for
> >> example, to send messages from one reactor to itself).
> >>
> >>
> >
> > You can just have a single reactor. E.g. if you do a listenTCP (e.g. on
> port 8080) on the reactor you can in the same process do a connectTCP to
> localhost in the same process on the same reactor; just connect to
> '127.0.0.1' or 'localhost' on port 8080.
>
> But of course you'd use <
> https://twistedmatrix.com/documents/15.4.0/core/howto/endpoints.html>,
> not 'listenTCP' and 'connectTCP' directly, right? :)


> -glyph
>
>
Glyph, the linked page:
https://twistedmatrix.com/documents/15.4.0/core/howto/servers.html has
endpoints example but the "Putting it All Together" still uses:

reactor.listenTCP(8123, ChatFactory())


Wouldn't it be better if that was replaced, too? Or maybe both versions be
visible on the same webpage? (one with listenTCP and one with endpoints)?

Pantelis
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Implementing Postfix Inet Policy Check Client

[Tom Boland]

Hi Exvito,

Thanks very much for this. I think I might just give a bit of context
about using this.

I have an SMTP proxy using Nginx.  It uses queries an HTTP service to
establish whether mail should be forwarded, and where to (on to
postfix/dovecot/exchange backends).  It uses Twisted for the HTTP
authorisation service, and I've been using a similar model for years for
POP and IMAP in production, and for the last year on an outbound SMTP
relay.  The Twisted http service handles password authentication as well
lookups that point connections on to the correct backend servers.  All
of our custom business logic is in the python (database lookups and the
like).  Twisted has performed brilliantly as the HTTP service for years.

I'm now writing an inbound SMTP proxy service, and initially, it just
does DNSBL checks, which will actually stop around 12 million emails a
day from reaching postfix and tying up smtpd processes!  I fire off a
number of DNSBL checks in a deferred chain, and just come back to it
once the chain has completed.  The postfix policy check was going to be
added to this chain of events and initially I was going to use it to
query the dovecot quota daemon running on the appropriate backend for a
given mailbox, therefore I'd have a few tens of connections open, one to
each of the policy daemons on the dovecot backend servers.  This is just
to avoid queuing and trying to deliver mail to a mailbox that's over-quota.

I think what you've provided me with is useful for me, but I think it's
backwards for my purposes, as I need to be connecting to the policy
daemon rather than being the policy daemon!

I wanted to do this with deferred calls in case one of the policy
daemons becomes unreachable and blocks my application.  Do you think I
should do something differently in that regard?  My SQL lookups are done
synchronously.  If the database server goes away, I've got bigger
problems anyway!

Many thanks for your help.  I'll work a bit on this this morning and
come back!

Thanks again.  Tom.



On 18/11/15 02:14, Ex Vito wrote:
> Tom,
>
> I guess LineReceiver can help you with that, it's a matter of using a
> different delimiter. Here is a rough sketch of something you can start
> exploring:
>
> from __future__ import print_function
>
> from twisted.internet import reactor, protocol, endpoints, defer
> from twisted.protocols import basic
>
>
> class PostfixProtocol(basic.LineReceiver):
>
> # Assuming Postfix sends '\r\n' line breaks (does it?)
> delimiter = '\r\n\r\n'
>
> def lineReceived(self, multi_line):
> input_dict = {
> k: v for k, v in (
> line.split('=') for line in multi_line.split('\r\n')
> )
> }
> self.gotPostfixRequest(input_dict)
>
> def gotPostfixRequest(self, request_dict):
> # Silly deferred-based implementation.
> d = defer.Deferred()
> d.addCallback(self.sendPostfixAction)
> # Simulate a deferred being fired with success.
> reactor.callLater(1, d.callback, 'OK')
>
> def sendPostfixAction(self, response):
> # NOTE: Sends self.delimiter after the payload.
> #   Use self.tranport.write if you don't want it.
> self.sendLine('action={}'.format(response))
>
>
> if __name__ == '__main__':
>
> ep = endpoints.serverFromString(reactor, 'tcp:1')
> f = protocol.Factory()
> f.protocol = PostfixProtocol
> ep.listen(f)
> reactor.run()
>
> Key ideas:
> - PostfixProtocol overrides LineReceiver's delimiter (setting it to
> '\r\n\r\n')
> - lineReceived parses the muti_line and calls gotPostfixRequest.
> - gotPostfixRequest should decide (or delegate, you choice) what kind
> of response to send back.
> - sendPostfixAction sends a response back to Postfix.
>
> Notes:
> - The multi_line parsing code is short(ish) but not very robust. It
> may fail in bad ways with invalid input.
> - The gotPostfixRequest implementation is written for the sake of a
> deferred-based example.
>
> Does this help you in any way?
> Cheers,
> --
> exvito
>
>
> On Tue, Nov 17, 2015 at 4:56 PM, Tom Boland  > wrote:
> > Greetings all.
> >
> > This may be a very basic question.  I'm hoping to implement a postfix
> > policy check client in twisted.  It's a simple protocol.  You send
> > newline separated key value pairs like:
> >
> > recipient=em...@ddr.ess
> > sender=anot...@ddr.ess
> > size=1024
> > helo_name=mail.server
> > etc..
> >
> > you terminate the request with an additional newline.  The response
> > comes back like
> >
> > action=OK
> >
> > You can send mutliple requests in the same connection.  What I'm
> > envisaging is a module that can be used to provide a deferred
> > request/response pairing to my calling application.  The module class
> > will manage the single connection to the postfix policy daemon (I'm
> > actually going to have persistent connections to a few daemons), and
> > reconnect when necessary etc.  Any request

Re: [Twisted-Python] Multiple reactors, connecting to self, or other solution?

[Glyph Lefkowitz]

> On Nov 18, 2015, at 1:21 AM, Pantelis Theodosiou 
>  wrote:
> 
> Wouldn't it be better if that was replaced, too? Or maybe both versions be 
> visible on the same webpage? (one with listenTCP and one with endpoints)?

Absolutely!  In this high-level explanation of how to write a server, really 
only the endpoints example should be shown.  Would you mind filing a ticket 
(and perhaps also sending a patch? :))

-glyph___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Streaming HTTP

[Cory Benfield]

> On 18 Nov 2015, at 05:55, Glyph Lefkowitz  wrote:
> 
> Does HTTP2 have choke/unchoke notifications on individual streams?  
> Ultimately this does resolve to TCP backpressure, though…

Not in such a binary form, no. The connection as a whole and each stream 
maintain independent flow control windows. This allows for pressure to be 
exerted on the sender to slow down, by allowing the flow control window to drop 
to zero. This means that there is some Twisted-level buffering, because we do 
have to get that data out of the socket and to queue at the application, but 
the amount of data to buffer is strictly bounded. Thus, if our application 
moves slowly, the remote side should be passively notified to slow down by the 
lack of window updates: we should only send those window updates once the 
application has actually taken some data from us.

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Twisted 15.5 Prerelease 1 Announcement

[Amber "Hawkie" Brown]

Hello everyone!

Hot off the presses comes Twisted 15.5.0pre1, the prerelease of Twisted 15.5, 
which has been described by some as "totally radical" and "off the wall".

In this release:

- Python 3.5 support on POSIX was added, and Python 2.6 support was dropped.
- More than nine additional modules have been ported to Python 3, ranging from 
Twisted Web's Agent and downloadPage, twisted.python.logfile, and many others, 
as well as...
- twistd is ported to Python 3, and its first plugin, web, is ported.
- twisted.python.url, a new URL/IRI abstraction, has been introduced to answer 
the question "just what IS a URL" in Twisted, once and for all.
- NPN and ALPN support has been added to Twisted's TLS implementation, paving 
the way for HTTP/2.
- Conch now supports the DH group14-sha1 and group-exchange-sha256 key exchange 
algorithms, as well as hmac-sha2-256 and hmac-sha2-512 MAC algorithms. Conch 
also works nicer with newer OpenSSH implementations.
- Twisted's IRC support now has a sendCommand() method, which enables the use 
of sending messages with tags.
- 55+ closed tickets overall.

As usual, it's available for download -- go here 
(https://twistedmatrix.com/Releases/pre/15.5.0pre1/) to get the prerelease 
tarballs and the full NEWS file.

Please let me know if you have any issues, as well as if you don't! If 
everything works well, that's a good thing for me to know :)

Thanks! <3

Amber Brown
Twisted Release Manager


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 15.5 Prerelease 1 Announcement

[Glyph Lefkowitz]

> On Nov 18, 2015, at 2:58 AM, Amber Hawkie Brown  
> wrote:
> 
> Hello everyone!
> 
> Hot off the presses comes Twisted 15.5.0pre1, the prerelease of Twisted 15.5, 
> which has been described by some as "totally radical" and "off the wall".
> 
> In this release:
> 
> - Python 3.5 support on POSIX was added, and Python 2.6 support was dropped.
> - More than nine additional modules have been ported to Python 3, ranging 
> from Twisted Web's Agent and downloadPage, twisted.python.logfile, and many 
> others, as well as...
> - twistd is ported to Python 3, and its first plugin, web, is ported.
> - twisted.python.url, a new URL/IRI abstraction, has been introduced to 
> answer the question "just what IS a URL" in Twisted, once and for all.
> - NPN and ALPN support has been added to Twisted's TLS implementation, paving 
> the way for HTTP/2.
> - Conch now supports the DH group14-sha1 and group-exchange-sha256 key 
> exchange algorithms, as well as hmac-sha2-256 and hmac-sha2-512 MAC 
> algorithms. Conch also works nicer with newer OpenSSH implementations.
> - Twisted's IRC support now has a sendCommand() method, which enables the use 
> of sending messages with tags.
> - 55+ closed tickets overall.
> 
> As usual, it's available for download -- go here 
> (https://twistedmatrix.com/Releases/pre/15.5.0pre1/) to get the prerelease 
> tarballs and the full NEWS file.
> 
> Please let me know if you have any issues, as well as if you don't! If 
> everything works well, that's a good thing for me to know :)
> 
> Thanks! <3
> 
> Amber Brown
> Twisted Release Manager

Fantastic job yet again - 2015 has seen some very exciting milestones and a 
record number of releases.  Thanks, Amber, for making that happen!

-glyph



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Streaming HTTP

[Glyph Lefkowitz]

> On Nov 13, 2015, at 4:36 AM, Cory Benfield  wrote:
> 
> Folks,
> 
> # Problem Statement
> 
> Thanks for your feedback on my HTTP/2 questions. I’ve started work 
> implementing a spike of a HTTP/2 protocol for twisted.web. I’m aiming to have 
> something that works in at least some cases by the end of the day.

Hooray!

> As part of my dive into twisted.web, I noticed something that surprised me: 
> it seems to have no support for ‘streaming’ request bodies. By this I mean 
> that the Request.requestReceived() method is not actually called until the 
> complete request body has been received. This is a somewhat unexpected 
> limitation for Twisted: why should I have to wait until the entire body has 
> been uploaded to start doing things with it?

This is exactly what  is about (as 
you note).

> This problem is thrown into sharp relief with HTTP/2, which essentially 
> always chunks the body, even if a content-length is provided. This means that 
> it is now very easy to receive data in delimited chunks, which an 
> implementation may want to have semantic meaning. However, the request is 
> unable to access this data in this way. It also makes it impossible to use a 
> HTTP/2 request/response pair as a long-running communication channel, as we 
> cannot safely call requestReceived until the response is terminated (which 
> also terminates the HTTP/2 stream).
> 
> Adi pointed me at a related issue, #6928[0], which itself points at what 
> appears to be an issue tracking exactly this request. That issue is issue 
> #288[1], which is 12 years old(!). This has clearly been a pain point for 
> quite some time.
> 
> Issue #6928 has glyph suggesting that we come to the mailing list to discuss 
> this, but the last time it was raised no responses were received[2]. I 
> believe that with HTTP/2 on the horizon, this issue is more acute than it was 
> before, and needs solving if Twisted is going to continue to remain relevant 
> for the web. It should also allow people to build more performant web 
> applications, as they should be able to handle how the data queues up in 
> their apps.
> 
> This does not immediately block my HTTP/2 work, so we can take some time and 
> get this right.

I'm very glad to hear you say this, because:

(A) we should absolutely make progress on this ticket now that there is some 
impetus to do so, but,
(B) we should absolutely NOT let this new API block any purely protocol-level 
HTTP2 work that needs to proceed (I say "protocol level" as a contrast to 
"feature level" because we could totally implement a thing that speaks HTTP 2 
against the current API, although it might be awkward to expose the 
advantageous parts of the protocol as API features until we do some cleanup)

> # Proposed Solution
> 
> To help us move forward, I’m providing a proposal for how I’d solve this 
> problem. This is not necessarily going to be the final approach, but is 
> instead a straw-man we can use to form the basis of a discussion about what 
> the correct fix should be.
> 
> My proposal is to deprecate the current Request/Resource model. It currently 
> functions and should continue to function, but as of this point we should 
> consider it a bad way to do things, and we should push people to move to a 
> fully asynchronous model.

We have considered it a bad way to do things for a long time.  There have been 
several attempts to rewrite it (Nevow's request model, web2) but none of them 
have really been the comprehensive re-design we need.

> We should then move to an API that is much more like the one used by Go: 
> specifically, that by default all requests/responses are streamed. Request 
> objects (and, logically, any other object that handles requests/responses, 
> such as Resource) should be extended to have a chunkReceived method that can 
> be overridden by users.

No.

Let me elaborate :-).

First of all, this is already the case (sort of).

twisted.web.server.Request inherits from twisted.web.http.Request; as you can 
see in http.Request, there is already a method called handleContentChunk, which 
is called by HTTPChannel. By overriding this method, you can already handle 
request data streaming in from off the wire.  This is one of the reasons that 
#288 is so interminable: using Twisted's public API, today, you can totally 
write an HTTP-based application that happily streams data from the wire.  The 
only problem is that this API does not propagate up to Resource objects, 
because Resource objects can expect the (still, as of this writing, 
undocumented) "content" attribute to have been filled out in getChild.  Mostly, 
they don't, actually!  But it's impossible to tell if they might in the general 
case.

You can (and many applications have) just broken the technical compatibility 
contract with Resource, and written a subclass of twisted.web.server.Site that 
has a custom requestFactory method that returns a 'streamed' resource.

So, if we're

Re: [Twisted-Python] Streaming HTTP

[Glyph Lefkowitz]

> On Nov 18, 2015, at 2:28 AM, Cory Benfield  wrote:
> 
> 
>> On 18 Nov 2015, at 05:55, Glyph Lefkowitz  wrote:
>> 
>> Does HTTP2 have choke/unchoke notifications on individual streams?  
>> Ultimately this does resolve to TCP backpressure, though…
> 
> Not in such a binary form, no. The connection as a whole and each stream 
> maintain independent flow control windows. This allows for pressure to be 
> exerted on the sender to slow down, by allowing the flow control window to 
> drop to zero. This means that there is some Twisted-level buffering, because 
> we do have to get that data out of the socket and to queue at the 
> application, but the amount of data to buffer is strictly bounded. Thus, if 
> our application moves slowly, the remote side should be passively notified to 
> slow down by the lack of window updates: we should only send those window 
> updates once the application has actually taken some data from us.

So the receiver sends explicit "unchoke" messages to increase the window, and 
the "choke" message is implicit?

-glyph
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Streaming HTTP

[Cory Benfield]

> On 18 Nov 2015, at 12:30, Glyph Lefkowitz  wrote:
> 
> So the receiver sends explicit "unchoke" messages to increase the window, and 
> the "choke" message is implicit?

Modulo some irrelevant distinctions, yes.

Cory



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Migration to Git / GitHub

[Jonathan Vanasco]

On Nov 17, 2015, at 12:52 PM, Donald Stufft wrote:

> I don’t disagree with the decision to not move to GitHub issues, however
> if you wanted to you could do what Golang did to minimize the data loss.
> They emailed prior participants to their bug tracker to ask them to grant
> their migration app an OAuth grant to post messages in their name. Anyone
> who did got the correct user account associated with their messages, and
> anyone who didn’t got the catch all golang migration bot.

That's a great partial solution.

On top of that I would suggest the following:

Put the trac into read-only mode.
Use the API to generate a "github wiki page" for each "trac user".
When importing, create a "template" for migrated issues that 
cross-references to the original ticket and the wiki.  Something like this:
** Migrated Issue **
Original Trac Ticket (read-only): {{trac url}}
AutoGenerated Author Wiki: {{github wiki url}}

Original Body:
{{body contents}}

Original Comments (if any):

{{format comments and link to the github wiki url}}

GitHub's wiki doesn't have normal API access.  It's actually just a 'secret' 
git  [https://help.github.com/articles/adding-and-editing-wiki-pages-locally/]

So you won't necessarily "lose" author information -- you'll just have the 
author info represented in another manner.

If any authors become known, you can just edit the wiki to reference their 
github account from that page (or possibly update the issue in the api, not 
sure).___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Implementing Postfix Inet Policy Check Client

[exvito here]

On Wed, Nov 18, 2015 at 9:28 AM, Tom Boland  wrote:

> I think what you've provided me with is useful for me, but I think it's
> backwards for my purposes, as I need to be connecting to the policy daemon
> rather than being the policy daemon!
>
> I wanted to do this with deferred calls in case one of the policy daemons
> becomes unreachable and blocks my application.  Do you think I should do
> something differently in that regard?  My SQL lookups are done
> synchronously.  If the database server goes away, I've got bigger problems
> anyway!
>

So maybe something like this is more likely to be useful:

#!/usr/bin/env python

from __future__ import print_function

from twisted.internet import reactor, protocol, endpoints, defer
from twisted.protocols import basic


class PostfixProtocol(basic.LineReceiver):

# Assuming Postfix uses '\r\n' line breaks (does it?)
delimiter = '\r\n'

def __init__(self):
self.action = None
self.action_deferred = None

def lineReceived(self, line):
if '=' in line:
self.action = line.split('=')[1]
elif line == '':
self.action_deferred.callback(self.action)
self.action_deferred = None
else:
# oops, bad input
pass

def sendPostfixRequest(self, request_dict):
if not self.action_deferred is None:
raise Exception('transaction pending')
for k, v in request_dict.items():
self.sendLine('{}={}'.format(k,v))
# Empty line signals we're done
self.sendLine('')
self.action_deferred = defer.Deferred()
return self.action_deferred

@defer.inlineCallbacks
def checkPostfixPolicy(request_dict):
ep = endpoints.clientFromString(reactor, 'tcp:host=127.0.0.1:
port=1')
p = yield endpoints.connectProtocol(ep, PostfixProtocol())
action = yield p.sendPostfixRequest(request_dict)
print('got: {}'.format(action))
reactor.stop()


if __name__ == '__main__':

request_dict = {
'recipient': 'em...@ddr.ess',
'sender': 'em...@ddr.ess',
}
reactor.callWhenRunning(checkPostfixPolicy, request_dict)
reactor.run()

Highlights:
- This is not the same protocol as before, in particular it uses a
different delimiter.
- It assumes the response is also terminated with an empty line (does it?).
- It more than one outstanding response: a different exception should be
used.
- The input processing is very rudimentary and failure-prone.
- checkPostfixPolicy could, of course, return instead of printing. :)

Cheers,
--
  exvito
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python