Re: [Twisted-Python] Streaming HTTP

[Cory Benfield]

> On 18 Nov 2015, at 12:18, Glyph Lefkowitz  wrote:
> 

Sorry about the delay in responding to this, but I wanted to make sure I knew 
at least a bit about what I was talking about before I responded!

>> What do people think of this approach?
> 
> So I think you're roughly on the right track but there are probably some 
> Twisted-level gaps to fill in.
> 
> I've already gestured in the direction of Tubes (as have others) and it's 
> something to think about.  But before we get to that, let's talk about a much 
> more basic deficiency in the API: although there's an "IRequest", and an 
> "IResource", there's no such thing as an "IResponse".  Instead, "IRequest" 
> stands in for both the request and the response, because you write directly 
> to a request (implicitly filling out its response as you do so).

So, I think in general this is interesting. One of the big difficulties I’m 
having right now is that I’m trying to combine this “streaming HTTP” work with 
the implementation of HTTP/2, which means that I need to keep the HTTP/2 work 
in mind whenever I talk about this *and* update the HTTP/2 design in response 
to decisions we make here. This means I’ve got quite a lot of balls in the air 
right now, and I am confident I’ll drop quite a few. One thing I’m deliberately 
not doing here is considering Tubes, in part because I’m extremely concerned 
about backward compatibility, and want the HTTP/2 work to function in the same 
environment.

Unfortunately, this means this conversation is blending into the HTTP/2 one, so 
I’m going to hijack this thread and bring in some concrete discussion of what 
I’m working on with the HTTP/2 stuff.

I was having a conversation about the HTTP/2 architecture on #twisted-dev 
yesterday, which has led towards my current working approach for HTTP/2, which 
will be to have two underlying objects. We’ll have H2Connection, which 
implements IProtocol, and H2Stream, which implements ITransport. These two 
objects will be *extremely* tightly coupled: H2Stream cannot meaningfully run 
over an arbitrary transport mechanism, and knows a great deal about how 
H2Connections work.

The reason we need to take this approach is because IConsumer doesn’t allow for 
us to have correlators, so even if we only had H2Connection it wouldn’t be able 
to identify a given producer with the stream it holds. By extension, IConsumer 
cannot consume multiple producers at once. For this reason, we need an 
interface between H2Connection and H2Stream that is similar to ITransport and 
IConsumer, but more featureful. Basically, H2Stream is a thin shim between a 
producer and H2Connection that adds a stream ID to a few function calls.

> Luckily we have an existing interface that might point the way to a better 
> solution, both for requests and responses: specifically, the client 
> IResponse: 
> https://twistedmatrix.com/documents/15.4.0/api/twisted.web.iweb.IResponse.html.
> 
> This interface is actually pretty close to what we want for a server 
> IResponse as well.  Perhaps even identical.  Its static data is all exposed 
> as attributes which can be relatively simply inspected, and the way it 
> delivers a streaming response is that it delivers its body to an IProtocol 
> implementation (via .deliverBody(aProtocol)).  This is not quite as graceful 
> as having a .bodyFount() method that returns an IFount from the tubes 
> package; however, the tubes package is still not exactly mature software, so 
> we may not want to block on depending on it.  Importantly though, this 
> delivers all the events you need as a primitive for interfacing with such a 
> high-level interface; it would definitely be better to add this sort of 
> interface Real Soon Now, because then the tubes package could simply have a 
> method, responseToFount (which it will need anyway to work with Agent) that 
> calls deliverBody internally.
> 
> This works as a primitive because you have all the hooks you need for 
> flow-control.  This protocol receives, to its 'makeConnection' method, an 
> ITransport which can provide the IProducer 
> https://twistedmatrix.com/documents/15.4.0/api/twisted.internet.interfaces.IProducer.html
>  and IConsumer 
> https://twistedmatrix.com/documents/15.4.0/api/twisted.internet.interfaces.IConsumer.html
>  interfaces for flow-control.  It receives dataReceived to tell it a chunk 
> has arrived and connectionLost to tell it the stream has terminated.

Just let me clarify how this is expected to work. Somewhere we have a 
t.w.s.Site, which builds some kind of HTTP protocol (currently HTTPChannel, in 
future some object that can transparently swap between HTTPChannel and 
H2Connection) when connections are received.

These two protocols each build an IGoodRequest, which is very similar to 
IRequest but has a deliverBody method. The consumer of this (whether IResource 
or some other thing). These objects, if they want to consume a stream, register 
a protocol via deliverBody. At this point, H2Connection

Re: [Twisted-Python] Implementing Postfix Inet Policy Check Client

[Tom Boland]

Thanks again for this.  It's really useful.  It turns out that the
delimiter is a plain old \n.  Who knows how consistent this will be
between different policy daemons, I don't know!

I've modified it to manage a DeferredQueue which hopefully means I can
just keep throwing requests at it without every being bounced away.  A
stripped down example (with even less error checking!) of what I've
managed to work your example in to is here:

class PostfixPolicyClient(LineReceiver):

delimiter = '\n'

def __init__(self):
self.resultQueue = DeferredQueue()


def lineReceived(self, line):
if '=' in line:
self.resultQueue.put(True if line.split('=')[1] == 'OK' else
False)


def sendPostfixPolicyClientRequest(self, request_dict):
for k, v in request_dict.items():
self.sendLine('{}={}'.format(k, v))
self.sendLine('')
return self.resultQueue.get()



Now, this isn't a working example, it's just the minimum that will
demonstrate my idea.  I just wonder if what I've done with the
DeferredQueue is sane.  If I return the .get() entry from the
DeferredQueue when doing the request, and then do a put() in
lineReceived, am I guaranteeing that I will get my results in the
correct order?

Thanks again for all your help!

Many thanks.  Tom.


On 19/11/15 07:53, exvito here wrote:
>
> On Wed, Nov 18, 2015 at 9:28 AM, Tom Boland  > wrote:
>
> I think what you've provided me with is useful for me, but I think
> it's backwards for my purposes, as I need to be connecting to the
> policy daemon rather than being the policy daemon!
>
> I wanted to do this with deferred calls in case one of the policy
> daemons becomes unreachable and blocks my application.  Do you
> think I should do something differently in that regard?  My SQL
> lookups are done synchronously.  If the database server goes away,
> I've got bigger problems anyway!
>
>
> So maybe something like this is more likely to be useful:
>
> #!/usr/bin/env python
>
> from __future__ import print_function
>
> from twisted.internet import reactor, protocol, endpoints, defer
> from twisted.protocols import basic
>
>
> class PostfixProtocol(basic.LineReceiver):
>
> # Assuming Postfix uses '\r\n' line breaks (does it?)
> delimiter = '\r\n'
>
> def __init__(self):
> self.action = None
> self.action_deferred = None
>
> def lineReceived(self, line):
> if '=' in line:
> self.action = line.split('=')[1]
> elif line == '':
> self.action_deferred.callback(self.action)
> self.action_deferred = None
> else:
> # oops, bad input
> pass
>
> def sendPostfixRequest(self, request_dict):
> if not self.action_deferred is None:
> raise Exception('transaction pending')
> for k, v in request_dict.items():
> self.sendLine('{}={}'.format(k,v))
> # Empty line signals we're done
> self.sendLine('')
> self.action_deferred = defer.Deferred()
> return self.action_deferred
>
> @defer.inlineCallbacks
> def checkPostfixPolicy(request_dict):
> ep = endpoints.clientFromString(reactor,
> 'tcp:host=127.0.0.1:port=1')
> p = yield endpoints.connectProtocol(ep, PostfixProtocol())
> action = yield p.sendPostfixRequest(request_dict)
> print('got: {}'.format(action))
> reactor.stop()
>
>
> if __name__ == '__main__':
>
> request_dict = {
> 'recipient': 'em...@ddr.ess',
> 'sender': 'em...@ddr.ess',
> }
> reactor.callWhenRunning(checkPostfixPolicy, request_dict)
> reactor.run()
>
> Highlights:
> - This is not the same protocol as before, in particular it uses a
> different delimiter.
> - It assumes the response is also terminated with an empty line (does
> it?).
> - It more than one outstanding response: a different exception should
> be used.
> - The input processing is very rudimentary and failure-prone.
> - checkPostfixPolicy could, of course, return instead of printing. :)
>
> Cheers,
> --
>   exvito
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Implementing Postfix Inet Policy Check Client

[exvito here]

On Thu, Nov 19, 2015 at 4:19 PM, Tom Boland  wrote:

> Thanks again for this.  It's really useful.  It turns out that the
> delimiter is a plain old \n.  Who knows how consistent this will be between
> different policy daemons, I don't know!
>

I would check the Postfix docs for that.


> Now, this isn't a working example, it's just the minimum that will
> demonstrate my idea.  I just wonder if what I've done with the
> DeferredQueue is sane.  If I return the .get() entry from the DeferredQueue
> when doing the request, and then do a put() in lineReceived, am I
> guaranteeing that I will get my results in the correct order?
>

The DeferredQueue is a nice approach: it ensures the get() results come out
in the same order as the put() calls.

The key question is whether or not the server handles multiple outstanding
requests within the same connection. Given your informal protocol
description, if the server supports it, it seems the responses must come
back in the same order as the requests were sent, otherwise there is
apparently no way to relate them; that being the case, what's the advantage
of pushing more than one request at a time if one slow request/response
"transaction" will delay a subsequent fast request/response "transaction"?

A variation of this, assuming the server only handles one outstanding
request at a time per connection, could be a protocol implementation that
would queue requests ensuring only one was sent at a time: this might
provide a cleaner client side API.

PS: My references to "Postfix docs" might need to be replaced by "your
policy server docs" which, hopefully, will match Postfix's... (you would
know that) :)

Cheers,
--
exvito
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] incompatible change - need revert before release

[Glyph Lefkowitz]

Tom Prince discovered a regression on 
https://twistedmatrix.com/trac/ticket/7016#comment:14 
 - I think that this was 
introduced after 15.4, so it needs to be rolled back (or fixed, if someone can 
get to it before the revert) in 15.5.

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Multiple reactors, connecting to self, or other solution?

[Oon-Ee Ng]

On Wed, Nov 18, 2015 at 4:55 PM, Glyph Lefkowitz
 wrote:
>
>> On Nov 15, 2015, at 9:54 AM, Itamar Turner-Trauring  
>> wrote:
>>
>> On 11/15/2015 10:19 AM, Oon-Ee Ng wrote:
>>> Based on my reading/searching, multiple reactors in the same process
>>> (even in multiple threads) is pretty much a no-go because
>>> twisted.internet.reactor is a global singleton.
>>>
>>> I'm also unable to find any information about connecting to self (for
>>> example, to send messages from one reactor to itself).
>>>
>>>
>>
>> You can just have a single reactor. E.g. if you do a listenTCP (e.g. on port 
>> 8080) on the reactor you can in the same process do a connectTCP to 
>> localhost in the same process on the same reactor; just connect to 
>> '127.0.0.1' or 'localhost' on port 8080.
>
> But of course you'd use 
> , not 
> 'listenTCP' and 'connectTCP' directly, right? :)
>
> -glyph


But, but... I LIKE listen/connectTCP fits better with how I think.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Multiple reactors, connecting to self, or other solution?

[Glyph Lefkowitz]

> On Nov 19, 2015, at 9:19 PM, Oon-Ee Ng  wrote:
> 
> On Wed, Nov 18, 2015 at 4:55 PM, Glyph Lefkowitz
>  wrote:
>> 
>>> On Nov 15, 2015, at 9:54 AM, Itamar Turner-Trauring  
>>> wrote:
>>> 
>>> On 11/15/2015 10:19 AM, Oon-Ee Ng wrote:
 Based on my reading/searching, multiple reactors in the same process
 (even in multiple threads) is pretty much a no-go because
 twisted.internet.reactor is a global singleton.
 
 I'm also unable to find any information about connecting to self (for
 example, to send messages from one reactor to itself).
 
 
>>> 
>>> You can just have a single reactor. E.g. if you do a listenTCP (e.g. on 
>>> port 8080) on the reactor you can in the same process do a connectTCP to 
>>> localhost in the same process on the same reactor; just connect to 
>>> '127.0.0.1' or 'localhost' on port 8080.
>> 
>> But of course you'd use 
>> , not 
>> 'listenTCP' and 'connectTCP' directly, right? :)
>> 
>> -glyph
> 
> 
> But, but... I LIKE listen/connectTCP fits better with how I think.

If you use connectTCP/listenTCP, you miss out on important functionality.  For 
example, listenTCP can't do encryption, which means it's unsuitable for use on 
the modern internet, unless your protocol calls startTLS right away.

Also you don't get stuff like https://txtorcon.readthedocs.org for free.

So, connectTCP/listenTCP are low-level APIs that should really only be used for 
*implementing* an endpoint, not used directly by applications.  If this does 
not fit with how you think then you need to change how you think :).

-glyph


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python