[Twisted-Python] HTTP PUT a GET's streaming response with treq

[Nagy, Attila]

Hi,

I would like to copy a file between two HTTP servers in the most 
efficient manner, which probably means a streaming down/upload (GET/PUT) 
with the producer/consumer scheme.


Can this be done easily, could somebody write a simple example?

Thanks,


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] HTTP PUT a GET's streaming response with treq

[Phil Mayers]

On 04/05/17 13:04, Nagy, Attila wrote:

Hi,

I would like to copy a file between two HTTP servers in the most 
efficient manner, which probably means a streaming down/upload (GET/PUT) 
with the producer/consumer scheme.


Can this be done easily, could somebody write a simple example?


I guess it'll look something like this:

https://gist.github.com/philmayers/692d04c15a71d3905e075b30f0543f23

Only lightly tested, but seems to work. There's a lot of debugging print 
statements in the above code which can obviously be discarded or turned 
into logging statements in production use.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] HTTP PUT a GET's streaming response with treq

[Nagy, Attila]

On 05/04/2017 03:04 PM, Phil Mayers wrote:

On 04/05/17 13:04, Nagy, Attila wrote:

Hi,

I would like to copy a file between two HTTP servers in the most 
efficient manner, which probably means a streaming down/upload 
(GET/PUT) with the producer/consumer scheme.


Can this be done easily, could somebody write a simple example?


I guess it'll look something like this:

https://gist.github.com/philmayers/692d04c15a71d3905e075b30f0543f23

Only lightly tested, but seems to work. There's a lot of debugging 
print statements in the above code which can obviously be discarded or 
turned into logging statements in production use.

In the meantime I've started to write one and came out with this:
https://gist.github.com/bra-fsn/42e556ddd74ff34467b17b9a6d613b33

Are there any problems with this approach?

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] process manager for twisted daemons?

[Burak Arslan]

Hello,


On 05/04/17 02:21, jonathan vanasco wrote:
> Is anyone deploying their twisted services with something other than 
> Supervisor?  If so, I'd love to know.

We're using djb's daemontools (with the -encore patchset) to manage our
twisted and non-twisted (C++) daemons. Two advantages:

1. Biggest advantage for us: cross-platform. Every Linux / BSD distro I
know of has daemontools supported. You won't have to depend on
systemd/openrc etc. Slightly related: http://cr.yp.to/compatibility.html

2. svc, daemontools' service manager is pretty straightforward to use.
See: https://cr.yp.to/daemontools/svc.html

3. Takes care of forking, logging, log rotation, etc. Can filter logs
based on a given regexp. Logs only stdout but if you do "exec 2>&1"
before starting your daemon, you'll get stderr too.

I also find it dead simple to integrate your app with daemontools. 
Examples and other advantages: https://cr.yp.to/daemontools/faq/create.html

The encore patchset (https://untroubled.org/daemontools-encore/) adds
stuff like "email me when the process is restarted". It's not strictly
needed as you can implement it in your run script but it's a nice-to-have.

I hope this helps.

Best,
Burak

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] HTTP PUT a GET's streaming response with treq

[Phil Mayers]

On 04/05/17 14:32, Nagy, Attila wrote:


Are there any problems with this approach?


You're poking at a private member of the .original IResponse there, and 
moreover the transport object which seems risky.


The single treq.collect call to consumer.write will also not necessarily 
respect the consumer-issued flow control (even if touching the 
_transport was valid, you don't know you won't get buffered data after 
you've been asked to stop)


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] HTTP PUT a GET's streaming response with treq

[Nagy, Attila]

On 05/04/2017 04:48 PM, Phil Mayers wrote:

On 04/05/17 14:32, Nagy, Attila wrote:


Are there any problems with this approach?


You're poking at a private member of the .original IResponse there, 
and moreover the transport object which seems risky.
I would like to use the simplest (and correct of course) solution. 
Juggling with buffering/data by hand seems even more risky to me.
Your implementation seems to do things which twisted should do, which 
(also) doesn't seem to be right. :)

It would be nice if this could be solved in an elegant way.



The single treq.collect call to consumer.write will also not 
necessarily respect the consumer-issued flow control (even if touching 
the _transport was valid, you don't know you won't get buffered data 
after you've been asked to stop)

It seems to do what I want to do. :)

BTW (without reading it), I've tried your solution (thank you very much 
for it!).
From a 1073741824 bytes source file, it transferred a 1056490639 
destination file.

Another run made a 1044534695 bytes destination file.
Well, each run yields a different size.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] HTTP PUT a GET's streaming response with treq

[Manish Tomar]

Have you looked at
http://treq.readthedocs.io/en/latest/howto.html#handling-streaming-responses
?

Regards,
Manish

On Thu, May 4, 2017 at 5:04 AM, Nagy, Attila  wrote:

> Hi,
>
> I would like to copy a file between two HTTP servers in the most efficient
> manner, which probably means a streaming down/upload (GET/PUT) with the
> producer/consumer scheme.
>
> Can this be done easily, could somebody write a simple example?
>
> Thanks,
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] [ANN] bloc - group membership framework

[Manish Tomar]

Hi all,

I've implemented a simple single-master group membership framework on
Twisted that can be used for mutually partitioning workload among multiple
nodes.

https://github.com/manishtomar/bloc

Please do try it out and let me know if it helps in any way.

Regards,
Manish
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [ANN] bloc - group membership framework

[Glyph]

> On May 4, 2017, at 12:03 PM, Manish Tomar  wrote:
> 
> Hi all,
> 
> I've implemented a simple single-master group membership framework on Twisted 
> that can be used for mutually partitioning workload among multiple nodes. 
> 
> https://github.com/manishtomar/bloc 
> 
> Please do try it out and let me know if it helps in any way.

Thanks Manish!  This README looks like a great intro.

Are there any plans to extend this to something consensus-based or is there a 
range of applications for which single-master is better?

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python