[python-uk] Python engineer/ Nameko/ Micro-services/ Soho

2017-04-11 Thread Sophie Hendley
Hey Guys, I am looking for a (mid-level) Python developer to join one of my favourite managers in their Platform Engineering team. The role is based at the office in soho where they build things that help students (hopefully you like students). As mentioned above the guys are working on the Namek

[python-uk] Handling large file uploads

2017-04-11 Thread Hansel Dunlop
Hello all I'm working on an application that has to accept large uploads. Think ~ 2GB+ size files getting uploaded over slowish connections. These files are eventually going to end up in S3. Uploading smallish files is not a problem. But things get a bit complicated when you're dealing with large

Re: [python-uk] Handling large file uploads

2017-04-11 Thread William Mayor
> On 11 Apr 2017, at 11:21, Hansel Dunlop wrote: > > Hello all > > I'm working on an application that has to accept large uploads. Think ~ 2GB+ > size files getting uploaded over slowish connections. These files are > eventually going to end up in S3. > > Uploading smallish files is not a pr

Re: [python-uk] Handling large file uploads

2017-04-11 Thread James Browne
Hi Hansel, Here's the documentation for browser-based POSTing using S3: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingHTTPPOST.html If you want to implement chunking the file and stitching together yourself, here's quite a nice JavaScript library to help with some of the heavy lifting on th

Re: [python-uk] Handling large file uploads

2017-04-11 Thread Javier Llopis
I haven't done anything similar, but this upload in chunks then reassemble idea bears some similarity to peer to peer file sharing software. I would look there for ideas in case I got stuck with some problem. J From: "Hansel Dunlop" Sent: 11 A

Re: [python-uk] Handling large file uploads

2017-04-11 Thread Roger Gammans
Indeed. Both the rsync and bittorrent protocols split files into chunks. There are a lot of similarities bwtween these protocols. Also it might be worth looking at are de-duping backup tools like duplicity. On Tue, 2017-04-11 at 11:33 +0100, Javier Llopis wrote: > I haven't done anything similar

Re: [python-uk] Handling large file uploads

2017-04-11 Thread Hansel Dunlop
Thanks everyone, Uploading directly to S3 sounds the avenue to pursue. It removes a whole load of complications! Cheers Hansel On Tue, Apr 11, 2017 at 12:18 PM, Roger Gammans wrote: > Indeed. > > Both the rsync and bittorrent protocols split files into chunks. There are > a lot of similaritie