I am attempting to stream the output of a curl command with Mojolicious. The curl command looks (roughly) like this:
curl -H "x-amz-server-side-encryption-customer-algorithm:AES256" -H "x-amz-server-side-encryption-cust\ omer-key:secretKey=" -H "x-amz-server-side-encryption-customer-key-M\ D5:HashOfASecret==" "https://mybucket.s3.amazonaws.com/obscureFileLocation?AWSAccessKeyId=secretStuff&Expires=1588568911&Signature=moreSecrets"; --\ dump-header header.461 --silent The curl command references a pre-signed URL of a file in AWS stored with Server Side Encryption with Client Side Keys (SSE-C), and supplies the necessary key information via HTTP headers (the -H command options). The curl command works - but I don't want my users to have to have a system with curl on it to access their files. The plan is to open the curl command as input to a pipe, and stream its output to the user's browser with Mojolicious. The curl command also dumps out the HTTP headers from Amazon, so they can be used by Mojolicious. They look like this: x-amz-id-2: sgMzHD2FJEGJrcbvzQwdhZK6mxUW+ePd6xdghTfgSlV45lMhliIw4prfk4cZMTHbS4fJN8N7xio= x-amz-request-id: 99B9CA56083DD9ED Date: Mon, 04 May 2020 04:57:22 GMT Last-Modified: Sat, 02 May 2020 03:47:35 GMT ETag: "b3a11409be2705e4581119fa59af79d3-1025" x-amz-server-side-encryption-customer-algorithm: AES256 x-amz-server-side-encryption-customer-key-MD5: HashOfSecretKey== Content-Disposition: attachment; filename = "fiveGigFile" Accept-Ranges: bytes Content-Type: application/octet-stream; charset=UTF-8 Content-Length: 5368709125 Server: AmazonS3 Note that the file is 5Gig. This is my stab at streaming with Mojolicious: use strict; use Mojolicious::Lite; use FileHandle; use Digest::MD5; any '/' => sub { my $c = shift; $c->render(template => "test"); }; any 'pickup' => sub { my $c = shift; my $nGigs = 0; my $nMegs = 0; $| = 1; open(CURLCMD,"curlCmd"); my $curlCmd = <CURLCMD>; if ($curlCmd =~ /dump-header\s*(\S+)\s+/) { my $headerFile = $1; open(my $curl,"$curlCmd |"); binmode $curl; my $initialized = 0; my $digester = Digest::MD5->new; my $transferLength = 0; my $drain; $drain = sub { my $c = shift; my $chunk; sysread($curl,$chunk,1024*1024); if (!$initialized) { # read the headers, and set up the transfer... open(HEADERS,$headerFile); while(my $line = <HEADERS>) { $c->res->headers->parse($line); } close(HEADERS); $initialized = 1; print "header initialization completed for the following headers\n"; print join("\n",@{$c->res->headers->names}),"\n"; } if ($initialized) { while (length($chunk)) { $digester->add($chunk); $transferLength += length($chunk); $c->write($chunk,$drain); my $currentMegs = int($transferLength/(1024*1024)); if (($currentMegs > $nMegs) && ($currentMegs < 1024)) { print "TransferLength: $transferLength\n"; $nMegs = $currentMegs; } my $currentGigs = int($transferLength/(1024*1024*1024)); if ($currentGigs > $nGigs) { print "TransferLength: $transferLength\n"; $nGigs = $currentGigs; } } if (length($chunk) <= 1) { if ($chunk == 0) { print "End of file found on curl pipe."; print "$transferLength bytes transmitted\n"; print "with an MD5 hash of ",$digester->hexdigest,"\n"; $drain = undef; } if (!defined $chunk) { print "Transfer error encountered on curl pipe.\n"; print "Error:",$!,"\n"; $drain = undef; } } } }; $c->$drain; } }; app->start; __DATA__ @@ test.html.ep <!DOCTYPE html> <html> <body> <a href="/pickup" >Test of curl streaming... </a> </body> </html> When I ran this the first time, it read about 606MB of data, and the server crashed with an "Out of memory!". Subsequent runs failed at about 139MB, with a server crash and no "Out of memory!" message. Obviously, I am an idiot. Some guidance in the precise way I am being an idiot would be greatly appreciated. Regards, Joe Fridy -- You received this message because you are subscribed to the Google Groups "Mojolicious" group. To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/mojolicious/86c878ad-8bd2-4a40-94fd-cc16fbe39201%40googlegroups.com.
