I'd like to propose adding functionaly to generate pre-signed URLs with s3cmd. Using pre-signed GET and PUT urls is a common pattern I deal with in distributed systems, where I have a central app that has the secret AWS keys and is delegating work onto a work queue. I need the workers to be able to GET and PUT objects from S3 without needing the AWS keys, and using pre- signed URLs is the best solution.
I see s3cmd already supports signing an arbitrary string, but I found this
difficult to use for making URLs in a shell script, mostly due to escaping:
#!/bin/bash
EXPIRES=$(date +%s --date "now 100000000 seconds")
SIGNATURE=$(s3cmd -c $S3CFG sign $'GET\n\n\n'$EXPIRES$'\n/'$OBJECT_PATH)
SIGNATURE=${SIGNATURE#Signature: }
SIGNATURE="$(perl -MURI::Escape -e 'print uri_escape($ARGV[0]);'
"$SIGNATURE")"
echo "
http://s3.amazonaws.com/$OBJECT_PATH?AWSAccessKeyId=$AWS_ACCESS_KEY_ID&Expires=$EXPIRES&Signature=$SIGNATURE
"
Attached is my attempt at adding this to s3cmd, and how it can be used with
cURL:
# pre-signed GET url that expires in 10 seconds
$ s3cmd url s3://mybucket/myobject.tgz
http://mybucket.s3.amazonaws.com/myobject.tgz?AWSAccessKeyId=AKIAI4YIPRHOVWWVWASQ&Expires=1307318865&Signature=AWTr4DGs4GvX/8MCJob/FScXaXc%3D
# pre-signed GET url that expires far in the future
$ s3cmd --method=GET --ttl=100000000 url s3://mybucket/myobject.tgz
http://mybucket.s3.amazonaws.com/myobject.tgz?AWSAccessKeyId=AKIAI4YIPRHOVWWVWASQ&Expires=1407319391&Signature=igYv9j0x6vCmkTtsKsW0ZLh0lp4%3D
# pre-signed PUT url that expires in one minute
$ s3cmd --method=PUT --ttl=60 url s3://mybucket/myobject.tgz
http://mybucket.s3.amazonaws.com/myobject.tgz?AWSAccessKeyId=AKIAI4YIPRHOVWWVWASQ&Expires=1307319107&Signature=rZ3Bm7ANTQMSgt4GJCrRwEXgs7Q%3D
# use with cURL, so workers don't need AWS keys or Python
curl $(s3cmd url s3://mybucket/myobject.tgz) -o /tmp/myobject.tgz
curl -X PUT -H 'Content-Type:' --data-binary @/tmp/myobject.tgz $(s3cmd
--method=PUT url s3://mybucket/myobject.tgz)
-Noah Zoschke
n...@heroku.com
pre-signed-urls.diff
Description: Binary data
------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________ S3tools-general mailing list S3tools-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/s3tools-general
