Hi Matt and all,
Thanks for the work with the signature v4.
I have a question about that work but I don't really know where to discuss
of that, so send it here.
Before the merge, I was thinking about replacing the "limited string
encoding code" inside "urlencode_string()" (S3/S3.py), by urlquote_plus(),
like what is done in signature_v4. As that looks to be the way right way
that should be encoded. (Also it is done like that in boto).
For info, urlencode_string() is used in "create_request()" to encode the
object uri string that is also used for generating the signature.
Currently I see that all the "urlquote" things are done in the sign_v4 code.
First, I'm wondering if there is not double quoting for the resource uri
when using signature v4. (That, sadly I have not tested).
Also, I'm wondering whether it wouldn't be better to have most of the
quoting that are done inside sign_v4 functions moved earlier in the
function stack, as the quoting that is done in the signature code, should
certainly be also done for the http request parameters.
What do you think?
Thank you,
--
Florent Viard
On Wed, Dec 17, 2014 at 5:52 PM, Matt Domsch <m...@domsch.com> wrote:
>
> With huge thanks to Vasileios Mitrousis (@vamitrou) and Michal Ludvig,
> s3cmd now supports AWS Signature v4 signing method, which is required by
> the eu-central-1 (Frankfurt) region, and is supported in all AWS regions.
> s3cmd defaults to using the Signature v4 method now, though will fall back
> to older v2 method in some cases.
>
> If you are using s3cmd against a non-AWS-S3 product, such as Dreamhost
> DreamObjects, Eucalyptus, Apache CloudStack, or the like, you may need to
> use the new command line option --signature-v2 to force it to fall back,
> until these systems (and your specific instances thereof) add support for
> signature v4.
>
>
> Also, huge thanks to Florent Viard (@fviard) for his persistence in
> reviewing and aggregating many outstanding pull requests, which are now
> merged.
>
>
> New options include:
> --signature-v2 force use of signature v2 method
> --ssl force use of https
> --no-ssl disable use of https
> --region synonym for --bucket-location (to match aws-cli
> behavior)
> --no-reduced-redundancy force standard storage policy instead of reduced
> redundancy
> --remove-header allow headers to be removed as well as added
> --ca-certs=FILE allow use of a sever self-signed certificate for https,
> where you know the matching CA-CERT
> --check-certificate (default) expect server certificate to be valid,
> including hostname
> --no-check-certificate disable hostname checking on an SSL certificate,
> needed when a hostname doesn't match its wildcard certificate.
>
>
> There have been quite a few fixes since the 1.5.0-rc1 release. I'm not
> quite ready to do a -rc2, but I do want to encourage people to get the
> github.com/s3tools/s3cmd
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__github.com_s3tools_s3cmd&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=zq-uMtjYpZFDpO-8D169_MAmk6B6ofuCvheGxJU8FGo&e=>
> master branch, give it a try, and report any outstanding issues.
>
> Thanks,
> Matt
>
>
> Shortlog:
>
> Damian Gerow (2):
> Support proxying SSL connections
> Only support proxied SSL with Python >= 2.7
>
> Daniel Harris (1):
> Fix potential divide by zero errors
>
> Eric Mill (1):
> rewrite of README into markdown
>
> Florent Viard (9):
> Minor changes in README to fix points 1 and 2 in my comment at
> https://github.com/s3tools/s3cmd/pull/369
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_s3tools_s3cmd_pull_369&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=No0iijJToKpV7k2anQ90FpprB5volMHDxIbJIEQBXFg&e=>
> .
> Cosmetic: removed one empty space in excess of end of run summary.
> Fix MD5 comparison with s3 compatible servers that are not Amazon S3.
> Small cleanup of code, using "in" instead of "find" for strings when
> possible.
> Fix an error in my previous commit:
> b5b61abc8eb159f5932b936c7ca47e618615f6a3. (.find() forgotten to be removed)
> Improve the exit code of s3cmd in the case of an error from S3 to
> give a clearer status of the reason of the failure. Some existing exit
> codes were modified when they were not already in use.
> Disable dns support for buckets when "host_bucket" hostname doesn't
> have a %(bucket)s item in it.
> Backported changes from the following pull request:
> https://github.com/s3tools/s3cmd/pull/369
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_s3tools_s3cmd_pull_369&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=No0iijJToKpV7k2anQ90FpprB5volMHDxIbJIEQBXFg&e=>
> regarding aws pricing.
> Updated README.md with latest changes of S3 prices:
> https://aws.amazon.com/fr/blogs/aws/aws-data-transfer-price-reduction/?sc_ichannel=EM&sc_icountry=global&sc_icampaigntype=launch&sc_icampaign=EM_128506540&sc_idetail=em_1778261631&ref_=pe_411040_128506540_12
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__aws.amazon.com_fr_blogs_aws_aws-2Ddata-2Dtransfer-2Dprice-2Dreduction_-3Fsc-5Fichannel-3DEM-26sc-5Ficountry-3Dglobal-26sc-5Ficampaigntype-3Dlaunch-26sc-5Ficampaign-3DEM-5F128506540-26sc-5Fidetail-3Dem-5F1778261631-26ref-5F-3Dpe-5F411040-5F128506540-5F12&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=3omO2nUE4XDWzrHcIrQRzJFchDDFExZoY9tRArLW-x4&e=>
>
> Gianni Chiappetta (1):
> Always attempt to set the content-type
>
> Kiana McNellis (2):
> Fixed bug where the error report will not show if there was an
> import error due to dependency on unicodise which was not successfully
> imported
> Changed the exit code for import errors to not rely on dependency of
> EX_GENERAL being defined when not imported
>
> Lance Batson (1):
> fix issue with error display after config modify
>
> Matt Domsch (54):
> Merge pull request #361 from staer/patch-1
> Merge pull request #360 from intothev01d/config-ws-strip
> Revert "fix issue with error display after config modify"
> compare_filelists size check needs both objects to have a size value
> Check that HTTP 400 response includes data
> debug() the signature-v4 headers
> refactor content_type() and add object_replace(), tests
> Merge pull request #386 from hrchu/bug/cannotMoveLargeFile
> Add --ca-cert=FILE option for python 2.7.9ish
> Have CloudFront use ConnMan class
> create default context, then load in cert chain if needed
> handle Debian/Fedora differences in new ssl libraries
> Merge pull request #430 from fviard/pull_request_batch2
> Use ca_certs_file="" so it will load from ~/.s3cfg
> read/modify/write on modify command the headers and ACL
> clarify sanitize_headers() list
> add --remove-headers option, used in [modify]
> bugfixes in content-type fixes
> tests: modify command doesn't need --acl-public anymore
> [modify] only change mime-type if it's passed on the command line
> tests: [modify] add and remove cache-control header
> tests: s/curl/wget/
> [modify] compare headers in lowercase
> Delete the lower-case version of the header
> Merge pull request #435 from gandikun/master
> Merge pull request #428 from mdomsch/bug/content-type
> set_acl(): return 501 Not Implemented for DreamHost DreamObjects.
> add missing import in S3/S3Uri.py
> run-tests: rename (NoSuchKey) exit code change
> add to list of regions in --help --bucket-location
> add --region alias for --bucket-location
> Merge pull request #436 from vszakats/patch-1
> Arch Linux 2014.12.01 and Fedora rawhide (will be F22) now includes
> Python 2.7.9 which turns on SSL certificate validation. This leads to
> to a problem: The Amazon S3 wildcard certificate specifies *.
> s3.amazonaws.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__s3.amazonaws.com&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=uzbDxoUaYUVWCL3wJ2yk_Ew-YJxBR8pebtoQZcSb9t0&e=>
> which does not match any DNS-style buckets. So you have to use
> --no-check-certificate for operations on these buckets.
> finish merge with master
> Don't call set_tunnel() for http (not https) connections
> add debug messages for http/https proxied/non-proxied connections
> Merge pull request #431 from mdomsch/bug/426-specify-SSL-CA-file
> fix --{no-}check-certificate and non-proxied https connections
> Automatically disable ssl certificate hostname checking against AWS
> S3
> Merge pull request #438 from mdomsch/bug/check-certificate
> explicitly use python2 as shell
> finish merge of signature-v4 into master
> finish merge cleanup
> CloudFront: fix connection put() using ConnMan
> lowercase all headers added with --add-header
> use lower() in the debug message too
> Warn if requests module is not present
> Merge branch 'bug/lowercase-headers' into upstream-master
> remove unused import requests
> remove extraneous import ExitCodes
> Update s3cmd --help with changes made to manpage.
> Update spec instructions
> Add --signature-v2 and signature_v2 config option
> Fix multipart uploads with v4 signature code
>
> Michal Ludvig (6):
> Merge branch 'master' of git://github.com/s3tools/s3cmd
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__github.com_s3tools_s3cmd&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=zq-uMtjYpZFDpO-8D169_MAmk6B6ofuCvheGxJU8FGo&e=>
> Merge pull request #4 from s3tools/master
> S3Request - Include 'body' in 'request' in preparation for AWS4
> Signature
> AWS4 - Moved signing methods to Crypto.py
> Merge ssh://github.com/s3tools/s3cmd
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__github.com_s3tools_s3cmd&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=zq-uMtjYpZFDpO-8D169_MAmk6B6ofuCvheGxJU8FGo&e=>
> Merge AWS4 Crypto Work in Progress
>
> Mikhail Gusarov (1):
> Specify default checks performed on files being synced
>
> PanManAms (1):
> Changed prices
>
> Shunichi Shinohara (1):
> Specify content-type for request with XML body
>
> Sugandi (1):
> Fix README wrong filename
>
> Vasileios Mitrousis (5):
> signature v4 signing
> added comment in signature selection
> added temporary bucket to region mapping
> handle status 400 - location, when no data are returned by API
> fix in params urlencode
>
> Viktor Szakáts (1):
> added options -s/--ssl and --no-ssl options
>
> hrchu (1):
> Fix removeNameSpace parsing problem
>
> sralmai (1):
> Unbreak Config on empty options
>
> vamitrou (2):
> fix in bad region responses
> fixed put file and multipart upload
>
>
>
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__pubads.g.doubleclick.net_gampad_clk-3Fid-3D164703151-26iu-3D_4140_ostg.clktrk&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=2VFxc1z5VWicOkDfAMfNXItfIXQetwVEmkqlOB-1o34&e=
>
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
S3tools-general mailing list
S3tools-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/s3tools-general
