The ~ character in a filename is certainly being encoded as %7E, which is
incorrect - it's not supposed to be encoded at all. This is causing a v4
signature validation failure.
On Wed, Dec 17, 2014 at 12:57 PM, Florent Viard <florent.vi...@seagate.com>
wrote:
>
> Hi Matt and all,
>
> Thanks for the work with the signature v4.
> I have a question about that work but I don't really know where to discuss
> of that, so send it here.
>
> Before the merge, I was thinking about replacing the "limited string
> encoding code" inside "urlencode_string()" (S3/S3.py), by urlquote_plus(),
> like what is done in signature_v4. As that looks to be the way right way
> that should be encoded. (Also it is done like that in boto).
>
> For info, urlencode_string() is used in "create_request()" to encode the
> object uri string that is also used for generating the signature.
>
> Currently I see that all the "urlquote" things are done in the sign_v4
> code.
> First, I'm wondering if there is not double quoting for the resource uri
> when using signature v4. (That, sadly I have not tested).
> Also, I'm wondering whether it wouldn't be better to have most of the
> quoting that are done inside sign_v4 functions moved earlier in the
> function stack, as the quoting that is done in the signature code, should
> certainly be also done for the http request parameters.
>
> What do you think?
>
> Thank you,
>
>
> --
> Florent Viard
>
>
> On Wed, Dec 17, 2014 at 5:52 PM, Matt Domsch <m...@domsch.com> wrote:
>
>> With huge thanks to Vasileios Mitrousis (@vamitrou) and Michal Ludvig,
>> s3cmd now supports AWS Signature v4 signing method, which is required by
>> the eu-central-1 (Frankfurt) region, and is supported in all AWS regions.
>> s3cmd defaults to using the Signature v4 method now, though will fall back
>> to older v2 method in some cases.
>>
>> If you are using s3cmd against a non-AWS-S3 product, such as Dreamhost
>> DreamObjects, Eucalyptus, Apache CloudStack, or the like, you may need to
>> use the new command line option --signature-v2 to force it to fall back,
>> until these systems (and your specific instances thereof) add support for
>> signature v4.
>>
>>
>> Also, huge thanks to Florent Viard (@fviard) for his persistence in
>> reviewing and aggregating many outstanding pull requests, which are now
>> merged.
>>
>>
>> New options include:
>> --signature-v2 force use of signature v2 method
>> --ssl force use of https
>> --no-ssl disable use of https
>> --region synonym for --bucket-location (to match aws-cli
>> behavior)
>> --no-reduced-redundancy force standard storage policy instead of
>> reduced redundancy
>> --remove-header allow headers to be removed as well as added
>> --ca-certs=FILE allow use of a sever self-signed certificate for https,
>> where you know the matching CA-CERT
>> --check-certificate (default) expect server certificate to be valid,
>> including hostname
>> --no-check-certificate disable hostname checking on an SSL certificate,
>> needed when a hostname doesn't match its wildcard certificate.
>>
>>
>> There have been quite a few fixes since the 1.5.0-rc1 release. I'm not
>> quite ready to do a -rc2, but I do want to encourage people to get the
>> github.com/s3tools/s3cmd
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__github.com_s3tools_s3cmd&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=zq-uMtjYpZFDpO-8D169_MAmk6B6ofuCvheGxJU8FGo&e=>
>> master branch, give it a try, and report any outstanding issues.
>>
>> Thanks,
>> Matt
>>
>>
>> Shortlog:
>>
>> Damian Gerow (2):
>> Support proxying SSL connections
>> Only support proxied SSL with Python >= 2.7
>>
>> Daniel Harris (1):
>> Fix potential divide by zero errors
>>
>> Eric Mill (1):
>> rewrite of README into markdown
>>
>> Florent Viard (9):
>> Minor changes in README to fix points 1 and 2 in my comment at
>> https://github.com/s3tools/s3cmd/pull/369
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_s3tools_s3cmd_pull_369&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=No0iijJToKpV7k2anQ90FpprB5volMHDxIbJIEQBXFg&e=>
>> .
>> Cosmetic: removed one empty space in excess of end of run summary.
>> Fix MD5 comparison with s3 compatible servers that are not Amazon
>> S3.
>> Small cleanup of code, using "in" instead of "find" for strings
>> when possible.
>> Fix an error in my previous commit:
>> b5b61abc8eb159f5932b936c7ca47e618615f6a3. (.find() forgotten to be removed)
>> Improve the exit code of s3cmd in the case of an error from S3 to
>> give a clearer status of the reason of the failure. Some existing exit
>> codes were modified when they were not already in use.
>> Disable dns support for buckets when "host_bucket" hostname doesn't
>> have a %(bucket)s item in it.
>> Backported changes from the following pull request:
>> https://github.com/s3tools/s3cmd/pull/369
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_s3tools_s3cmd_pull_369&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=No0iijJToKpV7k2anQ90FpprB5volMHDxIbJIEQBXFg&e=>
>> regarding aws pricing.
>> Updated README.md with latest changes of S3 prices:
>> https://aws.amazon.com/fr/blogs/aws/aws-data-transfer-price-reduction/?sc_ichannel=EM&sc_icountry=global&sc_icampaigntype=launch&sc_icampaign=EM_128506540&sc_idetail=em_1778261631&ref_=pe_411040_128506540_12
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__aws.amazon.com_fr_blogs_aws_aws-2Ddata-2Dtransfer-2Dprice-2Dreduction_-3Fsc-5Fichannel-3DEM-26sc-5Ficountry-3Dglobal-26sc-5Ficampaigntype-3Dlaunch-26sc-5Ficampaign-3DEM-5F128506540-26sc-5Fidetail-3Dem-5F1778261631-26ref-5F-3Dpe-5F411040-5F128506540-5F12&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=3omO2nUE4XDWzrHcIrQRzJFchDDFExZoY9tRArLW-x4&e=>
>>
>> Gianni Chiappetta (1):
>> Always attempt to set the content-type
>>
>> Kiana McNellis (2):
>> Fixed bug where the error report will not show if there was an
>> import error due to dependency on unicodise which was not successfully
>> imported
>> Changed the exit code for import errors to not rely on dependency
>> of EX_GENERAL being defined when not imported
>>
>> Lance Batson (1):
>> fix issue with error display after config modify
>>
>> Matt Domsch (54):
>> Merge pull request #361 from staer/patch-1
>> Merge pull request #360 from intothev01d/config-ws-strip
>> Revert "fix issue with error display after config modify"
>> compare_filelists size check needs both objects to have a size value
>> Check that HTTP 400 response includes data
>> debug() the signature-v4 headers
>> refactor content_type() and add object_replace(), tests
>> Merge pull request #386 from hrchu/bug/cannotMoveLargeFile
>> Add --ca-cert=FILE option for python 2.7.9ish
>> Have CloudFront use ConnMan class
>> create default context, then load in cert chain if needed
>> handle Debian/Fedora differences in new ssl libraries
>> Merge pull request #430 from fviard/pull_request_batch2
>> Use ca_certs_file="" so it will load from ~/.s3cfg
>> read/modify/write on modify command the headers and ACL
>> clarify sanitize_headers() list
>> add --remove-headers option, used in [modify]
>> bugfixes in content-type fixes
>> tests: modify command doesn't need --acl-public anymore
>> [modify] only change mime-type if it's passed on the command line
>> tests: [modify] add and remove cache-control header
>> tests: s/curl/wget/
>> [modify] compare headers in lowercase
>> Delete the lower-case version of the header
>> Merge pull request #435 from gandikun/master
>> Merge pull request #428 from mdomsch/bug/content-type
>> set_acl(): return 501 Not Implemented for DreamHost DreamObjects.
>> add missing import in S3/S3Uri.py
>> run-tests: rename (NoSuchKey) exit code change
>> add to list of regions in --help --bucket-location
>> add --region alias for --bucket-location
>> Merge pull request #436 from vszakats/patch-1
>> Arch Linux 2014.12.01 and Fedora rawhide (will be F22) now includes
>> Python 2.7.9 which turns on SSL certificate validation. This leads to
>> to a problem: The Amazon S3 wildcard certificate specifies *.
>> s3.amazonaws.com
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__s3.amazonaws.com&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=uzbDxoUaYUVWCL3wJ2yk_Ew-YJxBR8pebtoQZcSb9t0&e=>
>> which does not match any DNS-style buckets. So you have to use
>> --no-check-certificate for operations on these buckets.
>> finish merge with master
>> Don't call set_tunnel() for http (not https) connections
>> add debug messages for http/https proxied/non-proxied connections
>> Merge pull request #431 from mdomsch/bug/426-specify-SSL-CA-file
>> fix --{no-}check-certificate and non-proxied https connections
>> Automatically disable ssl certificate hostname checking against AWS
>> S3
>> Merge pull request #438 from mdomsch/bug/check-certificate
>> explicitly use python2 as shell
>> finish merge of signature-v4 into master
>> finish merge cleanup
>> CloudFront: fix connection put() using ConnMan
>> lowercase all headers added with --add-header
>> use lower() in the debug message too
>> Warn if requests module is not present
>> Merge branch 'bug/lowercase-headers' into upstream-master
>> remove unused import requests
>> remove extraneous import ExitCodes
>> Update s3cmd --help with changes made to manpage.
>> Update spec instructions
>> Add --signature-v2 and signature_v2 config option
>> Fix multipart uploads with v4 signature code
>>
>> Michal Ludvig (6):
>> Merge branch 'master' of git://github.com/s3tools/s3cmd
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__github.com_s3tools_s3cmd&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=zq-uMtjYpZFDpO-8D169_MAmk6B6ofuCvheGxJU8FGo&e=>
>> Merge pull request #4 from s3tools/master
>> S3Request - Include 'body' in 'request' in preparation for AWS4
>> Signature
>> AWS4 - Moved signing methods to Crypto.py
>> Merge ssh://github.com/s3tools/s3cmd
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__github.com_s3tools_s3cmd&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=zq-uMtjYpZFDpO-8D169_MAmk6B6ofuCvheGxJU8FGo&e=>
>> Merge AWS4 Crypto Work in Progress
>>
>> Mikhail Gusarov (1):
>> Specify default checks performed on files being synced
>>
>> PanManAms (1):
>> Changed prices
>>
>> Shunichi Shinohara (1):
>> Specify content-type for request with XML body
>>
>> Sugandi (1):
>> Fix README wrong filename
>>
>> Vasileios Mitrousis (5):
>> signature v4 signing
>> added comment in signature selection
>> added temporary bucket to region mapping
>> handle status 400 - location, when no data are returned by API
>> fix in params urlencode
>>
>> Viktor Szakáts (1):
>> added options -s/--ssl and --no-ssl options
>>
>> hrchu (1):
>> Fix removeNameSpace parsing problem
>>
>> sralmai (1):
>> Unbreak Config on empty options
>>
>> vamitrou (2):
>> fix in bad region responses
>> fixed put file and multipart upload
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>> Get technology previously reserved for billion-dollar corporations, FREE
>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__pubads.g.doubleclick.net_gampad_clk-3Fid-3D164703151-26iu-3D_4140_ostg.clktrk&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=2VFxc1z5VWicOkDfAMfNXItfIXQetwVEmkqlOB-1o34&e=
>>
>
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
S3tools-general mailing list
S3tools-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/s3tools-general
