In boto they use the quote_plus function like that:
https://github.com/boto/boto/blob/develop/boto/auth.py
urllib.parse.quote(param, safe='-_.~')
But python quote is supposed to never quote "-_." so I think that it could
be for us:
urllib.quote(param, safe='~')
and so, when using quote_plus and wanting to also preserve "/" :
urllib.quote(param, safe='~/')
--
Florent Viard
Software Development Engineer
...............................................
florent.vi...@seagate.com
www.seagate.com
On Fri, Dec 19, 2014 at 4:23 AM, Matt Domsch <m...@domsch.com> wrote:
>
> The ~ character in a filename is certainly being encoded as %7E, which is
> incorrect - it's not supposed to be encoded at all. This is causing a v4
> signature validation failure.
>
> On Wed, Dec 17, 2014 at 12:57 PM, Florent Viard <florent.vi...@seagate.com
> > wrote:
>>
>> Hi Matt and all,
>>
>> Thanks for the work with the signature v4.
>> I have a question about that work but I don't really know where to
>> discuss of that, so send it here.
>>
>> Before the merge, I was thinking about replacing the "limited string
>> encoding code" inside "urlencode_string()" (S3/S3.py), by urlquote_plus(),
>> like what is done in signature_v4. As that looks to be the way right way
>> that should be encoded. (Also it is done like that in boto).
>>
>> For info, urlencode_string() is used in "create_request()" to encode the
>> object uri string that is also used for generating the signature.
>>
>> Currently I see that all the "urlquote" things are done in the sign_v4
>> code.
>> First, I'm wondering if there is not double quoting for the resource uri
>> when using signature v4. (That, sadly I have not tested).
>> Also, I'm wondering whether it wouldn't be better to have most of the
>> quoting that are done inside sign_v4 functions moved earlier in the
>> function stack, as the quoting that is done in the signature code, should
>> certainly be also done for the http request parameters.
>>
>> What do you think?
>>
>> Thank you,
>>
>>
>> --
>> Florent Viard
>>
>>
>> On Wed, Dec 17, 2014 at 5:52 PM, Matt Domsch <m...@domsch.com> wrote:
>>
>>> With huge thanks to Vasileios Mitrousis (@vamitrou) and Michal Ludvig,
>>> s3cmd now supports AWS Signature v4 signing method, which is required by
>>> the eu-central-1 (Frankfurt) region, and is supported in all AWS regions.
>>> s3cmd defaults to using the Signature v4 method now, though will fall back
>>> to older v2 method in some cases.
>>>
>>> If you are using s3cmd against a non-AWS-S3 product, such as Dreamhost
>>> DreamObjects, Eucalyptus, Apache CloudStack, or the like, you may need to
>>> use the new command line option --signature-v2 to force it to fall back,
>>> until these systems (and your specific instances thereof) add support for
>>> signature v4.
>>>
>>>
>>> Also, huge thanks to Florent Viard (@fviard) for his persistence in
>>> reviewing and aggregating many outstanding pull requests, which are now
>>> merged.
>>>
>>>
>>> New options include:
>>> --signature-v2 force use of signature v2 method
>>> --ssl force use of https
>>> --no-ssl disable use of https
>>> --region synonym for --bucket-location (to match aws-cli
>>> behavior)
>>> --no-reduced-redundancy force standard storage policy instead of
>>> reduced redundancy
>>> --remove-header allow headers to be removed as well as added
>>> --ca-certs=FILE allow use of a sever self-signed certificate for
>>> https, where you know the matching CA-CERT
>>> --check-certificate (default) expect server certificate to be valid,
>>> including hostname
>>> --no-check-certificate disable hostname checking on an SSL
>>> certificate, needed when a hostname doesn't match its wildcard certificate.
>>>
>>>
>>> There have been quite a few fixes since the 1.5.0-rc1 release. I'm not
>>> quite ready to do a -rc2, but I do want to encourage people to get the
>>> github.com/s3tools/s3cmd
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__github.com_s3tools_s3cmd&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=zq-uMtjYpZFDpO-8D169_MAmk6B6ofuCvheGxJU8FGo&e=>
>>> master branch, give it a try, and report any outstanding issues.
>>>
>>> Thanks,
>>> Matt
>>>
>>>
>>> Shortlog:
>>>
>>> Damian Gerow (2):
>>> Support proxying SSL connections
>>> Only support proxied SSL with Python >= 2.7
>>>
>>> Daniel Harris (1):
>>> Fix potential divide by zero errors
>>>
>>> Eric Mill (1):
>>> rewrite of README into markdown
>>>
>>> Florent Viard (9):
>>> Minor changes in README to fix points 1 and 2 in my comment at
>>> https://github.com/s3tools/s3cmd/pull/369
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_s3tools_s3cmd_pull_369&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=No0iijJToKpV7k2anQ90FpprB5volMHDxIbJIEQBXFg&e=>
>>> .
>>> Cosmetic: removed one empty space in excess of end of run summary.
>>> Fix MD5 comparison with s3 compatible servers that are not Amazon
>>> S3.
>>> Small cleanup of code, using "in" instead of "find" for strings
>>> when possible.
>>> Fix an error in my previous commit:
>>> b5b61abc8eb159f5932b936c7ca47e618615f6a3. (.find() forgotten to be removed)
>>> Improve the exit code of s3cmd in the case of an error from S3 to
>>> give a clearer status of the reason of the failure. Some existing exit
>>> codes were modified when they were not already in use.
>>> Disable dns support for buckets when "host_bucket" hostname
>>> doesn't have a %(bucket)s item in it.
>>> Backported changes from the following pull request:
>>> https://github.com/s3tools/s3cmd/pull/369
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_s3tools_s3cmd_pull_369&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=No0iijJToKpV7k2anQ90FpprB5volMHDxIbJIEQBXFg&e=>
>>> regarding aws pricing.
>>> Updated README.md with latest changes of S3 prices:
>>> https://aws.amazon.com/fr/blogs/aws/aws-data-transfer-price-reduction/?sc_ichannel=EM&sc_icountry=global&sc_icampaigntype=launch&sc_icampaign=EM_128506540&sc_idetail=em_1778261631&ref_=pe_411040_128506540_12
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__aws.amazon.com_fr_blogs_aws_aws-2Ddata-2Dtransfer-2Dprice-2Dreduction_-3Fsc-5Fichannel-3DEM-26sc-5Ficountry-3Dglobal-26sc-5Ficampaigntype-3Dlaunch-26sc-5Ficampaign-3DEM-5F128506540-26sc-5Fidetail-3Dem-5F1778261631-26ref-5F-3Dpe-5F411040-5F128506540-5F12&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=3omO2nUE4XDWzrHcIrQRzJFchDDFExZoY9tRArLW-x4&e=>
>>>
>>> Gianni Chiappetta (1):
>>> Always attempt to set the content-type
>>>
>>> Kiana McNellis (2):
>>> Fixed bug where the error report will not show if there was an
>>> import error due to dependency on unicodise which was not successfully
>>> imported
>>> Changed the exit code for import errors to not rely on dependency
>>> of EX_GENERAL being defined when not imported
>>>
>>> Lance Batson (1):
>>> fix issue with error display after config modify
>>>
>>> Matt Domsch (54):
>>> Merge pull request #361 from staer/patch-1
>>> Merge pull request #360 from intothev01d/config-ws-strip
>>> Revert "fix issue with error display after config modify"
>>> compare_filelists size check needs both objects to have a size
>>> value
>>> Check that HTTP 400 response includes data
>>> debug() the signature-v4 headers
>>> refactor content_type() and add object_replace(), tests
>>> Merge pull request #386 from hrchu/bug/cannotMoveLargeFile
>>> Add --ca-cert=FILE option for python 2.7.9ish
>>> Have CloudFront use ConnMan class
>>> create default context, then load in cert chain if needed
>>> handle Debian/Fedora differences in new ssl libraries
>>> Merge pull request #430 from fviard/pull_request_batch2
>>> Use ca_certs_file="" so it will load from ~/.s3cfg
>>> read/modify/write on modify command the headers and ACL
>>> clarify sanitize_headers() list
>>> add --remove-headers option, used in [modify]
>>> bugfixes in content-type fixes
>>> tests: modify command doesn't need --acl-public anymore
>>> [modify] only change mime-type if it's passed on the command line
>>> tests: [modify] add and remove cache-control header
>>> tests: s/curl/wget/
>>> [modify] compare headers in lowercase
>>> Delete the lower-case version of the header
>>> Merge pull request #435 from gandikun/master
>>> Merge pull request #428 from mdomsch/bug/content-type
>>> set_acl(): return 501 Not Implemented for DreamHost DreamObjects.
>>> add missing import in S3/S3Uri.py
>>> run-tests: rename (NoSuchKey) exit code change
>>> add to list of regions in --help --bucket-location
>>> add --region alias for --bucket-location
>>> Merge pull request #436 from vszakats/patch-1
>>> Arch Linux 2014.12.01 and Fedora rawhide (will be F22) now
>>> includes Python 2.7.9 which turns on SSL certificate validation. This
>>> leads to to a problem: The Amazon S3 wildcard certificate specifies
>>> *.s3.amazonaws.com
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__s3.amazonaws.com&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=uzbDxoUaYUVWCL3wJ2yk_Ew-YJxBR8pebtoQZcSb9t0&e=>
>>> which does not match any DNS-style buckets. So you have to use
>>> --no-check-certificate for operations on these buckets.
>>> finish merge with master
>>> Don't call set_tunnel() for http (not https) connections
>>> add debug messages for http/https proxied/non-proxied connections
>>> Merge pull request #431 from mdomsch/bug/426-specify-SSL-CA-file
>>> fix --{no-}check-certificate and non-proxied https connections
>>> Automatically disable ssl certificate hostname checking against
>>> AWS S3
>>> Merge pull request #438 from mdomsch/bug/check-certificate
>>> explicitly use python2 as shell
>>> finish merge of signature-v4 into master
>>> finish merge cleanup
>>> CloudFront: fix connection put() using ConnMan
>>> lowercase all headers added with --add-header
>>> use lower() in the debug message too
>>> Warn if requests module is not present
>>> Merge branch 'bug/lowercase-headers' into upstream-master
>>> remove unused import requests
>>> remove extraneous import ExitCodes
>>> Update s3cmd --help with changes made to manpage.
>>> Update spec instructions
>>> Add --signature-v2 and signature_v2 config option
>>> Fix multipart uploads with v4 signature code
>>>
>>> Michal Ludvig (6):
>>> Merge branch 'master' of git://github.com/s3tools/s3cmd
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__github.com_s3tools_s3cmd&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=zq-uMtjYpZFDpO-8D169_MAmk6B6ofuCvheGxJU8FGo&e=>
>>> Merge pull request #4 from s3tools/master
>>> S3Request - Include 'body' in 'request' in preparation for AWS4
>>> Signature
>>> AWS4 - Moved signing methods to Crypto.py
>>> Merge ssh://github.com/s3tools/s3cmd
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__github.com_s3tools_s3cmd&d=AwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=zq-uMtjYpZFDpO-8D169_MAmk6B6ofuCvheGxJU8FGo&e=>
>>> Merge AWS4 Crypto Work in Progress
>>>
>>> Mikhail Gusarov (1):
>>> Specify default checks performed on files being synced
>>>
>>> PanManAms (1):
>>> Changed prices
>>>
>>> Shunichi Shinohara (1):
>>> Specify content-type for request with XML body
>>>
>>> Sugandi (1):
>>> Fix README wrong filename
>>>
>>> Vasileios Mitrousis (5):
>>> signature v4 signing
>>> added comment in signature selection
>>> added temporary bucket to region mapping
>>> handle status 400 - location, when no data are returned by API
>>> fix in params urlencode
>>>
>>> Viktor Szakáts (1):
>>> added options -s/--ssl and --no-ssl options
>>>
>>> hrchu (1):
>>> Fix removeNameSpace parsing problem
>>>
>>> sralmai (1):
>>> Unbreak Config on empty options
>>>
>>> vamitrou (2):
>>> fix in bad region responses
>>> fixed put file and multipart upload
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>>> Get technology previously reserved for billion-dollar corporations, FREE
>>>
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__pubads.g.doubleclick.net_gampad_clk-3Fid-3D164703151-26iu-3D_4140_ostg.clktrk&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=GEhQqSrCDlzPsOu9ww_S8dL0RpfPwWzg7DpciZD7d7Y&m=Qgq9-lB0ySjYALG2AJQDX_U5zMb7D31xh1_0hRqakQk&s=2VFxc1z5VWicOkDfAMfNXItfIXQetwVEmkqlOB-1o34&e=
>>>
>>
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
S3tools-general mailing list
S3tools-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/s3tools-general
