Sven As I read the code, there are two methods in ZnMetaResourceUtils class which specify percent encoding of queries, #queryKeyValueSafeSet and #querySafeSet. The first is used in encoding the (key, value) pairs, the second in encoding the fragment after the #. It is not clear whether we need separate safe sets for the two cases, but in any case the first one was obviously the one to change to fix Jimmie's original problem.
Having been accused of monkeying with your code in an unreliable way, I shall keep my head down in future. It's a pity - I find it quite enjoyable trying to help people, and trying to prove that my elderly brain is not completely addled. Peter Kenny -----Original Message----- From: Pharo-users [mailto:pharo-users-boun...@lists.pharo.org] On Behalf Of Sven Van Caekenberghe Sent: 11 June 2015 07:35 To: Any question about pharo is welcome Subject: Re: [Pharo-users] ZnClient and percent characters @everybody The key method that defines how the query part of a URL is percent encoded is ZnMetaResourceUtils class>>#querySafeSet Years ago, Zinc HTTP Components followed the better safe than sorry approach of encoding almost every character except for the ones that are safe in all contexts. Later on, we began reading the specs better and decided to follow them more closely, that is why there are now different safe sets. Now, we can (and should) all read the different specs, and try to learn from things in the wild as well from other implementations. The quote from http://en.wikipedia.org/wiki/Query_string was incomplete, it said 'for HTML 5 when submitting a form using GET', which is a very specific context. ZnUrl was written against RFC 3986 mostly. Now, maybe we made a mistake, maybe not. But maybe it also would be a good idea to allow users to decide this for themselves on a case by case basis. > On 11 Jun 2015, at 05:18, Jimmie Houchin <jlhouc...@gmail.com> wrote: > > Thanks for the reply. > > I implemented Peter's suggestion as an easy keep moving solution. > > As I said, I am not expert in what is or is not legal according to the > standards. > However, looking at Python, their urllib library in the quote and urlencode > methods they encode the commas by default. > > _ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' > b'abcdefghijklmnopqrstuvwxyz' > b'0123456789' > b'_.-') > > https://docs.python.org/3/library/urllib.parse.html > https://hg.python.org/cpython/file/3.4/Lib/urllib/parse.py > > That's at least how one major language understands the standard. And Python > 2.7 is the same. > > According to Wikipedia > http://en.wikipedia.org/wiki/Query_string > • Characters that cannot be converted to the correct charset are > replaced with HTML numeric character references[9] • SPACE is encoded as '+' > • Letters (A–Z and a–z), numbers (0–9) and the characters '*','-','.' > and '_' are left as-is > > It appeared in the stackoverflow article I quoted previously that ASP.NET > encodes commas. I could misunderstand or be reading into it. > http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encode > d Just a little more information to add to the discussion. > > Thanks. > > Jimmie > > > > > On 06/10/2015 05:56 PM, Norbert Hartl wrote: >> Just to clarify: >> >> " >> Characters in the "reserved" set are not reserved in >> all contexts. >> >> The set of characters actually reserved within any given URI >> component is defined by that component. In general, a character is >> reserved if the semantics of the URI changes if the character is >> replaced with its escaped US-ASCII encoding." >> >> If I were you I'd subclass ZnUrl and implement >> #encodeQuery:on: >> on that class. You could have an extension method in ZnResourceMetaUtils >> that returns the character set you need to have encoded. In ZnClient you >> just set your ZnUrl derived class object as #url: >> Cannot think of anything better for a quick resolve of your problem. >> Norbert >>> Am 11.06.2015 um 00:26 schrieb Jimmie Houchin <jlhouc...@gmail.com>: >>> >>> I am not an expert on URIs or encoding. However, this is a requirement of >>> the API I am using and I am required to submit an encoded URI with %2C and >>> no commas. >>> >>> As far as commas needing to be escaped, it seems from other sources that >>> they should be. >>> >>> From https://www.ietf.org/rfc/rfc2396.txt >>> The plus "+", dollar "$", and comma "," characters have been added to >>> those in the "reserved" set, since they are treated as reserved >>> within the query component. >>> >>> States that commas are reserved within the query component. >>> >>> >>> http://stackoverflow.com/questions/8828702/why-is-the-comma-url-enco >>> ded >>> >>> >>> Regardless of what is or is not required, I do need the ability to have a >>> query string with commas encoded as %2C in order to satisfy and use the API >>> which states. >>> >>> fields: Optional An URL encoded (%2C) comma separated list of instrument >>> fields that are to be returned in the response. The instrument field will >>> be returned regardless of the input to this query parameter. Please see the >>> Response Parameters section below for a list of valid values. >>> >>> Which will look like this or something similar. >>> >>> fields=displayName%2Cinstrument%2Cpip >>> >>> >>> Thanks. >>> >>> Jimmie >>> >>> >>> On 06/10/2015 03:27 PM, Norbert Hartl wrote: >>>> That's because the comma does not need to be escaped in the query part of >>>> the uri. >>>> >>>> Norbert >>>> >>>> >>>>> Am 10.06.2015 um 22:00 schrieb Jimmie Houchin >>>>> <jlhouc...@gmail.com> >>>>> : >>>>> >>>>> On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote: >>>>> >>>>>>> On 10 Jun 2015, at 17:24, David <stormb...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>> El Wed, 10 Jun 2015 10:14:37 -0500 Jimmie Houchin >>>>>>> <jlhouc...@gmail.com> >>>>>>> >>>>>>> escribió: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> I am attempting to use ZnClient to request data. The request >>>>>>>> requires a %2C (comma) delimited string as part of the query. >>>>>>>> Below is a snippet. >>>>>>>> >>>>>>>> znClient >>>>>>>> addPath: '/v1/instruments'; >>>>>>>> queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip'; >>>>>>>> get ; >>>>>>>> contents) >>>>>>>> >>>>>>>> The string 'displayName%2Cinstrument%2Cpip' >>>>>>>> is being converted to 'displayName%252Cinstrument%252Cpip' >>>>>>>> which causes the request to fail. >>>>>>>> >>>>>>>> The query needs to be >>>>>>>> fields=displayName%2Cinstrument%2Cpip >>>>>>>> >>>>>>>> I have not found how to do this correctly. >>>>>>>> Any help greatly appreciated. >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> Jimmie >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> Maybe a silly thing, but since %2C = , ... Did you tried already >>>>>>> to make itself encode that? Like znClient >>>>>>> addPath: '/v1/instruments'; >>>>>>> queryAt: 'fields' putAll: 'displayName,instrument,pip'; >>>>>>> get ; >>>>>>> contents) >>>>>>> >>>>>>> I suspect it is using encoding internally, that is why % is also >>>>>>> encoded if you try to put it. >>>>>>> >>>>>>> I hope that works >>>>>>> >>>>>> Not silly and no need to suspect, but absolutely correct ! >>>>>> >>>>>> Sven >>>>>> >>>>> My apologies for not having full disclosure. >>>>> >>>>> Pharo 4, new image, freshly installed Zinc stable version. >>>>> Xubuntu 15.04 >>>>> >>>>> >>>>> >>>>> That is what I thought would happen and what I tried first. But it is not >>>>> being encoded from what I can find. >>>>> >>>>> Inspect this in a workspace/playground. >>>>> >>>>> ZnClient new >>>>> https; >>>>> host: ' >>>>> google.com >>>>> '; >>>>> addPath: '/commaTest'; >>>>> queryAt: 'fields' put: 'displayName,instrument,pip'; >>>>> yourself >>>>> >>>>> View the request / requestLine / uri. The commas are still present in >>>>> the URI. >>>>> So I tried encoding myself and get the other error. >>>>> >>>>> Of course Google won't understand this and in this snippet won't receive >>>>> it. >>>>> >>>>> And please let me know if I am doing something wrong. >>>>> >>>>> Any help greatly appreciated. >>>>> >>>>> Jimmie >>>>> >>>>> >>>>> >>>>> >>> >> >
