Sven One thing which has not been looked at in this discussion is why Jimmie's first attempt to solve the problem failed. He replaced every comma in the query by its percent encoding, '%2C'. This failed because ZnClient replaced this by '%252C', i.e. it percent encoded the percent sign. In my first attempt to help him, I tried replacing the query line by: queryAt: 'fields' put: 'displayName,instrument,pip' urlEncoded; but this produced the same result. The puzzling thing is why ZnClient is doing this.
In http://www.w3.org/Addressing/URL/uri-spec.html it is stated that: " The percent sign ("%", ASCII 25 hex) is used as the escape character in the encoding scheme and is never allowed for anything else." Further on, there are two examples of strings which are illegal as URIs because they contain percent signs which are not followed by two decodable hex characters. Hence, if ZnClient encounters a percent sign in a query string, either it is the beginning of a percent encoded character or it is illegal. In neither case is it right to percent encode the percent sign. Presumably the correct action for ZnClient is to examine the two characters following the percent sign and see if together they can be decoded to a legal character. Of course, if this were strictly enforced, it would still be possible for some perverse user to smuggle a percent sign through as part of a literal string by percent encoding it first - but this would not involve any illegality. On the other hand, anyone in Jimmie's position would be able to pre-encode a comma to %2C and have it accepted, while still allowing commas to be used where the site is happy with them. I may have misunderstood something, but anyway that's my 2p worth. Peter Kenny -----Original Message----- From: Pharo-users [mailto:pharo-users-boun...@lists.pharo.org] On Behalf Of Sven Van Caekenberghe Sent: 11 June 2015 15:49 To: Any question about pharo is welcome Subject: Re: [Pharo-users] ZnClient and percent characters Here is one older discussion that let to Zn going away from the better safe than sorry approach to encoding: http://forum.world.st/Zinc-How-to-use-the-character-in-an-URL-td4716386.html#a4716459 You have to read it up to the end. > On 11 Jun 2015, at 16:19, Jimmie Houchin <jlhouc...@gmail.com> wrote: > > This is exactly why I expressly state I am not a language lawyer and > explicitly do not know what is and is not expressly forbidden or allowed with > regards to a comma. > > You are correct about the Wikipedia article. > > Is it every wrong or illegal to use a complete safely encoded request? Is it > just simply not required? > So not fully encoded is still valid and legal and also the fully encode is > also fully valid and legal. Yes, any server should accept all encodings. However, the following are different: http://foo.com/bar?x=1 http://foo.com/bar?x%3D1 In the second case, you take away the meaning of = by encoding it. So you just can't encode everything everywhere. > eg: > http://yourserver.com/path?options=eggs,toast,coffee > is fully valid and legal, but may encounter problems depending to whom the > request is made and their implementation? > Encoding is not required but is at the discretion of the server > implementation? > > http://yourserver.com/path?options=eggs%2Ctoast%2Ccoffee > is fully valid and legal and is always usable and will never encounter > problems? > All valid server side implementations will handle this properly? > > Since I am sure that the API that I am writing for is probably not the only > server implementation which requires the comma to be encoded. Actually, this is the first time I hear of a problem with , So, from that perspective, the server might be in error. > And regardless of legality of the use of comma it appears that some > implementers are on the "to be safe we encode everything" side of things. It > would be nice to have some option which allows us to encode all to be safe > option. Yes, maybe such an option would be good, but only if it is really needed. > Thanks for listening and your help. > > Jimmie > > > > > On 06/11/2015 01:35 AM, Sven Van Caekenberghe wrote: >> @everybody >> >> The key method that defines how the query part of a URL is percent >> encoded is ZnMetaResourceUtils class>>#querySafeSet >> >> Years ago, Zinc HTTP Components followed the better safe than sorry approach >> of encoding almost every character except for the ones that are safe in all >> contexts. >> >> Later on, we began reading the specs better and decided to follow them more >> closely, that is why there are now different safe sets. >> >> Now, we can (and should) all read the different specs, and try to learn from >> things in the wild as well from other implementations. >> >> The quote from http://en.wikipedia.org/wiki/Query_string was incomplete, it >> said 'for HTML 5 when submitting a form using GET', which is a very specific >> context. >> >> ZnUrl was written against RFC 3986 mostly. >> >> Now, maybe we made a mistake, maybe not. >> >> But maybe it also would be a good idea to allow users to decide this for >> themselves on a case by case basis. >> >>> On 11 Jun 2015, at 05:18, Jimmie Houchin <jlhouc...@gmail.com> wrote: >>> >>> Thanks for the reply. >>> >>> I implemented Peter's suggestion as an easy keep moving solution. >>> >>> As I said, I am not expert in what is or is not legal according to the >>> standards. >>> However, looking at Python, their urllib library in the quote and urlencode >>> methods they encode the commas by default. >>> >>> _ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' >>> b'abcdefghijklmnopqrstuvwxyz' >>> b'0123456789' >>> b'_.-') >>> >>> https://docs.python.org/3/library/urllib.parse.html >>> https://hg.python.org/cpython/file/3.4/Lib/urllib/parse.py >>> >>> That's at least how one major language understands the standard. And Python >>> 2.7 is the same. >>> >>> According to Wikipedia >>> http://en.wikipedia.org/wiki/Query_string >>> • Characters that cannot be converted to the correct charset are >>> replaced with HTML numeric character references[9] • SPACE is encoded as '+' >>> • Letters (A–Z and a–z), numbers (0–9) and the characters >>> '*','-','.' and '_' are left as-is >>> >>> It appeared in the stackoverflow article I quoted previously that ASP.NET >>> encodes commas. I could misunderstand or be reading into it. >>> http://stackoverflow.com/questions/8828702/why-is-the-comma-url-enco >>> ded Just a little more information to add to the discussion. >>> >>> Thanks. >>> >>> Jimmie >>> >>> >>> >>> >>> On 06/10/2015 05:56 PM, Norbert Hartl wrote: >>>> Just to clarify: >>>> >>>> " >>>> Characters in the "reserved" set are not reserved in >>>> all contexts. >>>> >>>> The set of characters actually reserved within any given URI >>>> component is defined by that component. In general, a character is >>>> reserved if the semantics of the URI changes if the character is >>>> replaced with its escaped US-ASCII encoding." >>>> >>>> If I were you I'd subclass ZnUrl and implement >>>> #encodeQuery:on: >>>> on that class. You could have an extension method in ZnResourceMetaUtils >>>> that returns the character set you need to have encoded. In ZnClient you >>>> just set your ZnUrl derived class object as #url: >>>> Cannot think of anything better for a quick resolve of your problem. >>>> Norbert >>>>> Am 11.06.2015 um 00:26 schrieb Jimmie Houchin <jlhouc...@gmail.com>: >>>>> >>>>> I am not an expert on URIs or encoding. However, this is a requirement of >>>>> the API I am using and I am required to submit an encoded URI with %2C >>>>> and no commas. >>>>> >>>>> As far as commas needing to be escaped, it seems from other sources that >>>>> they should be. >>>>> >>>>> From https://www.ietf.org/rfc/rfc2396.txt >>>>> The plus "+", dollar "$", and comma "," characters have been added to >>>>> those in the "reserved" set, since they are treated as reserved >>>>> within the query component. >>>>> >>>>> States that commas are reserved within the query component. >>>>> >>>>> >>>>> http://stackoverflow.com/questions/8828702/why-is-the-comma-url-en >>>>> coded >>>>> >>>>> >>>>> Regardless of what is or is not required, I do need the ability to have a >>>>> query string with commas encoded as %2C in order to satisfy and use the >>>>> API which states. >>>>> >>>>> fields: Optional An URL encoded (%2C) comma separated list of instrument >>>>> fields that are to be returned in the response. The instrument field will >>>>> be returned regardless of the input to this query parameter. Please see >>>>> the Response Parameters section below for a list of valid values. >>>>> >>>>> Which will look like this or something similar. >>>>> >>>>> fields=displayName%2Cinstrument%2Cpip >>>>> >>>>> >>>>> Thanks. >>>>> >>>>> Jimmie >>>>> >>>>> >>>>> On 06/10/2015 03:27 PM, Norbert Hartl wrote: >>>>>> That's because the comma does not need to be escaped in the query part >>>>>> of the uri. >>>>>> >>>>>> Norbert >>>>>> >>>>>> >>>>>>> Am 10.06.2015 um 22:00 schrieb Jimmie Houchin >>>>>>> <jlhouc...@gmail.com> >>>>>>> : >>>>>>> >>>>>>> On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote: >>>>>>> >>>>>>>>> On 10 Jun 2015, at 17:24, David <stormb...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> El Wed, 10 Jun 2015 10:14:37 -0500 Jimmie Houchin >>>>>>>>> <jlhouc...@gmail.com> >>>>>>>>> >>>>>>>>> escribió: >>>>>>>>> >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> I am attempting to use ZnClient to request data. The request >>>>>>>>>> requires a %2C (comma) delimited string as part of the query. >>>>>>>>>> Below is a snippet. >>>>>>>>>> >>>>>>>>>> znClient >>>>>>>>>> addPath: '/v1/instruments'; >>>>>>>>>> queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip'; >>>>>>>>>> get ; >>>>>>>>>> contents) >>>>>>>>>> >>>>>>>>>> The string 'displayName%2Cinstrument%2Cpip' >>>>>>>>>> is being converted to 'displayName%252Cinstrument%252Cpip' >>>>>>>>>> which causes the request to fail. >>>>>>>>>> >>>>>>>>>> The query needs to be >>>>>>>>>> fields=displayName%2Cinstrument%2Cpip >>>>>>>>>> >>>>>>>>>> I have not found how to do this correctly. >>>>>>>>>> Any help greatly appreciated. >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> Jimmie >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> Maybe a silly thing, but since %2C = , ... Did you tried >>>>>>>>> already to make itself encode that? Like znClient >>>>>>>>> addPath: '/v1/instruments'; >>>>>>>>> queryAt: 'fields' putAll: 'displayName,instrument,pip'; >>>>>>>>> get ; >>>>>>>>> contents) >>>>>>>>> >>>>>>>>> I suspect it is using encoding internally, that is why % is >>>>>>>>> also encoded if you try to put it. >>>>>>>>> >>>>>>>>> I hope that works >>>>>>>>> >>>>>>>> Not silly and no need to suspect, but absolutely correct ! >>>>>>>> >>>>>>>> Sven >>>>>>>> >>>>>>> My apologies for not having full disclosure. >>>>>>> >>>>>>> Pharo 4, new image, freshly installed Zinc stable version. >>>>>>> Xubuntu 15.04 >>>>>>> >>>>>>> >>>>>>> >>>>>>> That is what I thought would happen and what I tried first. But it is >>>>>>> not being encoded from what I can find. >>>>>>> >>>>>>> Inspect this in a workspace/playground. >>>>>>> >>>>>>> ZnClient new >>>>>>> https; >>>>>>> host: ' >>>>>>> google.com >>>>>>> '; >>>>>>> addPath: '/commaTest'; >>>>>>> queryAt: 'fields' put: 'displayName,instrument,pip'; >>>>>>> yourself >>>>>>> >>>>>>> View the request / requestLine / uri. The commas are still present in >>>>>>> the URI. >>>>>>> So I tried encoding myself and get the other error. >>>>>>> >>>>>>> Of course Google won't understand this and in this snippet won't >>>>>>> receive it. >>>>>>> >>>>>>> And please let me know if I am doing something wrong. >>>>>>> >>>>>>> Any help greatly appreciated. >>>>>>> >>>>>>> Jimmie >>>>>>> >>>>>>> >>>>>>> >>>>>>> >> > >
