[issue2134] Add new attribute to TokenInfo to report specific token IDs

2012-01-18 Thread Meador Inge
Meador Inge added the comment: Fixed. Thanks for the reviews everyone. -- resolution: -> fixed stage: patch review -> committed/rejected status: open -> closed ___ Python tracker _

[issue2134] Add new attribute to TokenInfo to report specific token IDs

2012-01-18 Thread Roundup Robot
Roundup Robot added the comment: New changeset 75baef657770 by Meador Inge in branch '2.7': Issue #2134: Clarify token.OP handling rationale in tokenize documentation. http://hg.python.org/cpython/rev/75baef657770 New changeset dfd74d752b0e by Meador Inge in branch '3.2': Issue #2134: Clarify t

[issue2134] Add new attribute to TokenInfo to report specific token IDs

2012-01-15 Thread Ezio Melotti
Changes by Ezio Melotti : -- stage: needs patch -> patch review ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue2134] Add new attribute to TokenInfo to report specific token IDs

2011-12-24 Thread Meador Inge
Changes by Meador Inge : Added file: http://bugs.python.org/file24089/tokenize-docs-2.7-3.2.patch ___ Python tracker ___ ___ Python-bugs-list m

[issue2134] Add new attribute to TokenInfo to report specific token IDs

2011-12-24 Thread Meador Inge
Meador Inge added the comment: > The cmdoption directive should be used with a program directive. Ah, nice. Thanks for the tip Éric. Updated patch attached along with a patch for the 2.7/3.2 doc update attached. -- Added file: http://bugs.python.org/file24088/tokenize-exact-type-v1.p

[issue2134] Add new attribute to TokenInfo to report specific token IDs

2011-12-21 Thread Éric Araujo
Éric Araujo added the comment: The cmdoption directive should be used with a program directive. See library/trace for an example of how to use it and to see the anchors and index entries it generates. -- nosy: +eric.araujo ___ Python tracker

[issue2134] Add new attribute to TokenInfo to report specific token IDs

2011-12-18 Thread Nick Coghlan
Nick Coghlan added the comment: Meador's patch looks good to me. The docs change for 2.7 and 3.2 would be similar, just with text like "Specific tokens can be distinguished by checking the ``string`` attribute of OP tokens for a match with the expected character sequence." replacing the refer

[issue2134] Add new attribute to TokenInfo to report specific token IDs

2011-12-18 Thread Meador Inge
Meador Inge added the comment: The proposed documentation text seems too complicated and language expert speaky to me. We should try to link to standard definitions when possible to reduce the text here. For example, I believe the "Operators" and "Delimiters" tokens in the "Lexical Analysis

[issue2134] Add new attribute to TokenInfo to report specific token IDs

2011-12-15 Thread Ezio Melotti
Changes by Ezio Melotti : -- nosy: +ezio.melotti ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyt

[issue2134] Add new attribute to TokenInfo to report specific token IDs

2011-12-15 Thread Terry J. Reedy
Terry J. Reedy added the comment: Both the proposed text and 3.3 addition look good to me. -- ___ Python tracker ___ ___ Python-bugs-l

[issue2134] Add new attribute to TokenInfo to report specific token IDs

2011-12-15 Thread Eric Snow
Changes by Eric Snow : -- nosy: +eric.snow ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.or

[issue2134] Add new attribute to TokenInfo to report specific token IDs

2011-12-15 Thread Meador Inge
Changes by Meador Inge : -- nosy: +meador.inge ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pytho

[issue2134] Add new attribute to TokenInfo to report specific token IDs

2011-12-14 Thread Nick Coghlan
Nick Coghlan added the comment: Ah, I didn't read it as suggested documentation at all - you moved seamlessly from personal commentary to a docs suggestion without separating the two, so it appeared to be a complete non sequitur to me. As for the docs suggestion, I think it works as the expla

[issue2134] Add new attribute to TokenInfo to report specific token IDs

2011-12-14 Thread Terry J. Reedy
Terry J. Reedy added the comment: If you are responding to me, I am baffled. I gave a concise way to document the current behavior with respect to .OP, which you said you wanted. -- ___ Python tracker

[issue2134] Add new attribute to TokenInfo to report specific token IDs

2011-12-14 Thread Nick Coghlan
Nick Coghlan added the comment: Sure, but what does that have to do with anything? tokenize isn't a general purpose tokenizer, it's specifically for tokenizing Python source code. The *problem* is that it doesn't currently fully tokenize everything, but doesn't explicitly say that in the modu

[issue2134] Add new attribute to TokenInfo to report specific token IDs

2011-12-14 Thread Terry J. Reedy
Terry J. Reedy added the comment: I believe that that list includes all symbols and symbol combinations that are syntactically significant in expressions. This is the generalized meaning of 'operator' that is being used. What do not appear are '#' which marks comments, '_' which is a name cha

[issue2134] Add new attribute to TokenInfo to report specific token IDs

2011-12-14 Thread Nick Coghlan
Nick Coghlan added the comment: There are a *lot* of characters with semantic significance that are reported by the tokenize module as generic "OP" tokens: token.LPAR token.RPAR token.LSQB token.RSQB token.COLON token.COMMA token.SEMI token.PLUS token.MINUS token.STAR token.SLASH token.VBAR to