Re: Re - bug in tokenize module

2016-06-20 Thread Steven D'Aprano
On Tue, 21 Jun 2016 09:01 am, Harrison Chudleigh wrote: > Sorry. Only works with text files. But my point is still valid. What point? Without context, how are we supposed to know what you're talking about? We're not mind-readers you know. -- Steven -- https://mail.python.org/mailman/listin

Re - bug in tokenize module

2016-06-20 Thread Harrison Chudleigh
Sorry. Only works with text files. But my point is still valid. *** This message is intended for the addressee named and may contain privileged information or confidential information or both. If you are not the intended

Re: tokenize module

2009-06-30 Thread Tim Roberts
bootkey wrote: > >Thanks for the reply. I wonder why the tokenizer classifies all >operators simply as OP, instead of the various operators listed in the >tok_name dictionary. I imagine it's just an exercise left to the reader. It's not that hard of an extension. -- Tim Roberts, t...@probo.com

Re: tokenize module

2009-06-28 Thread bootkey
On Jun 28, 1:46 am, Tim Roberts wrote: > Jim wrote: > >I'm trying to understand the output of the tokenize.generate_tokens() > >generator.  The token types returned seem to be more general than I'd > >expect.  For example, when fed the following line of code: > > >def func_a(): > >... > >It seems

Re: tokenize module

2009-06-28 Thread Tim Roberts
Jim wrote: >I'm trying to understand the output of the tokenize.generate_tokens() >generator. The token types returned seem to be more general than I'd >expect. For example, when fed the following line of code: > >def func_a(): >... >It seems to me that the token '(' should be identified as 'LP

tokenize module

2009-06-27 Thread Jim
I'm trying to understand the output of the tokenize.generate_tokens() generator. The token types returned seem to be more general than I'd expect. For example, when fed the following line of code: def func_a(): the (abbreviated) returned token tuples are as follows: (NAME,def,

Re: tokenize module after installation

2007-08-09 Thread Martin v. Löwis
> After installation my program that uses tokenize module,when I run > myprogram.exe (vgsveki.exe): > > AttributeError: 'module' object has no attribute 'untokenize' Perhaps you use the 2.4 version of tokenize.py here, which did not have untokenize. Regards, M

tokenize module after installation

2007-08-08 Thread vedrandekovic
Hi, I have one more question about installation. After installation my program that uses tokenize module,when I run myprogram.exe (vgsveki.exe): Traceback (most recent call last): File "vgsveki.py", line 307, in kompajlati File "vgsveki.py", line 302, in kompajlati_proce

Re: Problem with tokenize module and indents

2006-08-24 Thread Fredrik Lundh
Tim wrote: > I ran into a problem with a script i was playing with to check code > indents and need some direction. It seems to depend on if tabsize is > set to 4 in editor and spaces and tabs indents are mixed on consecutive > lines. Works fine when editors tabsize was 8 regardless if indent

Re: Problem with tokenize module and indents

2006-08-24 Thread Tim Scheidemantle
Tim wrote: > I ran into a problem with a script i was playing with to check code > indents and need some direction. It seems to depend on if tabsize is > set to 4 in editor and spaces and tabs indents are mixed on consecutive > lines. Works fine when editors tabsize was 8 regardless if inden

Re: Problem with tokenize module and indents

2006-08-23 Thread Simon Forman
Tim wrote: > I ran into a problem with a script i was playing with to check code > indents and need some direction. It seems to depend on if tabsize is > set to 4 in editor and spaces and tabs indents are mixed on consecutive > lines. Works fine when editors tabsize was 8 regardless if indents ar

Problem with tokenize module and indents

2006-08-23 Thread Tim
I ran into a problem with a script i was playing with to check code indents and need some direction. It seems to depend on if tabsize is set to 4 in editor and spaces and tabs indents are mixed on consecutive lines. Works fine when editors tabsize was 8 regardless if indents are mixed. Below

Re: Howto or Tutorial for tokenize module for a newbie?

2006-07-11 Thread Fuzzyman
TY wrote: > Hi, > > Can someone point me to a Howto or Tutorial for tokenize module for a > newbie? I read the documentation but it doesn't have a lot of info... Here's another useful example : http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52298 All th

Re: Howto or Tutorial for tokenize module for a newbie?

2006-07-10 Thread John J. Lee
"TY" <[EMAIL PROTECTED]> writes: > Can someone point me to a Howto or Tutorial for tokenize module for a > newbie? I read the documentation but it doesn't have a lot of info... Here's a simple example: http://groups.google.com/group/comp.lang.python/msg/

Re: Howto or Tutorial for tokenize module for a newbie?

2006-07-10 Thread sreekant
TY wrote: > Hi, > > Can someone point me to a Howto or Tutorial for tokenize module for a > newbie? I read the documentation but it doesn't have a lot of info... > Thanks! > Hi there I don't know if I got your requirement. But I used a own version of tokenizer

Howto or Tutorial for tokenize module for a newbie?

2006-07-10 Thread TY
Hi, Can someone point me to a Howto or Tutorial for tokenize module for a newbie? I read the documentation but it doesn't have a lot of info... Thanks! -- http://mail.python.org/mailman/listinfo/python-list