The tokenize.generate_tokens function seems to handle in a context-
sensitive manner the new line after a comment:
>>> from StringIO import StringIO
>>> from tokenize import generate_tokens
>>>
>>> text = '''
... # hello world
... x = (
... # hello world
... )
... '''
>>>
>>> for t in generate_tokens(StringIO(text).readline):
... print repr(t[1])
...
'\n'
'# hello world\n'
'x'
'='
'('
'\n'
'# hello world'
'\n'
')'
'\n'
''
Is there a reason that the newline is included in the first comment
but not in the second, or is it a bug ?
George
--
http://mail.python.org/mailman/listinfo/python-list
