Re: tokenize.untokenize adding line continuation characters

2017-01-17 Thread Rotwang
On Tuesday, January 17, 2017 at 11:11:27 AM UTC, Peter Otten wrote: > Rotwang wrote: > > > Here's something odd I've found with the tokenize module: tokenizing 'if > > x:\ny' and then untokenizing the result adds '\\\n' to the end. > > Attempting to tokenize the result again fails because of t

Re: tokenize.untokenize adding line continuation characters

2017-01-17 Thread Peter Otten
Rotwang wrote: > Here's something odd I've found with the tokenize module: tokenizing 'if > x:\ny' and then untokenizing the result adds '\\\n' to the end. > Attempting to tokenize the result again fails because of the backslash > continuation with nothing other than a newline after it. On the

Re: tokenize.untokenize adding line continuation characters

2017-01-17 Thread sg552
On Tuesday, January 17, 2017 at 2:47:03 AM UTC, Steven D'Aprano wrote: > On Tuesday 17 January 2017 09:42, Rotwang wrote: > > > Here's something odd I've found with the tokenize module: > [...] > > Copypasted from iPython: > > It's not impossible that iPython is doing something funny with the to

Re: tokenize.untokenize adding line continuation characters

2017-01-16 Thread Steven D'Aprano
On Tuesday 17 January 2017 09:42, Rotwang wrote: > Here's something odd I've found with the tokenize module: [...] > Copypasted from iPython: It's not impossible that iPython is doing something funny with the tokenize module. Before reporting it as a bug, I recommend that you confirm that it al

tokenize.untokenize adding line continuation characters

2017-01-16 Thread Rotwang
Here's something odd I've found with the tokenize module: tokenizing 'if x:\n y' and then untokenizing the result adds '\\\n' to the end. Attempting to tokenize the result again fails because of the backslash continuation with nothing other than a newline after it. On the other hand, if the o