[issue36654] Add example to tokenize.tokenize

Berker Peksag added the comment: Wow, I managed to make typos in all three commits! PR 12947 has some discussion about why adding these examples would be a good idea as we now have two different APIs for unicode and bytes input. Thanks for the PR, Windson. -- resolution: -> fixed s

[issue36654] Add example to tokenize.tokenize

Berker Peksag added the comment: New changeset 6dbd843dedc9e05c0e3f4714294837f0a83deebe by Berker Peksag (Miss Islington (bot)) in branch '3.7': bpo-36654: Add examples for using tokenize module programmatically (GH-12947) https://github.com/python/cpython/commit/6dbd843dedc9e05c0e3f471429483

[issue36654] Add example to tokenize.tokenize

Berker Peksag added the comment: New changeset 1cf0df4f1bcc38dfd70a152af20cf584de531ea7 by Berker Peksag (Miss Islington (bot)) in branch '3.8': bpo-36654: Add examples for using tokenize module programmatically (GH-18187) https://github.com/python/cpython/commit/1cf0df4f1bcc38dfd70a152af20cf

[issue36654] Add example to tokenize.tokenize

Change by miss-islington : -- pull_requests: +17570 pull_request: https://github.com/python/cpython/pull/18187 ___ Python tracker ___ __

[issue36654] Add example to tokenize.tokenize

Change by miss-islington : -- pull_requests: +17571 pull_request: https://github.com/python/cpython/pull/18188 ___ Python tracker ___ __

[issue36654] Add example to tokenize.tokenize

Berker Peksag added the comment: New changeset 4b09dc79f4d08d85f2cc945563e9c8ef1e531d7b by Berker Peksag (Windson yang) in branch 'master': bpo-36654: Add examples for using tokenize module programmically (#12947) https://github.com/python/cpython/commit/4b09dc79f4d08d85f2cc945563e9c8ef1e531d

[issue36654] Add example to tokenize.tokenize

Serhiy Storchaka added the comment: I do not think a new example is needed. The existing example already demonstrates the use of file's readline method. If you need an example for opening a file, the tokenize module documentation is not an appropriate place for this. -- nosy: +serhi

[issue36654] Add example to tokenize.tokenize

Change by Windson Yang : -- keywords: +patch pull_requests: +12871 stage: -> patch review ___ Python tracker ___ ___ Python-bugs-li

[issue36654] Add example to tokenize.tokenize

Windson Yang added the comment: Yes, I can make a PR for it. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscrib

[issue36654] Add example to tokenize.tokenize

Lisa Roach added the comment: This could be added to the examples section of the tokenize doc, would you want to make the PR Windson? -- nosy: +lisroach ___ Python tracker __

[issue36654] Add example to tokenize.tokenize

New submission from Windson Yang : > The tokenize() generator requires one argument, readline, which must be a > callable object which provides the same interface as the io.IOBase.readline() > method of file objects. Each call to the function should return one line of > input as bytes. Add a