[Python-Dev] Range information in the AST -- once more
Hello! I'm writing a static language analyzer for an IDE which reuses the CPython parser (for parsing) [1]. Two years ago, I asked about a few changes to be made to the AST provided by CPython, but the discussion thread dried up before a definite decision was made. I decided to just copy the parser code to my project and make the necessary changes there back then. I'm bringing this up again now since after my project has seen its first release recently, packagers are (understandably) a bit unhappy about the python fork residing in its repository. I would really like to get rid of that fork and link against the vanilla libpython instead. There's two things which are at the very least required to make this work: 1. The col_offset and lineno of an Attribute must give the beginning of the word that names the attribute, not the beginning of the expression. Example: In "foo.bar.baz", the col_offset of the Attribute belonging to "bar" says "0" currently, it would need to be "4". 2. Column offsets and line numbers would need to be available for function arguments (those with and without stars), and for alias nodes. In total, this requires very little change to the existing code, "a few tens of lines changed at most" order of magnitude; those are mostly trivial changes. For what I can tell, the impact on existing code using the AST stuff will be about zero. Even if there was some really obscure case where the change would matter, porting would only require about three lines of python code. Additionally, there's a few more things which would be useful to have available from the AST (namely the ranges of class and function names when they are defined -- currently only the start of the first decorator is available), but since those are reasonably easy to work around it's not that important. It would still be nice tough. If you think this is a reasonable suggestion then I'll be happy to provide a patch for more detailed discussion. Greetings, Sven [1] See https://projects.kde.org/kdev-python ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Range information in the AST -- once more
On Thu, Dec 27, 2012 at 11:03 PM, Sven Brauch wrote: > Hello! > > I'm writing a static language analyzer for an IDE which reuses the > CPython parser (for parsing) [1]. Two years ago, I asked about a few > changes to be made to the AST provided by CPython, but the discussion > thread dried up before a definite decision was made. I decided to just > copy the parser code to my project and make the necessary changes > there back then. I'm bringing this up again now since after my project > has seen its first release recently, packagers are (understandably) a > bit unhappy about the python fork residing in its repository. I would > really like to get rid of that fork and link against the vanilla > libpython instead. It certainly sounds like its worth considering for 3.4. It's a new feature, though, so it unfortunately wouldn't be possible to backport it to any earlier releases. Regards, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Range information in the AST -- once more
2012/12/27 Nick Coghlan : > It certainly sounds like its worth considering for 3.4. It's a new > feature, though, so it unfortunately wouldn't be possible to backport > it to any earlier releases. Yes, that is understandable. It wouldn't be much of a problem tough, my whole project is pretty bleeding-edge anyways, and depending on python >= 3.4 wouldn't hurt. For me it would only be important to have an acceptable solution for this long-term. Greetings, Sven ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] How old Python version should be supported in tests?
I found a code like "if sys.version_info < (2, 4):" in some tests. Should old versions (< 2.6) be supported in tests? Can such support code be removed (note that other tests likely doesn't compatible with old versions)? ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Range information in the AST -- once more
So just submit a patch to the tracker... --Guido On Thursday, December 27, 2012, Sven Brauch wrote: > 2012/12/27 Nick Coghlan >: > > It certainly sounds like its worth considering for 3.4. It's a new > > feature, though, so it unfortunately wouldn't be possible to backport > > it to any earlier releases. > > Yes, that is understandable. It wouldn't be much of a problem tough, > my whole project is pretty bleeding-edge anyways, and depending on > python >= 3.4 wouldn't hurt. For me it would only be important to have > an acceptable solution for this long-term. > > Greetings, > Sven > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How old Python version should be supported in tests?
2012/12/27 Serhiy Storchaka : > I found a code like "if sys.version_info < (2, 4):" in some tests. Should > old versions (< 2.6) be supported in tests? Can such support code be removed > (note that other tests likely doesn't compatible with old versions)? It would be great if it could all be killed, but I suppose it might be in some externally maintained module. Which tests? -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How old Python version should be supported in tests?
On Thu, 27 Dec 2012 10:24:37 -0600, Benjamin Peterson wrote: > 2012/12/27 Serhiy Storchaka : > > I found a code like "if sys.version_info < (2, 4):" in some tests. Should > > old versions (< 2.6) be supported in tests? Can such support code be removed > > (note that other tests likely doesn't compatible with old versions)? > > It would be great if it could all be killed, but I suppose it might be > in some externally maintained module. Which tests? There are also a few cases where for one reason or another the module maintainer wants it to stay backward compatible. I'm thinking specifically of the platform module, which I'm pretty sure has that restriction. So, it has to be considered on a case by case basis. --David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How old Python version should be supported in tests?
2012/12/27 Serhiy Storchaka : > четвер 27 грудень 2012 18:24:37 ви написали: >> It would be great if it could all be killed, but I suppose it might be >> in some externally maintained module. Which tests? > > They are bsddb, sqlite3, ctypes and multiprocessing. I don't see the point in permuting thing too much in the 2.7 branch. -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (merge 2.7 -> 2.7): Null merge.
On Thu, Dec 27, 2012 at 12:05 PM, serhiy.storchaka wrote: > http://hg.python.org/cpython/rev/26eb2979465c > changeset: 81094:26eb2979465c > branch: 2.7 > parent: 81086:ccbb16719540 > parent: 81090:d3c81ef728ae > user:Serhiy Storchaka > date:Thu Dec 27 22:00:12 2012 +0200 > summary: > Null merge. Great to see your first check-ins, Serhiy. Congratulations! I think for this case we usually say "Merge heads," which is different from the case of a null merge (i.e. where the diff is empty, for example when registering that a 3.x commit should not be forward-ported to a later version). --Chris > > files: > Lib/idlelib/EditorWindow.py | 2 +- > Misc/NEWS | 3 +++ > 2 files changed, 4 insertions(+), 1 deletions(-) > > > diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py > --- a/Lib/idlelib/EditorWindow.py > +++ b/Lib/idlelib/EditorWindow.py > @@ -1611,7 +1611,7 @@ > try: > try: > _tokenize.tokenize(self.readline, self.tokeneater) > -except _tokenize.TokenError: > +except (_tokenize.TokenError, SyntaxError): > # since we cut off the tokenizer early, we can trigger > # spurious errors > pass > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -168,6 +168,9 @@ > Library > --- > > +- Issue #16504: IDLE now catches SyntaxErrors raised by tokenizer. Patch by > + Roger Serwy. > + > - Issue #16702: test_urllib2_localnet tests now correctly ignores proxies for >localhost tests. > > > -- > Repository URL: http://hg.python.org/cpython > > ___ > Python-checkins mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-checkins > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Range information in the AST -- once more
2012/12/27 Guido van Rossum : > So just submit a patch to the tracker... > > --Guido > > > On Thursday, December 27, 2012, Sven Brauch wrote: >> >> 2012/12/27 Nick Coghlan : >> > It certainly sounds like its worth considering for 3.4. It's a new >> > feature, though, so it unfortunately wouldn't be possible to backport >> > it to any earlier releases. >> >> Yes, that is understandable. It wouldn't be much of a problem tough, >> my whole project is pretty bleeding-edge anyways, and depending on >> python >= 3.4 wouldn't hurt. For me it would only be important to have >> an acceptable solution for this long-term. >> >> Greetings, >> Sven >> ___ >> Python-Dev mailing list >> [email protected] >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > -- > --Guido van Rossum (python.org/~guido) I submitted a patch to the tracker, see http://bugs.python.org/issue16795. The patch only contains the very minimum set of changes which are necessary. There's still a few things you'll need to work around when writing a static language analyzer, but none of them is too much work, so I didn't include them for now in order to keep things compact. Thanks and best regards, Sven ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
