Re: [Python-Dev] Static checker for common Python programming errors

2014-11-18 Thread Stefan Bucur 
Thanks for the pointer! There seem indeed to be more formal analysis tools
for JavaScript than for Python (e.g., the most recent one for JS I know of
is the Jalangi framework [1]). I assume the main reason is that JavaScript
is standardized and somewhat simpler, so it's easier to construct formal
specs for all language features than it is for Python, which is also
evolving faster and relies on a lot of hard-to-model native functionality.

That's why I'm planning to reuse as much as possible the "implicit specs"
of the interpreter implementation, instead of re-stating them in an
explicit model.

We already have an execution engine that uses the interpreter to
automatically explore multiple paths through a piece of Python code (you
can read here [2] the academic paper, with case studies for Python and
Lua). In turn, we could use that engine to discover paths, while checking
program properties along each path.

Guido's suggestion for a type checker raises some interesting applications
of this multi-path analysis. For instance, we could examine the type of the
objects assigned to a static variable across all discovered execution paths
and determine its consistency. This analysis could either start with no
type annotations and output suggested types, or take existing annotations
and check them against the actual types.

Thanks again,
Stefan

[1] https://github.com/SRA-SiliconValley/jalangi
[2] http://dslab.epfl.ch/pubs/chef.pdf

On Mon Nov 17 2014 at 8:50:21 PM Francis Giraldeau <
[email protected]> wrote:

> If I may, there are prior work on JavaScript that may be worth
> investigating. Formal verification of dynamically typed software is a
> challenging endeavour, but it is very valuable to avoid errors at runtime,
> providing benefits from strongly type language without the rigidity.
>
> http://cs.au.dk/~amoeller/papers/tajs/
>
> Good luck!
>
> Francis
>
> 2014-11-17 9:49 GMT-05:00 Stefan Bucur :
>
>> I'm developing a Python static analysis tool that flags common
>> programming errors in Python programs. The tool is meant to complement
>> other tools like Pylint (which perform checks at lexical and syntactic
>> level) by going deeper with the code analysis and keeping track of the
>> possible control flow paths in the program (path-sensitive analysis).
>>
>> For instance, a path-sensitive analysis detects that the following
>> snippet of code would raise an AttributeError exception:
>>
>> if object is None: # If the True branch is taken, we know the object is
>> None
>>   object.doSomething() # ... so this statement would always fail
>>
>> I'm writing first to the Python developers themselves to ask, in their
>> experience, what common pitfalls in the language & its standard library
>> such a static checker should look for. For instance, here [1] is a list of
>> static checks for the C++ language, as part of the Clang static analyzer
>> project.
>>
>> My preliminary list of Python checks is quite rudimentary, but maybe
>> could serve as a discussion starter:
>>
>> * Proper Unicode handling (for 2.x)
>>   - encode() is not called on str object
>>   - decode() is not called on unicode object
>> * Check for integer division by zero
>> * Check for None object dereferences
>>
>> Thanks a lot,
>> Stefan Bucur
>>
>> [1] http://clang-analyzer.llvm.org/available_checks.html
>>
>>
>> ___
>> Python-Dev mailing list
>> [email protected]
>> https://mail.python.org/mailman/listinfo/python-dev
>>
> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/francis.giraldeau%40gmail.com
>>
>>
>
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] understanding ubuntu's patches to cpython

2014-11-18 Thread Matthias Klose 

On 11/16/2014 11:23 PM, Buck Evan wrote:

This particular patch caused virtualenv issue #118 and a subsequent
workaround:
http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/trusty/python2.7/trusty/view/head:/debian/patches/distutils-install-layout.diff#L281


Although this patch sits on the trusty branch of cpython patckaging, in
point of fact on that line i have `return
os.path.join(get_config_var('LIBPL'), "Makefile")`. I see that other
patches in the same project also touch this line, but none of them have
this LIBPL code that I find in my actual installation.


I'm not sure what issue you have here.  Indeed, as Ned suggested, best thing 
would be to open a bug report in Launchpad [1].



Can any of you all help me understand what's going on here? I'm trying to
pinpoint exactly when/how the problematic code (.replace("/usr/local","/usr")
went away.


See the README.Debian.  In the past people installed packages into the system 
path using "sudo python setup.py --install", and breaking applications shipped 
with Ubuntu.  This avoids accidental installation of such packages. 
Unfortunately you can do this again with pip, because pip acting on the system 
python silently overwrites system installed python modules.


  Matthias

[1] https://bugs.launchpad.net/ubuntu/+source/python2.7/+filebug

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Static checker for common Python programming errors

2014-11-18 Thread Guido van Rossum 
Please also check [email protected].
On Nov 18, 2014 3:06 AM, "Stefan Bucur"  wrote:

> Thanks for the pointer! There seem indeed to be more formal analysis tools
> for JavaScript than for Python (e.g., the most recent one for JS I know of
> is the Jalangi framework [1]). I assume the main reason is that JavaScript
> is standardized and somewhat simpler, so it's easier to construct formal
> specs for all language features than it is for Python, which is also
> evolving faster and relies on a lot of hard-to-model native functionality.
>
> That's why I'm planning to reuse as much as possible the "implicit specs"
> of the interpreter implementation, instead of re-stating them in an
> explicit model.
>
> We already have an execution engine that uses the interpreter to
> automatically explore multiple paths through a piece of Python code (you
> can read here [2] the academic paper, with case studies for Python and
> Lua). In turn, we could use that engine to discover paths, while checking
> program properties along each path.
>
> Guido's suggestion for a type checker raises some interesting applications
> of this multi-path analysis. For instance, we could examine the type of the
> objects assigned to a static variable across all discovered execution paths
> and determine its consistency. This analysis could either start with no
> type annotations and output suggested types, or take existing annotations
> and check them against the actual types.
>
> Thanks again,
> Stefan
>
> [1] https://github.com/SRA-SiliconValley/jalangi
> [2] http://dslab.epfl.ch/pubs/chef.pdf
>
> On Mon Nov 17 2014 at 8:50:21 PM Francis Giraldeau <
> [email protected]> wrote:
>
>> If I may, there are prior work on JavaScript that may be worth
>> investigating. Formal verification of dynamically typed software is a
>> challenging endeavour, but it is very valuable to avoid errors at runtime,
>> providing benefits from strongly type language without the rigidity.
>>
>> http://cs.au.dk/~amoeller/papers/tajs/
>>
>> Good luck!
>>
>> Francis
>>
>> 2014-11-17 9:49 GMT-05:00 Stefan Bucur :
>>
>>> I'm developing a Python static analysis tool that flags common
>>> programming errors in Python programs. The tool is meant to complement
>>> other tools like Pylint (which perform checks at lexical and syntactic
>>> level) by going deeper with the code analysis and keeping track of the
>>> possible control flow paths in the program (path-sensitive analysis).
>>>
>>> For instance, a path-sensitive analysis detects that the following
>>> snippet of code would raise an AttributeError exception:
>>>
>>> if object is None: # If the True branch is taken, we know the object is
>>> None
>>>   object.doSomething() # ... so this statement would always fail
>>>
>>> I'm writing first to the Python developers themselves to ask, in their
>>> experience, what common pitfalls in the language & its standard library
>>> such a static checker should look for. For instance, here [1] is a list of
>>> static checks for the C++ language, as part of the Clang static analyzer
>>> project.
>>>
>>> My preliminary list of Python checks is quite rudimentary, but maybe
>>> could serve as a discussion starter:
>>>
>>> * Proper Unicode handling (for 2.x)
>>>   - encode() is not called on str object
>>>   - decode() is not called on unicode object
>>> * Check for integer division by zero
>>> * Check for None object dereferences
>>>
>>> Thanks a lot,
>>> Stefan Bucur
>>>
>>> [1] http://clang-analyzer.llvm.org/available_checks.html
>>>
>>>
>>> ___
>>> Python-Dev mailing list
>>> [email protected]
>>> https://mail.python.org/mailman/listinfo/python-dev
>>>
>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/francis.giraldeau%40gmail.com
>>>
>>>
>>
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] understanding ubuntu's patches to cpython

2014-11-18 Thread Buck Evan 
I feel that you've misunderstood my request entirely. I'm not trying to
report an issue, just requesting help understanding how the resulting
package relates to version-controlled source. It's possible that I have the
wrong source entirely.

Your comments seem to indicate that you believe I've clobbered my
installation with something other than the standard ubuntu package. I
attempt to show otherwise below. I'm pretty confident that the source on
disk ('LIBPL') represents the package that all trusty users have. My aim is
to track down what (version-controlled) change caused this function to
change between precise and trusty.


$ ls -l /usr/lib/python2.7/sysconfig.py
-rw-r--r-- 1 root root 25048 Mar 22  2014 /usr/lib/python2.7/sysconfig.py

$ sudo touch !$
sudo touch /usr/lib/python2.7/sysconfig.py

$ ls -l /usr/lib/python2.7/sysconfig.py
-rw-r--r-- 1 root root 25048 Nov 18 08:57 /usr/lib/python2.7/sysconfig.py

$ sudo apt-get install --reinstall libpython2.7-minimal

Preparing to unpack .../libpython2.7-minimal_2.7.6-8_amd64.deb ...
Unpacking libpython2.7-minimal:amd64 (2.7.6-8) over (2.7.6-8) ...
Setting up libpython2.7-minimal:amd64 (2.7.6-8) ...

$ ls -l /usr/lib/python2.7/sysconfig.py
-rw-r--r-- 1 root root 25048 Mar 22  2014 /usr/lib/python2.7/sysconfig.py

$ grep -A3 def\ _get_makefile_filename /usr/lib/python2.7/sysconfig.py
def _get_makefile_filename():
if _PYTHON_BUILD:
return os.path.join(_PROJECT_BASE, "Makefile")
return os.path.join(get_config_var('LIBPL'), "Makefile")

$ apt-cache show libpython2.7-dev
Package: libpython2.7-dev
Priority: optional
Section: libdevel
Installed-Size: 32956
Maintainer: Ubuntu Developers 
Original-Maintainer: Matthias Klose 
Architecture: amd64
Source: python2.7
Version: 2.7.6-8
Replaces: python2.7 (<< 2.7-3), python2.7-dev (<< 2.7.3-10),
python2.7-minimal (<< 2.7.3-10)
Depends: libpython2.7-stdlib (= 2.7.6-8), libpython2.7 (= 2.7.6-8),
libexpat1-dev
Pre-Depends: multiarch-support
Recommends: libc6-dev | libc-dev
Filename: pool/main/p/python2.7/libpython2.7-dev_2.7.6-8_amd64.deb
Size: 22005790
MD5sum: 6b088fe5ee9bdb69296447a0b51f926e
SHA1: 6356820a6b7fce13e1539dbe88b4292a0c8f2a8d
SHA256: 646170fe0b0976d38252e474a8ac4ef5d52af93158b5a489da92c2225fbd1640
Description-en: Header files and a static library for Python (v2.7)
 Header files, a static library and development tools for building
 Python (v2.7) modules, extending the Python interpreter or embedding
 Python (v2.7) in applications.
 .
 Maintainers of Python packages should read README.maintainers.
 .
 This package contains development files. It is normally not
 used on it's[sic] own, but as a dependency of python2.7-dev.
Description-md5: 2ef5e0390998d9ab1872bc0f6b694ff7
Multi-Arch: same
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Origin: Ubuntu
Supported: 5y


On Tue, Nov 18, 2014 at 2:56 AM, Matthias Klose  wrote:

> On 11/16/2014 11:23 PM, Buck Evan wrote:
>
>> This particular patch caused virtualenv issue #118 and a subsequent
>> workaround:
>> http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/trusty/
>> python2.7/trusty/view/head:/debian/patches/distutils-
>> install-layout.diff#L281
>> > python2.7/trusty-proposed/view/head:/debian/patches/
>> distutils-install-layout.diff#L281>
>>
>> Although this patch sits on the trusty branch of cpython patckaging, in
>> point of fact on that line i have `return
>> os.path.join(get_config_var('LIBPL'), "Makefile")`. I see that other
>> patches in the same project also touch this line, but none of them have
>> this LIBPL code that I find in my actual installation.
>>
>
> I'm not sure what issue you have here.  Indeed, as Ned suggested, best
> thing would be to open a bug report in Launchpad [1].
>
>  Can any of you all help me understand what's going on here? I'm trying to
>> pinpoint exactly when/how the problematic code
>> (.replace("/usr/local","/usr")
>> went away.
>>
>
> See the README.Debian.  In the past people installed packages into the
> system path using "sudo python setup.py --install", and breaking
> applications shipped with Ubuntu.  This avoids accidental installation of
> such packages. Unfortunately you can do this again with pip, because pip
> acting on the system python silently overwrites system installed python
> modules.
>
>   Matthias
>
> [1] https://bugs.launchpad.net/ubuntu/+source/python2.7/+filebug
>
>
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com