Re: [Python-Dev] Python 3 as a Default in Linux Distros
On Wed, Jul 24, 2013 at 7:22 PM, Laurent Gautier wrote: > On 07/24/2013 06:30 PM, Chris Angelico wrote: >> >> On Thu, Jul 25, 2013 at 2:21 AM, Laurent Gautier >> wrote: >>> >>> - errors that are typical of "Python 2 script running with Python >>> 3"-specific are probably limited (e.g., use of unicode, use of xrange, >>> etc...) >>> >> The most common, in interactive scripts at least, is likely to be: >> > print "Hello, world!" >> >> SyntaxError: invalid syntax >> >> How helpful it's possible to make that one, I don't know. Is it safe >> to presume that it's more likely a syntax error will come from an >> interpreter version mismatch than a code bug? > > > The wrapper in /usr/bin/python: > - could use what is in 2to3. I think that most of the cases are solved > there. > - whenever interactive, could have an intermediate layer between the input > in the console and execution. So you suggest that instead of a clear SyntaxError you should end up with a confusing error (something has no attribute xyz or so) after a while (if say somone tries to load twisted via 2to3). ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 as a Default in Linux Distros
On Jul 24, 2013 6:37 AM, "Brett Cannon" wrote: > The key, though, is adding python2 and getting your code to use that binary specifically so that shifting the default name is more of a convenience than something which might break existing code not ready for the switch. > Applicable to this, does anyone know whether distutils, setuptools, distlib, or any of the other standard build+install tools are doing shebang requiring? Are they doing the right thing wrt python vs python2? -Toshio ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 as a Default in Linux Distros
On 07/25/2013 11:45 AM, Maciej Fijalkowski wrote: On Wed, Jul 24, 2013 at 7:22 PM, Laurent Gautier wrote: On 07/24/2013 06:30 PM, Chris Angelico wrote: On Thu, Jul 25, 2013 at 2:21 AM, Laurent Gautier wrote: - errors that are typical of "Python 2 script running with Python 3"-specific are probably limited (e.g., use of unicode, use of xrange, etc...) The most common, in interactive scripts at least, is likely to be: print "Hello, world!" SyntaxError: invalid syntax How helpful it's possible to make that one, I don't know. Is it safe to presume that it's more likely a syntax error will come from an interpreter version mismatch than a code bug? The wrapper in /usr/bin/python: - could use what is in 2to3. I think that most of the cases are solved there. - whenever interactive, could have an intermediate layer between the input in the console and execution. So you suggest that instead of a clear SyntaxError you should end up with a confusing error (something has no attribute xyz or so) after a while (if say somone tries to load twisted via 2to3). In a sense, yes. I think that the a priori (expectations) a user has plays a role in whether something is confusing or not. Consider the two following situations: - a programmer is working on code, and is testing it. If the program fails, this is because he/she made a mistake and the error message should indicate where the error originates from - a user is running a python script (he expects to be working), and is using the default /usr/bin/python (formerly Python 2, now Python 3). If the program fails because of obvious Python 2-only idioms, reporting this rather that the SyntaxError is much less confusing. Having that said, the comments are pointing out that this is may be not a completely good idea (most notably because of how slow 2to3 is). A wrapper producing an unconditional warning about the default python being changed to Python 3, and differences between Python 2 and Python 3 the possible source of errors, (as I think it was suggested) is going to be a better idea. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 as a Default in Linux Distros
On Thu, Jul 25, 2013 at 12:41 PM, Laurent Gautier wrote: > - a user is running a python script (he expects to be working), and is using > the default /usr/bin/python (formerly Python 2, now Python 3). If the > program fails because of obvious Python 2-only idioms, reporting this rather > that the SyntaxError is much less confusing. 2to3 will not report this. It will get rid of the SyntaxError (which at least is fairly clear) and give you a completely different and even more obscure error. You have replaced a somewhat unclear error with a very unclear error. If we want to report a problem, then /usr/bin/python should just report that you should use /usr/bin/python2 or /usr/bin/python3. That's clear. //Lennart ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 as a Default in Linux Distros
On 07/25/2013 01:19 PM, Lennart Regebro wrote: On Thu, Jul 25, 2013 at 12:41 PM, Laurent Gautier wrote: - a user is running a python script (he expects to be working), and is using the default /usr/bin/python (formerly Python 2, now Python 3). If the program fails because of obvious Python 2-only idioms, reporting this rather that the SyntaxError is much less confusing. 2to3 will not report this. My meaning was the use of 2to3's machinery (and fire a message if a translation occurs) not 2to3 itself, obviously. It will get rid of the SyntaxError (which at least is fairly clear) and give you a completely different and even more obscure error. You have replaced a somewhat unclear error with a very unclear error. If we want to report a problem, then /usr/bin/python should just report that you should use /usr/bin/python2 or /usr/bin/python3. That's clear. //Lennart ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 as a Default in Linux Distros
On 25 July 2013 20:38, Toshio Kuratomi wrote: > > On Jul 24, 2013 6:37 AM, "Brett Cannon" wrote: >> The key, though, is adding python2 and getting your code to use that >> binary specifically so that shifting the default name is more of a >> convenience than something which might break existing code not ready for the >> switch. >> > Applicable to this, does anyone know whether distutils, setuptools, distlib, > or any of the other standard build+install tools are doing shebang > requiring? Are they doing the right thing wrt python vs python2? It occurs to me they're almost certainly using "sys.executable" to set the shebang line, so probably not :( distutils-sig could probably offer a better answer though, most of the packaging folks don't hang out here. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 as a Default in Linux Distros
On 25 July 2013 10:06, Nick Coghlan wrote: > > On 25 Jul 2013 05:30, "Toshio Kuratomi" wrote: >> >> On Wed, Jul 24, 2013 at 12:42:09PM -0400, Barry Warsaw wrote: >> > On Jul 25, 2013, at 01:41 AM, Nick Coghlan wrote: >> > >> > >How's this for an updated wording in the abstract: >> > > >> > > * for the time being, all distributions should ensure that python >> > >refers to the same target as python2 >> > > * however, users should be aware that python refers to python3 on at >> > >least Arch Linux (that change is >> > >what prompted the creation of this PEP), so "python" should be >> > >used in the shebang line only for >> > >scripts that are source compatible with both Python 2 and 3 >> > >> > +1 >> > >> +1 as well. Much clearer. > > OK, unless someone gets to it before me, I'll update it tonight. Update is here: http://hg.python.org/peps/rev/4b379a690ae2 I added one final bullet point to the abstract with advice for users: * in preparation for an eventual change in the default version of Python, Python 2 only scripts should either be updated to be source compatible with Python 3 or else to use python2 in the shebang line. I also rewrote the first two bullet points in the Migration Notes section, as I realised they contributed heavily to the notion that distros like Fedora should switch aggressively, while only the conservative distros like RHEL and Debian stable should preserve the old behaviour (which wasn't the intent at all). Finally, I added a couple of other notes: * one pointing out the kind of time frame we're thinking about (i.e. years) by mentioning the 2.7 switch to security fix only mode * one noting that the Python launcher for Windows also default to Python 2 if both 2.x and 3.x are installed, so we're also aiming for cross platform consistency here Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Official github mirror for CPython?
Hi all, I've been looking for a Github mirror for Python, and found two: * https://github.com/python-git/python has a lot of forks/watches/starts but seems to be very out of date (last updated 4 years ago) * https://github.com/python-mirror/python doesn't appear to be very popular but is updated daily Are some of you the owners of these repositories? Should we consolidate to a single "semi-official" mirror? Eli ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Official github mirror for CPython?
On Fri, Jul 26, 2013 at 12:29 AM, Eli Bendersky wrote: > Hi all, > > I've been looking for a Github mirror for Python, and found two: > > * https://github.com/python-git/python has a lot of forks/watches/starts but > seems to be very out of date (last updated 4 years ago) > * https://github.com/python-mirror/python doesn't appear to be very popular > but is updated daily Also https://github.com/akheron/cpython seems to be the same again. ChrisA ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Official github mirror for CPython?
Am 25.07.2013 16:29, schrieb Eli Bendersky: > Hi all, > > I've been looking for a Github mirror for Python, and found two: > > * https://github.com/python-git/python has a lot of forks/watches/starts > but seems to be very out of date (last updated 4 years ago) > * https://github.com/python-mirror/python doesn't appear to be very > popular but is updated daily > > Are some of you the owners of these repositories? Should we consolidate > to a single "semi-official" mirror? +1 Does the PSF have an official account on github? We have one on bitbucket... Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Official github mirror for CPython?
On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes wrote: > Am 25.07.2013 16:29, schrieb Eli Bendersky: >> Hi all, >> >> I've been looking for a Github mirror for Python, and found two: >> >> * https://github.com/python-git/python has a lot of forks/watches/starts >> but seems to be very out of date (last updated 4 years ago) >> * https://github.com/python-mirror/python doesn't appear to be very >> popular but is updated daily >> >> Are some of you the owners of these repositories? Should we consolidate >> to a single "semi-official" mirror? > > +1 > > Does the PSF have an official account on github? We have one on bitbucket... I don't remember who runs this, and I thought I was in it (maybe just on BB), but: https://github.com/python ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Official github mirror for CPython?
Am 25.07.2013 16:48, schrieb Brian Curtin: > On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes > wrote: >> Am 25.07.2013 16:29, schrieb Eli Bendersky: >>> Hi all, >>> >>> I've been looking for a Github mirror for Python, and found two: >>> >>> * https://github.com/python-git/python has a lot of forks/watches/starts >>> but seems to be very out of date (last updated 4 years ago) >>> * https://github.com/python-mirror/python doesn't appear to be very >>> popular but is updated daily >>> >>> Are some of you the owners of these repositories? Should we consolidate >>> to a single "semi-official" mirror? >> >> +1 >> >> Does the PSF have an official account on github? We have one on bitbucket... > > I don't remember who runs this, and I thought I was in it (maybe just > on BB), but: https://github.com/python Jesse might know more about it. He is an admin of https://bitbucket.org/PSF/, too. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Official github mirror for CPython?
On Thu, Jul 25, 2013 at 7:48 AM, Brian Curtin wrote: > On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes > wrote: > > Am 25.07.2013 16:29, schrieb Eli Bendersky: > >> Hi all, > >> > >> I've been looking for a Github mirror for Python, and found two: > >> > >> * https://github.com/python-git/python has a lot of > forks/watches/starts > >> but seems to be very out of date (last updated 4 years ago) > >> * https://github.com/python-mirror/python doesn't appear to be very > >> popular but is updated daily > >> > >> Are some of you the owners of these repositories? Should we consolidate > >> to a single "semi-official" mirror? > > > > +1 > > > > Does the PSF have an official account on github? We have one on > bitbucket... > > I don't remember who runs this, and I thought I was in it (maybe just > on BB), but: https://github.com/python > This appears to be a good place, but it doesn't have an actual mirror (yet?) ;-) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Daemon creation code in the standard library (was: Inherance of file descriptor and handles on Windows (PEP 446))
Le Thu, 25 Jul 2013 12:08:18 +1000, Ben Finney a écrit : > Guido van Rossum writes: > > > To reduce the need for 3rd party subprocess creation code, we should > > have better daemon creation code in the stdlib -- I wrote some damn > > robust code for this purpose in my previous job, but it never saw > > the light of day. > > Work continues on the PEP 3143-compatible ‘python-daemon’, porting it > to Python 3 and aiming for inclusion in the standard library. The PEP hasn't been formally accepted yet, however. Skimming back through the archives, one sticking point was the default value of the "umask" parameter. Setting the umask to 0 if the user didn't ask for something else is a disaster, security-wise. Another problem I've had when using it is that the `pidfile` combines the notion of pidfile and process lock in one unique attribute. This is quite inflexible when you're using something else than Skip Montanaro's "lockfile" library. I'm using a separate lock based on locket.py: https://github.com/mwilliamson/locket.py because it is based on POSIX advisory locks, and therefore doesn't suffer from the "stale pid file" issues you get when a process (or the whole system) crashes. Therefore I'd be -1 on the PEP until those issues are alleviated. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Official github mirror for CPython?
On Thu, Jul 25, 2013 at 11:18 AM, Eli Bendersky wrote: > > > > On Thu, Jul 25, 2013 at 7:48 AM, Brian Curtin wrote: > >> On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes >> wrote: >> > Am 25.07.2013 16:29, schrieb Eli Bendersky: >> >> Hi all, >> >> >> >> I've been looking for a Github mirror for Python, and found two: >> >> >> >> * https://github.com/python-git/python has a lot of >> forks/watches/starts >> >> but seems to be very out of date (last updated 4 years ago) >> >> * https://github.com/python-mirror/python doesn't appear to be very >> >> popular but is updated daily >> >> >> >> Are some of you the owners of these repositories? Should we consolidate >> >> to a single "semi-official" mirror? >> > >> > +1 >> > >> > Does the PSF have an official account on github? We have one on >> bitbucket... >> >> I don't remember who runs this, and I thought I was in it (maybe just >> on BB), but: https://github.com/python >> > > This appears to be a good place, but it doesn't have an actual mirror > (yet?) ;-) > Based on the list of people who are members of github.com/python it's as official as it's going to get (depends on who of that group owns it). But assuming whomever owns it is okay with hosting a mirror, what exactly is going to be required to have it updated regularly? Someone is going to have to write the scripts to pull from the hg repo and then push up to github else it's just going to end up out-of-date on top of maintaining whatever setup is devised. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 as a Default in Linux Distros
On Thu, Jul 25, 2013 at 1:53 PM, Laurent Gautier wrote: > My meaning was the use of 2to3's machinery (and fire a message if a > translation occurs) not 2to3 itself, obviously. I don't understand what you mean is the difference. //Lennart ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 as a Default in Linux Distros
On Thu, Jul 25, 2013 at 10:25:26PM +1000, Nick Coghlan wrote: > On 25 July 2013 20:38, Toshio Kuratomi wrote: > > > > On Jul 24, 2013 6:37 AM, "Brett Cannon" wrote: > >> The key, though, is adding python2 and getting your code to use that > >> binary specifically so that shifting the default name is more of a > >> convenience than something which might break existing code not ready for > >> the > >> switch. > >> > > Applicable to this, does anyone know whether distutils, setuptools, distlib, > > or any of the other standard build+install tools are doing shebang > > requiring? Are they doing the right thing wrt python vs python2? > > It occurs to me they're almost certainly using "sys.executable" to set > the shebang line, so probably not :( > > distutils-sig could probably offer a better answer though, most of the > packaging folks don't hang out here. > Thanks! For other Linux distributors following along, here's my message to distutils-sig: http://mail.python.org/pipermail/distutils-sig/2013-July/022001.html -Toshio pgpRDSrmks3t3.pgp Description: PGP signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Official github mirror for CPython?
On Thu, Jul 25, 2013 at 5:30 PM, Brett Cannon wrote: > > > > On Thu, Jul 25, 2013 at 11:18 AM, Eli Bendersky wrote: >> >> >> >> >> On Thu, Jul 25, 2013 at 7:48 AM, Brian Curtin wrote: >>> >>> On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes >>> wrote: >>> > Am 25.07.2013 16:29, schrieb Eli Bendersky: >>> >> Hi all, >>> >> >>> >> I've been looking for a Github mirror for Python, and found two: >>> >> >>> >> * https://github.com/python-git/python has a lot of >>> >> forks/watches/starts >>> >> but seems to be very out of date (last updated 4 years ago) >>> >> * https://github.com/python-mirror/python doesn't appear to be very >>> >> popular but is updated daily >>> >> >>> >> Are some of you the owners of these repositories? Should we >>> >> consolidate >>> >> to a single "semi-official" mirror? >>> > >>> > +1 >>> > >>> > Does the PSF have an official account on github? We have one on >>> > bitbucket... >>> >>> I don't remember who runs this, and I thought I was in it (maybe just >>> on BB), but: https://github.com/python >> >> >> This appears to be a good place, but it doesn't have an actual mirror >> (yet?) ;-) > > > Based on the list of people who are members of github.com/python it's as > official as it's going to get (depends on who of that group owns it). > > But assuming whomever owns it is okay with hosting a mirror, what exactly is > going to be required to have it updated regularly? Someone is going to have > to write the scripts to pull from the hg repo and then push up to github > else it's just going to end up out-of-date on top of maintaining whatever > setup is devised. > It's slightly more work than this. You need to keep all the features relying on hg work, like the revision version it was compiled from etc. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Coverity Scan
Hello, this is an update on my work and the current status of Coverity Scan. Maybe you have noticed a checkins made be me that end with the line "CID #". These are checkins that fix an issue that was discovered by the static code analyzer Coverity. Coverity is a commercial product but it's a free service for some Open Source projects. Python has been analyzed by Coverity since about 2007. Guido, Neal, Brett, Stefan and some other developers have used Coverity before I took over. I fixed a couple of issues before 3.3 reached the RC phase and more bugs in the last couple of months. Coverity is really great and its web GUI is fun to use, too. I was able to identify and fix resource leaks, NULL pointer issues, buffer overflows and missing checks all over the place. Because it's a static analyzer that follows data-flows and control-flows the tool can detect issues in error paths that are hardly visited at all. I have started to document Coverity here: http://docs.python.org/devguide/coverity.html Interview - A week ago I was contacted by Coverity. They have started a series of articles and press releases about Open Source projects that use their free service Coverity Scan, see http://www.coverity.com/company/press-releases/read/coverity-introduces-monthly-spotlight-series-for-coverity-scan-open-source-projects Two days ago I had a lovely phone interview about my involvement in the Python project and our development style. They are going to release a nice article in a couple of weeks. In the mean time we have time to fix the remaining couple issues. We *might* be able to reach the highest coverity integrity level! I have dealt with all major issues so we just have to fix a couple of issues. Current stats - Lines of Code: 396,179 Defect Density: 0.05 Total defects: 1,054 Outstanding: 21 (Coverity Connect shows less) Dismissed:222 Fixed:811 http://i.imgur.com/NoELjcj.jpg http://i.imgur.com/eJSzTUX.jpg open issues --- http://bugs.python.org/issue17899 http://bugs.python.org/issue18556 http://bugs.python.org/issue18555 http://bugs.python.org/issue18552 http://bugs.python.org/issue18551 http://bugs.python.org/issue18550 http://bugs.python.org/issue18528 Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 as a Default in Linux Distros
On 07/25/2013 06:03 PM, Lennart Regebro wrote: On Thu, Jul 25, 2013 at 1:53 PM, Laurent Gautier wrote: My meaning was the use of 2to3's machinery (and fire a message if a translation occurs) not 2to3 itself, obviously. I don't understand what you mean is the difference. //Lennart bullet-point version: - 2to3 is a code translator - to function, it needs to identify Python 2-only idioms so they can be translated - the idea _was_ (*) to re-use that engine (identification-only, not translation) so the presence of Python 2-only idioms would cause a "Python 3 is now the default for "python", buddy. This script was seemingly written for Python 2, so you should run it with the command python2" sort of error message. The speed issue could be smaller than an full run of 2to3 since the error would be triggered at the first snippet of Python 2-only code encountered, but *. (*: I withdrew the suggestion few emails back) L. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coverity Scan
On 7/25/2013 2:48 PM, Christian Heimes wrote: Hello, this is an update on my work and the current status of Coverity Scan. Great work. Maybe you have noticed a checkins made be me that end with the line "CID #". These are checkins that fix an issue that was discovered by the static code analyzer Coverity. Coverity is a commercial product but it's a free service for some Open Source projects. Python has been analyzed by Coverity since about 2007. Guido, Neal, Brett, Stefan and some other developers have used Coverity before I took over. I fixed a couple of issues before 3.3 reached the RC phase and more bugs in the last couple of months. The benefit for us is not just improving Python having external verification of its excellence in relation both to other open-source projects and commercial software. Coverity is really great and its web GUI is fun to use, too. I was able to identify and fix resource leaks, NULL pointer issues, buffer overflows and missing checks all over the place. Because it's a static analyzer that follows data-flows and control-flows the tool can detect issues in error paths that are hardly visited at all. I have started to document Coverity here: http://docs.python.org/devguide/coverity.html Interview - A week ago I was contacted by Coverity. They have started a series of articles and press releases about Open Source projects that use their free service Coverity Scan, see http://www.coverity.com/company/press-releases/read/coverity-introduces-monthly-spotlight-series-for-coverity-scan-open-source-projects The intention is to promote the best of open source to industry. Two days ago I had a lovely phone interview about my involvement in the Python project and our development style. They are going to release a nice article in a couple of weeks. In the mean time we have time to fix the remaining couple issues. We *might* be able to reach the highest coverity integrity level! I have dealt with all major issues so we just have to fix a couple of issues. Current stats - Lines of Code: 396,179 C only? or does Python code now count as 'source code'? Defect Density: 0.05 = defects per thousand lines = 20/400 Anything under 1 is good. The release above reports Samba now at .6. http://www.pcworld.com/article/2038244/linux-code-is-the-benchmark-of-quality-study-concludes.html reports Linux 3.8 as having the same for 7.6 million lines. Total defects: 1,054 Outstanding: 21 (Coverity Connect shows less) Dismissed:222 This implies that they accept our designation of some things as False Positives or Intentional. Does Coverity do any review of such designations, so a project cannot cheat? Fixed:811 http://i.imgur.com/NoELjcj.jpg http://i.imgur.com/eJSzTUX.jpg open issues --- http://bugs.python.org/issue17899 http://bugs.python.org/issue18556 http://bugs.python.org/issue18555 http://bugs.python.org/issue18552 http://bugs.python.org/issue18551 http://bugs.python.org/issue18550 http://bugs.python.org/issue18528 -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coverity Scan
On 7/25/2013 6:00 PM, Terry Reedy wrote: Defect Density:0.05 = defects per thousand lines = 20/400 Anything under 1 is good. The release above reports Samba now at .6. http://www.pcworld.com/article/2038244/linux-code-is-the-benchmark-of-quality-study-concludes.html reports Linux 3.8 as having the same for 7.6 million lines. Total defects:1,054 Outstanding: 21 (Coverity Connect shows less) Dismissed: 222 This implies that they accept our designation of some things as False Positives or Intentional. Does Coverity do any review of such designations, so a project cannot cheat? I found the answer here https://docs.google.com/file/d/0B5wQCOK_TiRiMWVqQ0xPaDEzbkU/edit Coverity Integrity Level 1 is 1 (defect/1000 lines) Level 2 is .1 (we have passed that) Level 3 is .01 + no major defects + <20% (all all defects?) false positives as that is their normal rate.# A higher false positive rates requires auditing by Coverity. They claim "A higher false positive rate indicates misconfiguration, usage of unusual idioms, or incorrect diagnosis of a large number of defects." They else add "or a flaw in our analysis." # Since false positives should stay constant as true positives are reduced toward 0, false / all should tend toward 1 (100%) if I understand the ratio correctly. Fixed: 811 http://i.imgur.com/NoELjcj.jpg http://i.imgur.com/eJSzTUX.jpg -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coverity Scan
On Thu, 25 Jul 2013 18:00:55 -0400 Terry Reedy wrote: > On 7/25/2013 2:48 PM, Christian Heimes wrote: > > Hello, > > > > this is an update on my work and the current status of Coverity Scan. > > Great work. > > > > > Maybe you have noticed a checkins made be me that end with the line "CID > > #". These are checkins that fix an issue that was discovered by the > > static code analyzer Coverity. Coverity is a commercial product but it's > > a free service for some Open Source projects. Python has been analyzed > > by Coverity since about 2007. Guido, Neal, Brett, Stefan and some other > > developers have used Coverity before I took over. I fixed a couple of > > issues before 3.3 reached the RC phase and more bugs in the last couple > > of months. > > The benefit for us is not just improving Python having external > verification of its excellence in relation both to other open-source > projects and commercial software. "Excellence"? The term is too weak, I would say "perfection" at least, but perhaps we should go as far as "divinity". Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coverity Scan
Am 26.07.2013 00:32, schrieb Terry Reedy:
> I found the answer here
> https://docs.google.com/file/d/0B5wQCOK_TiRiMWVqQ0xPaDEzbkU/edit
> Coverity Integrity Level 1 is 1 (defect/1000 lines)
> Level 2 is .1 (we have passed that)
> Level 3 is .01 + no major defects + <20% (all all defects?) false
> positives as that is their normal rate.#
>
> A higher false positive rates requires auditing by Coverity. They claim
> "A higher false positive rate indicates misconfiguration, usage of
> unusual idioms, or incorrect diagnosis of a large number of defects."
> They else add "or a flaw in our analysis."
>
> # Since false positives should stay constant as true positives are
> reduced toward 0, false / all should tend toward 1 (100%) if I
> understand the ratio correctly.
About 40% of the dismissed cases are cause by a handful of issues. I
have documented these issues as "known limitations"
http://docs.python.org/devguide/coverity.html#known-limitations .
For example about 35 false positives are related to PyLong_FromLong()
and our small integer optimization. A correct modeling file would
eliminate the false positive defects. My attempts don't work as hoped
and I don't have access to all professional coverity tools to debug my
trials.
Nearly 20 false positives are caused by Py_BuildValue("N"). I'm still
astonished that Coverity understands Python's reference counting most of
the time. :)
Did I mention that we have almost reached Level 3? All major defects
have been dealt with (one of them locally on the test machine until
Larry pushes his patch soonish), 4 of 7 minor issues must be closed and
our dismissed rate is just little over 20% (222 out of 1054 = 21%).
Christian
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coverity Scan
Am 26.07.2013 00:50, schrieb Antoine Pitrou: > "Excellence"? The term is too weak, I would say "perfection" at least, > but perhaps we should go as far as "divinity". Don't forget that Python can offer lots of places to keep your bike clean and dry ... *scnr* ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coverity Scan
Am 26.07.2013 00:00, schrieb Terry Reedy: >> http://www.coverity.com/company/press-releases/read/coverity-introduces-monthly-spotlight-series-for-coverity-scan-open-source-projects >> > > The intention is to promote the best of open source to industry. I think it's also a marketing tool. They like to sell their product. I don't have a problem with that. After all Coverity provides a useful service for free that supplements our own debugging tools. >> Lines of Code:396,179 > > C only? or does Python code now count as 'source code'? It's just C code and headers. Coverity doesn't analyze Python code. According to cloc Python has 296707 + 78126 == 374833 lines of code in C and header files. I'm not sure why Coverity detects more. > >> Defect Density:0.05 > > = defects per thousand lines = 20/400 > > Anything under 1 is good. The release above reports Samba now at .6. > http://www.pcworld.com/article/2038244/linux-code-is-the-benchmark-of-quality-study-concludes.html > > reports Linux 3.8 as having the same for 7.6 million lines. These are amazing numbers. Python is much smaller. > >> Total defects:1,054 >> Outstanding: 21 (Coverity Connect shows less) >> Dismissed: 222 > > This implies that they accept our designation of some things as False > Positives or Intentional. Does Coverity do any review of such > designations, so a project cannot cheat? What's the point of cheating? :) I could dismiss any remaining defect as intentionally or false positive but that would only harm ourselves. As you already pointed out Coverity reserves the right to inspect dismissed bugs for their highest ranking. I'm in the process of looking through all dismissed defects. Some of them are relics of deleted files and removed code. Some other may go away with proper modeling. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coverity Scan
On 7/25/2013 6:56 PM, Christian Heimes wrote:
Am 26.07.2013 00:32, schrieb Terry Reedy:
# Since false positives should stay constant as true positives are
reduced toward 0, false / all should tend toward 1 (100%) if I
understand the ratio correctly.
Which I did not ;-).
About 40% of the dismissed cases are cause by a handful of issues. I
have documented these issues as "known limitations"
http://docs.python.org/devguide/coverity.html#known-limitations .
For example about 35 false positives are related to PyLong_FromLong()
and our small integer optimization. A correct modeling file would
eliminate the false positive defects. My attempts don't work as hoped
and I don't have access to all professional coverity tools to debug my
trials.
Perhaps Coverity will help when doing an audit.
Nearly 20 false positives are caused by Py_BuildValue("N"). I'm still
astonished that Coverity understands Python's reference counting most of
the time. :)
Did I mention that we have almost reached Level 3? All major defects
It is hard to measure the benefit of preventitive medicine, but I
imagine that we should see fewer mysterious crashes and heisenbugs than
we would have. In any case, Level 3 certification should help people
promoting the use of Python in organizational settings, whether as
employees or consultants.
have been dealt with (one of them locally on the test machine until
Larry pushes his patch soonish), 4 of 7 minor issues must be closed and
.1 * 390 allows 3 defects (or 4 if they round up) -- astonishingly good!
our dismissed rate is just little over 20% (222 out of 1054 = 21%).
So merely verifying the 35 PyLong_FromLong dismissals will put us under.
Thanks for clarifying the proper denominator -- all defects ever found.
It seems obvious in retrospect, but I was focused on current stats, not
the history.
--
Terry Jan Reedy
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Official github mirror for CPython?
On 26 Jul 2013 03:02, "Maciej Fijalkowski" wrote: > > On Thu, Jul 25, 2013 at 5:30 PM, Brett Cannon wrote: > > > > > > > > On Thu, Jul 25, 2013 at 11:18 AM, Eli Bendersky wrote: > >> > >> > >> > >> > >> On Thu, Jul 25, 2013 at 7:48 AM, Brian Curtin wrote: > >>> > >>> On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes < [email protected]> > >>> wrote: > >>> > Am 25.07.2013 16:29, schrieb Eli Bendersky: > >>> >> Hi all, > >>> >> > >>> >> I've been looking for a Github mirror for Python, and found two: > >>> >> > >>> >> * https://github.com/python-git/python has a lot of > >>> >> forks/watches/starts > >>> >> but seems to be very out of date (last updated 4 years ago) > >>> >> * https://github.com/python-mirror/python doesn't appear to be very > >>> >> popular but is updated daily > >>> >> > >>> >> Are some of you the owners of these repositories? Should we > >>> >> consolidate > >>> >> to a single "semi-official" mirror? > >>> > > >>> > +1 > >>> > > >>> > Does the PSF have an official account on github? We have one on > >>> > bitbucket... > >>> > >>> I don't remember who runs this, and I thought I was in it (maybe just > >>> on BB), but: https://github.com/python > >> > >> > >> This appears to be a good place, but it doesn't have an actual mirror > >> (yet?) ;-) > > > > > > Based on the list of people who are members of github.com/python it's as > > official as it's going to get (depends on who of that group owns it). > > > > But assuming whomever owns it is okay with hosting a mirror, what exactly is > > going to be required to have it updated regularly? Someone is going to have > > to write the scripts to pull from the hg repo and then push up to github > > else it's just going to end up out-of-date on top of maintaining whatever > > setup is devised. > > > > It's slightly more work than this. You need to keep all the features > relying on hg work, like the revision version it was compiled from > etc. To be honest, if people are going to spend time tinkering with our VCS infrastructure, one of the most interesting things we could do is explore what would be involved in setting up RhodeCode on hg.python.org :) (For those that haven't seen it, RhodeCode seems broadly comparable to BitBucket feature wise, but because of the way it is licensed, the source code is freely available to all, and running your own instance is free-as-in-beer for non-profits and open source projects). Cheers, Nick. > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Daemon creation code in the standard library (was: Inherance of file descriptor and handles on Windows (PEP 446))
On 25Jul2013 17:26, Antoine Pitrou wrote: | Le Thu, 25 Jul 2013 12:08:18 +1000, | Ben Finney a écrit : | > Work continues on the PEP 3143-compatible ‘python-daemon’, porting it | > to Python 3 and aiming for inclusion in the standard library. | | The PEP hasn't been formally accepted yet, however. | Skimming back through the archives, one sticking point was the default | value of the "umask" parameter. Setting the umask to 0 if the user | didn't ask for something else is a disaster, security-wise. I have to say, +10 here. I have always found the convention that daemons have a umask of 0 to be utterly bogus, because almost all library code relies on the umask to set default security policy for initial file permissions. Prone to rant on this at length if required... Cheers, -- Cameron Simpson Cordless hoses have been around for quite some time. They're called buckets. - Dan Prener ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
