Re: [Python-Dev] Daemon creation code in the standard library

2013-07-26 Thread Ben Finney 
Antoine Pitrou  writes:

> Therefore I'd be -1 on [PEP 3143] until those issues are alleviated.

Cameron Simpson  writes:

> I have always found the convention that daemons have a umask of 0
> to be utterly bogus, because almost all library code relies on the
> umask to set default security policy for initial file permissions.
>
> Prone to rant on this at length if required...


Thanks folks. We'd love to have this discussion over at the
‘python-daemon-devel’ discussion forum if you want to have it in more
detail.

-- 
 \ “We are all agreed that your theory is crazy. The question that |
  `\  divides us is whether it is crazy enough to have a chance of |
_o__)being correct.” —Niels Bohr (to Wolfgang Pauli), 1958 |
Ben Finney

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Daemon creation code in the standard library

2013-07-26 Thread Ronald Oussoren 

On 25 Jul, 2013, at 4:18, Ben Finney  wrote:

> Ben Finney  writes:
> 
>> Work continues on the PEP 3143-compatible ‘python-daemon’, porting it to
>> Python 3 and aiming for inclusion in the standard library.

At first glance the library appears to close all open files, with an option
to exclude some specific file descriptors (that is, you need to pass a list
of files that shouldn't be closed). 

That makes it a lot harder to do some initialization before daemonizing.
I prefer to perform at least some initialization early in program startup to
be able to give sensible error messages. I've had too many initscripts that
claimed to have started a daemon sucessfully, only to have that daemon stop
right away because it noticed a problem right after it detached itself.

Ronald

> 
> At PyPI http://pypi.python.org/pypi/python-daemon/>, and
> development co-ordinated at Alioth
> https://alioth.debian.org/projects/python-daemon/>.
> 
>> Interested parties are invited to join us on the discussion forums
> 
> The correct link for the ‘python-daemon-devel’ forum is
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/python-daemon-devel>.
> For announcements only, we have 
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/python-daemon-announce>.
> 
> -- 
> \“This sentence contradicts itself — no actually it doesn't.” |
>  `\   —Douglas Hofstadter |
> _o__)  |
> Ben Finney
> 
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Official github mirror for CPython?

2013-07-26 Thread Antoine Pitrou 
Le Fri, 26 Jul 2013 09:31:50 +1000,
Nick Coghlan  a écrit :
> 
> To be honest, if people are going to spend time tinkering with our VCS
> infrastructure, one of the most interesting things we could do is
> explore what would be involved in setting up RhodeCode on
> hg.python.org :)
> 
> (For those that haven't seen it, RhodeCode seems broadly comparable to
> BitBucket feature wise, but because of the way it is licensed, the
> source code is freely available to all, and running your own instance
> is free-as-in-beer for non-profits and open source projects).

By "freely available", do you mean actual open source / free software?

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Official github mirror for CPython?

2013-07-26 Thread Paul Moore 
On 26 July 2013 08:50, Antoine Pitrou  wrote:

> > (For those that haven't seen it, RhodeCode seems broadly comparable to
> > BitBucket feature wise, but because of the way it is licensed, the
> > source code is freely available to all, and running your own instance
> > is free-as-in-beer for non-profits and open source projects).
>
> By "freely available", do you mean actual open source / free software?
>

Looks like it - from the website: "Open source source control management
system for Mercurial and GIT with code-reviews, built in push/pull server,
LDAP/AD, permissions system and full text search."

Paul
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Official github mirror for CPython?

2013-07-26 Thread Ronald Oussoren 

On 26 Jul, 2013, at 9:50, Antoine Pitrou  wrote:

> Le Fri, 26 Jul 2013 09:31:50 +1000,
> Nick Coghlan  a écrit :
>> 
>> To be honest, if people are going to spend time tinkering with our VCS
>> infrastructure, one of the most interesting things we could do is
>> explore what would be involved in setting up RhodeCode on
>> hg.python.org :)
>> 
>> (For those that haven't seen it, RhodeCode seems broadly comparable to
>> BitBucket feature wise, but because of the way it is licensed, the
>> source code is freely available to all, and running your own instance
>> is free-as-in-beer for non-profits and open source projects).
> 
> By "freely available", do you mean actual open source / free software?

It appears to be GPLv3, with a for-pay enterprise edition. The latter is
free-as-in-beer for non-profits and open source projects.

Ronald

(See 
)

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Official github mirror for CPython?

2013-07-26 Thread Dirkjan Ochtman 
On Fri, Jul 26, 2013 at 9:50 AM, Antoine Pitrou  wrote:
>> (For those that haven't seen it, RhodeCode seems broadly comparable to
>> BitBucket feature wise, but because of the way it is licensed, the
>> source code is freely available to all, and running your own instance
>> is free-as-in-beer for non-profits and open source projects).
>
> By "freely available", do you mean actual open source / free software?

It seems to be licensed under the GPLv3.

https://secure.rhodecode.org/rhodecode/files/433d6385b216da52f68fa871ed1ff99f8d618613/COPYING
https://rhodecode.com/blog/25/new-rhodecode-licensing

Cheers,

Dirkjan
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Official github mirror for CPython?

2013-07-26 Thread Antoine Pitrou 
Le Fri, 26 Jul 2013 10:03:26 +0200,
Dirkjan Ochtman  a écrit :
> On Fri, Jul 26, 2013 at 9:50 AM, Antoine Pitrou 
> wrote:
> >> (For those that haven't seen it, RhodeCode seems broadly
> >> comparable to BitBucket feature wise, but because of the way it is
> >> licensed, the source code is freely available to all, and running
> >> your own instance is free-as-in-beer for non-profits and open
> >> source projects).
> >
> > By "freely available", do you mean actual open source / free
> > software?
> 
> It seems to be licensed under the GPLv3.
> 
> https://secure.rhodecode.org/rhodecode/files/433d6385b216da52f68fa871ed1ff99f8d618613/COPYING
> https://rhodecode.com/blog/25/new-rhodecode-licensing

Ah, right. Apparently a certain @ncoghlan_dev contributed to that...

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Official github mirror for CPython?

2013-07-26 Thread Nick Coghlan 
On 26 July 2013 18:14, Antoine Pitrou  wrote:
> Le Fri, 26 Jul 2013 10:03:26 +0200,
> Dirkjan Ochtman  a écrit :
>> On Fri, Jul 26, 2013 at 9:50 AM, Antoine Pitrou 
>> wrote:
>> >> (For those that haven't seen it, RhodeCode seems broadly
>> >> comparable to BitBucket feature wise, but because of the way it is
>> >> licensed, the source code is freely available to all, and running
>> >> your own instance is free-as-in-beer for non-profits and open
>> >> source projects).
>> >
>> > By "freely available", do you mean actual open source / free
>> > software?
>>
>> It seems to be licensed under the GPLv3.
>>
>> https://secure.rhodecode.org/rhodecode/files/433d6385b216da52f68fa871ed1ff99f8d618613/COPYING
>> https://rhodecode.com/blog/25/new-rhodecode-licensing
>
> Ah, right. Apparently a certain @ncoghlan_dev contributed to that...

Yeah, I've been publicly lamenting the closed source nature of GitHub
and BitBucket for a while now, dislike the Allura UI enough that "but
it's open source!" wasn't enough to win me over, and most of the other
free hosting options don't support Mercurial (or have anything
resembling a coherent business model). When RhodeCode brought their
hosted version to my attention, it aligned so well with what I had
been saying I wanted (a code hosting service making a genuine
commitment to openness, with a good user experience and an apparently
viable business model) that I felt *obliged* to check it out. While I
haven't migrated all my personal repos yet (due to some domain name
issues specific to the Hosted version), I was quite happy with the
user experience of the tool itself and plan to do that migration
eventually :)

As far as the licensing goes, rereading the comments below the
licensing announcement as well as the licenses themselves:

1. Versions prior to 2.0 (latest: 1.7.1) are already published under GPLv3
2. RhodeCode 2.0 (aka "RhodeCode Enterprise") will be the first
published using a Business Source license
3. The Business Source licensing means that 2.0 *will* become open
source under "GPLv2 or later" terms in August 2015 - RhodeCode won't
have the option of changing their mind and deciding not to open source
it after all. For the next two years, even though the source will be
made available, it's not technically an open source project yet, since
free usage under the business source license is limited to
installations with no more than 20 Active users. More users than that
requires a commercial license, which you either pay for, or is offered
for free to schools and universities, public open source projects and
also for development, environment and human rights NGOs.

I believe Hosted is already running 2.0, so that's the one that
impressed me. I'm not sure what 1.7.1 is like, since I have never used
it :)

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Official github mirror for CPython?

2013-07-26 Thread Antoine Pitrou 
Le Fri, 26 Jul 2013 20:51:16 +1000,
Nick Coghlan  a écrit :
> 
> I believe Hosted is already running 2.0, so that's the one that
> impressed me.

Strange. Apparently, free trial accounts get a 1.7.1 version:
"RhodeCode 1.7.1.hosted.4356".
From my POV, it would need a ton of UI polish before being pleasant
to use...

Also, it's not obvious whether you can integrate your repo with a
bug tracker and a code review tool.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Official github mirror for CPython?

2013-07-26 Thread Nick Coghlan 
On 26 July 2013 21:46, Antoine Pitrou  wrote:
> Le Fri, 26 Jul 2013 20:51:16 +1000,
> Nick Coghlan  a écrit :
>>
>> I believe Hosted is already running 2.0, so that's the one that
>> impressed me.
>
> Strange. Apparently, free trial accounts get a 1.7.1 version:
> "RhodeCode 1.7.1.hosted.4356".
> From my POV, it would need a ton of UI polish before being pleasant
> to use...

I tolerate Bugzilla and Gerrit every work day, which may have affected
my standards for what I judge to be an acceptable user interface ;)

> Also, it's not obvious whether you can integrate your repo with a
> bug tracker and a code review tool.

A self-hosted instance still has Mercurial on the backend, so any
Mercurial based integration should still work, even if it isn't
configurable through the front end.

However, I hadn't actually brought it up previously because I know
there would be a *lot* of work in taking it from "vague notion" to
"concrete infrastructure update proposal with a solid rationale behind
it" and I'm not in a position to do that work myself. I just couldn't
resist mentioning it once the idea of improved GitHub syncing was
brought up :)

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-07-26 Thread Victor Stinner 
2013/7/24 Guido van Rossum :
> But I'm also ready to propose that all this is such a mess that we
> *should* change the default fd/handle inheritance to False, *across
> platforms*, and damn the torpedoes -- i.e. accept breaking all
> existing 3rd party UNIX code for subprocess creation that bypasses the
> subprocess module, as well as breaking uses of os.spawn*() on both
> platforms that depend on FD inheritance beyond stdin/stdout/stderr).
>
> With the new, sane default, all we need instead of PEP 446 is a way to
> make an FD inheritable after it's been created, which can be a single
> os.make_inheritable(fd) call that you must apply to the fileno() of
> the stream or socket object you want inherited (or directly to a FD
> you created otherwise, e.g. with os.pipe()). On Windows, this should
> probably only work with os.spawn*(), since otherwise you need *handle*
> inheritance, not *FD* inheritance, and that's a non-portable concept
> anyway.

After having written 2 PEP on the topic, I slowly agree that make all
file descriptors non-inheritable is the best *compromise*. It solves
most, or all, issues.

The main drawback is the additionnal syscalls: on some platforms, 2
additional syscalls are need to make a file descriptor non-inheritable
for each creation of file descriptor. According to my benchmark on the
implementation of the PEP 433: the overhead of making a file
descriptor non-inheritable is between 1% and 3% (7.8 µs => 7.9 or 8.0
µs) on Linux 3.6.
http://www.python.org/dev/peps/pep-0433/#performances

Having to make a file descriptor inheritable after creating it
non-inheritable is also not optimal. Making it first non-inheritable
requires 0, 1 or 2 extra syscalls, and making it inheritable again
require also 1 or 2 syscalls. So f=open(...);
os.make_inheritable(f.fileno()) can take up to 5 syscalls (1 open + 4
fnctl), whereas it can be done in only 1 syscall (1 open). One of the
motivation of the PEP 433 an 446 is to reduce the number of syscalls,
even if the use case was to make sockets *non-inheritable*.

If we consider that the most common case is to use non-inheritable
file descriptors, having to call os.make_inheritable() may be
acceptable. Windows and recent operating syscalls support creating
file descriptor directly non-inheritable in a single syscalls. ioctl()
can be also be used instead of fcntl() to use 1 syscall instead of 2.

> We can fix multiprocessing any anything else in the stdlib that this
> breaks, I presume.

The CGI code rely on inheritance of file descriptors 0, 1 and 2 which
are pipes. The file descriptors 0, 1 and 2 are replaced with the pipes
using os.dup2().

Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-07-26 Thread Antoine Pitrou 
Le Fri, 26 Jul 2013 14:08:35 +0200,
Victor Stinner  a écrit :
> 
> After having written 2 PEP on the topic, I slowly agree that make all
> file descriptors non-inheritable is the best *compromise*. It solves
> most, or all, issues.

Even stdin/stdout/stderr? I think inheriting them is the sane default.

> The main drawback is the additionnal syscalls: on some platforms, 2
> additional syscalls are need to make a file descriptor non-inheritable
> for each creation of file descriptor. According to my benchmark on the
> implementation of the PEP 433: the overhead of making a file
> descriptor non-inheritable is between 1% and 3% (7.8 µs => 7.9 or 8.0
> µs) on Linux 3.6.

1% and 3% of what?
You're telling us there's a 0.1µs overhead. It's positively tiny.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Coverity Scan

2013-07-26 Thread Eli Bendersky 
Just a quick question - is there a chance to convince Coverity to detect
Python refcounting leaks in C API code :-) ? This could be useful not only
for Python but for extensions too. As it stands now, Coverity's leak
detection is Python must be pretty weak because almost everything is done
via PyObject refcounts.

Eli


On Thu, Jul 25, 2013 at 11:48 AM, Christian Heimes wrote:

> Hello,
>
> this is an update on my work and the current status of Coverity Scan.
>
> Maybe you have noticed a checkins made be me that end with the line "CID
> #". These are checkins that fix an issue that was discovered by the
> static code analyzer Coverity. Coverity is a commercial product but it's
> a free service for some Open Source projects. Python has been analyzed
> by Coverity since about 2007. Guido, Neal, Brett, Stefan and some other
> developers have used Coverity before I took over. I fixed a couple of
> issues before 3.3 reached the RC phase and more bugs in the last couple
> of months.
>
> Coverity is really great and its web GUI is fun to use, too. I was able
> to identify and fix resource leaks, NULL pointer issues, buffer
> overflows and missing checks all over the place. Because it's a static
> analyzer that follows data-flows and control-flows the tool can detect
> issues in error paths that are hardly visited at all. I have started to
> document Coverity here:
>
>   http://docs.python.org/devguide/coverity.html
>
>
> Interview
> -
>
> A week ago I was contacted by Coverity. They have started a series of
> articles and press releases about Open Source projects that use their
> free service Coverity Scan, see
>
>
>
> http://www.coverity.com/company/press-releases/read/coverity-introduces-monthly-spotlight-series-for-coverity-scan-open-source-projects
>
> Two days ago I had a lovely phone interview about my involvement in the
> Python project and our development style. They are going to release a
> nice article in a couple of weeks. In the mean time we have time to fix
> the remaining couple issues. We *might* be able to reach the highest
> coverity integrity level! I have dealt with all major issues so we just
> have to fix a couple of issues.
>
>
> Current stats
> -
>
> Lines of Code:  396,179
> Defect Density: 0.05
> Total defects:  1,054
> Outstanding:   21 (Coverity Connect shows less)
> Dismissed:222
> Fixed:811
>
> http://i.imgur.com/NoELjcj.jpg
> http://i.imgur.com/eJSzTUX.jpg
>
>
> open issues
> ---
>
> http://bugs.python.org/issue17899
> http://bugs.python.org/issue18556
> http://bugs.python.org/issue18555
> http://bugs.python.org/issue18552
> http://bugs.python.org/issue18551
> http://bugs.python.org/issue18550
> http://bugs.python.org/issue18528
>
>
> Christian
>
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/eliben%40gmail.com
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Coverity Scan

2013-07-26 Thread Christian Heimes 
Am 26.07.2013 14:56, schrieb Eli Bendersky:
> Just a quick question - is there a chance to convince Coverity to detect
> Python refcounting leaks in C API code :-) ? This could be useful not
> only for Python but for extensions too. As it stands now, Coverity's
> leak detection is Python must be pretty weak because almost everything
> is done via PyObject refcounts.

Coverity is able to detect some cases of refcount leaks. I don't know if
the software is able to keep track of all reference counts. But it
understands missing Py_DECREF() in error branches.

For example:

PyObject *n = PyLong_FromLong(0);
PyObject *u = PyUnicode_FromString("example");

if (u == NULL) {
return NULL;
/* Coverity detects that 'n' leaks memory */
}

Christian
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Coverity Scan

2013-07-26 Thread Brett Cannon 
On Thu, Jul 25, 2013 at 6:56 PM, Christian Heimes wrote:

> Am 26.07.2013 00:32, schrieb Terry Reedy:
> > I found the answer here
> > https://docs.google.com/file/d/0B5wQCOK_TiRiMWVqQ0xPaDEzbkU/edit
> > Coverity Integrity Level 1 is 1 (defect/1000 lines)
> > Level 2 is .1 (we have passed that)
> > Level 3 is .01 + no major defects + <20% (all all defects?) false
> > positives as that is their normal rate.#
> >
> > A higher false positive rates requires auditing by Coverity. They claim
> > "A higher false positive rate indicates misconfiguration, usage of
> > unusual idioms, or incorrect diagnosis of a large number of defects."
> > They else add "or a flaw in our analysis."
> >
> > # Since false positives should stay constant as true positives are
> > reduced toward 0, false / all should tend toward 1 (100%) if I
> > understand the ratio correctly.
>
> About 40% of the dismissed cases are cause by a handful of issues. I
> have documented these issues as "known limitations"
> http://docs.python.org/devguide/coverity.html#known-limitations .
>
> For example about 35 false positives are related to PyLong_FromLong()
> and our small integer optimization. A correct modeling file would
> eliminate the false positive defects. My attempts don't work as hoped
> and I don't have access to all professional coverity tools to debug my
> trials.
>

Have you tried asking for help from Coverity? They have been rather nice so
far and they may be willing to just give us free help in getting the
modeling file set up properly.

-Brett


>
> Nearly 20 false positives are caused by Py_BuildValue("N"). I'm still
> astonished that Coverity understands Python's reference counting most of
> the time. :)
>
> Did I mention that we have almost reached Level 3? All major defects
> have been dealt with (one of them locally on the test machine until
> Larry pushes his patch soonish), 4 of 7 minor issues must be closed and
> our dismissed rate is just little over 20% (222 out of 1054 = 21%).
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Coverity Scan

2013-07-26 Thread Brett Cannon 
On Fri, Jul 26, 2013 at 8:56 AM, Eli Bendersky  wrote:

> Just a quick question - is there a chance to convince Coverity to detect
> Python refcounting leaks in C API code :-) ?
>

You can always ask. =)


> This could be useful not only for Python but for extensions too. As it
> stands now, Coverity's leak detection is Python must be pretty weak because
> almost everything is done via PyObject refcounts.
>

Just an FYI (mostly for others  since I think Eli was at PyCon in the
relevant talk), David Malcolm has his work with gcc and refleak detection.
But yes, it would be nice if it was in Coverity as it would then be part of
the daily check.

-Brett


>
> Eli
>
>
> On Thu, Jul 25, 2013 at 11:48 AM, Christian Heimes 
> wrote:
>
>> Hello,
>>
>> this is an update on my work and the current status of Coverity Scan.
>>
>> Maybe you have noticed a checkins made be me that end with the line "CID
>> #". These are checkins that fix an issue that was discovered by the
>> static code analyzer Coverity. Coverity is a commercial product but it's
>> a free service for some Open Source projects. Python has been analyzed
>> by Coverity since about 2007. Guido, Neal, Brett, Stefan and some other
>> developers have used Coverity before I took over. I fixed a couple of
>> issues before 3.3 reached the RC phase and more bugs in the last couple
>> of months.
>>
>> Coverity is really great and its web GUI is fun to use, too. I was able
>> to identify and fix resource leaks, NULL pointer issues, buffer
>> overflows and missing checks all over the place. Because it's a static
>> analyzer that follows data-flows and control-flows the tool can detect
>> issues in error paths that are hardly visited at all. I have started to
>> document Coverity here:
>>
>>   http://docs.python.org/devguide/coverity.html
>>
>>
>> Interview
>> -
>>
>> A week ago I was contacted by Coverity. They have started a series of
>> articles and press releases about Open Source projects that use their
>> free service Coverity Scan, see
>>
>>
>>
>> http://www.coverity.com/company/press-releases/read/coverity-introduces-monthly-spotlight-series-for-coverity-scan-open-source-projects
>>
>> Two days ago I had a lovely phone interview about my involvement in the
>> Python project and our development style. They are going to release a
>> nice article in a couple of weeks. In the mean time we have time to fix
>> the remaining couple issues. We *might* be able to reach the highest
>> coverity integrity level! I have dealt with all major issues so we just
>> have to fix a couple of issues.
>>
>>
>> Current stats
>> -
>>
>> Lines of Code:  396,179
>> Defect Density: 0.05
>> Total defects:  1,054
>> Outstanding:   21 (Coverity Connect shows less)
>> Dismissed:222
>> Fixed:811
>>
>> http://i.imgur.com/NoELjcj.jpg
>> http://i.imgur.com/eJSzTUX.jpg
>>
>>
>> open issues
>> ---
>>
>> http://bugs.python.org/issue17899
>> http://bugs.python.org/issue18556
>> http://bugs.python.org/issue18555
>> http://bugs.python.org/issue18552
>> http://bugs.python.org/issue18551
>> http://bugs.python.org/issue18550
>> http://bugs.python.org/issue18528
>>
>>
>> Christian
>>
>> ___
>> Python-Dev mailing list
>> [email protected]
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/eliben%40gmail.com
>>
>
>
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Coverity Scan

2013-07-26 Thread Antoine Pitrou 
Le Fri, 26 Jul 2013 16:29:59 +0200,
Christian Heimes  a écrit :
> Coverity is able to detect some cases of refcount leaks. I don't know
> if the software is able to keep track of all reference counts. But it
> understands missing Py_DECREF() in error branches.
> 
> For example:
> 
> PyObject *n = PyLong_FromLong(0);
> PyObject *u = PyUnicode_FromString("example");
> 
> if (u == NULL) {
> return NULL;
> /* Coverity detects that 'n' leaks memory */
> }

But 'n' doesn't leak memory since PyLong_FromLong(0) is statically
allocated ;-)

More generally, in similar cases (e.g. replace "0" with a non-small
integer), you don't need any knowledge of reference counts to infer
that there is a memory leak. When the code discards the only existing
pointer to a heap-allocated memory area, there's a leak.

What we call "refcount leaks" is generally when an area is still
pointer-accessible, but failure to decrement the reference count
appropriately means it will never be released.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Coverity Scan

2013-07-26 Thread Eli Bendersky 
On Fri, Jul 26, 2013 at 7:29 AM, Christian Heimes wrote:

> Am 26.07.2013 14:56, schrieb Eli Bendersky:
> > Just a quick question - is there a chance to convince Coverity to detect
> > Python refcounting leaks in C API code :-) ? This could be useful not
> > only for Python but for extensions too. As it stands now, Coverity's
> > leak detection is Python must be pretty weak because almost everything
> > is done via PyObject refcounts.
>
> Coverity is able to detect some cases of refcount leaks. I don't know if
> the software is able to keep track of all reference counts. But it
> understands missing Py_DECREF() in error branches.
>
> For example:
>
> PyObject *n = PyLong_FromLong(0);
> PyObject *u = PyUnicode_FromString("example");
>
> if (u == NULL) {
> return NULL;
> /* Coverity detects that 'n' leaks memory */
> }
>

Interesting.

I was thinking of something more general though. Especially if we can mark
function arguments and return values as stealing references / creating new
ones / etc, many many common refcount bugs can be detected with static
analysis. This is definitely research-y, probably too much for our current
stage of relationship with Coverity :)

Eli
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Summary of Python tracker Issues

2013-07-26 Thread Python tracker 

ACTIVITY SUMMARY (2013-07-19 - 2013-07-26)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open4114 (+22)
  closed 26220 (+35)
  total  30334 (+57)

Open issues with patches: 1856 


Issues opened (37)
==

#18505: Duplicate function names in test_email.py
http://bugs.python.org/issue18505  opened by vajrasky

#18507: import_init() should not use Py_FatalError() but return an err
http://bugs.python.org/issue18507  opened by haypo

#18509: CJK decoders should return MBERR_EXCEPTION on PyUnicodeWriter 
http://bugs.python.org/issue18509  opened by haypo

#18512: sys.stdout.write does not allow bytes in Python 3.x
http://bugs.python.org/issue18512  opened by ntrrgc

#18515: zipfile._ZipDecryptor generates wasteful crc32 table on import
http://bugs.python.org/issue18515  opened by dholth

#18516: Typos in Lib/email/generator.py and Lib/email/architecture.rst
http://bugs.python.org/issue18516  opened by vajrasky

#18517: "xxlimited" extension declared incorrectly in setup.py
http://bugs.python.org/issue18517  opened by ned.deily

#18518: return-ing within code timed with timeit.timeit causes wrong r
http://bugs.python.org/issue18518  opened by icedream91

#18519: test_sqlite crashes on OS X tiger 3.x buildbot
http://bugs.python.org/issue18519  opened by ned.deily

#18521: [cppcheck] Full report
http://bugs.python.org/issue18521  opened by serval2412

#18526: Add resource management/guarding to unittest
http://bugs.python.org/issue18526  opened by zach.ware

#18527: Upgrade Modules/zlib to 1.2.8
http://bugs.python.org/issue18527  opened by christian.heimes

#18528: Possible fd leak in socketmodule
http://bugs.python.org/issue18528  opened by christian.heimes

#18529: Use long dash
http://bugs.python.org/issue18529  opened by serhiy.storchaka

#18530: posixpath.ismount performs extra lstat calls
http://bugs.python.org/issue18530  opened by alex

#18531: Undocumented different between METH_KEYWORDS and **kws
http://bugs.python.org/issue18531  opened by barry

#18532: hashlib.HASH objects should officially expose the hash name
http://bugs.python.org/issue18532  opened by jason.coombs

#18533: Avoid error from repr() of recursive dictview
http://bugs.python.org/issue18533  opened by bennorth

#18534: State clearly that open() 'file' param is "name" attr of the r
http://bugs.python.org/issue18534  opened by ncoghlan

#18535: termios.tcgetattr should return namedtuple
http://bugs.python.org/issue18535  opened by techtonik

#18538: `python -m dis ` relying on argparse
http://bugs.python.org/issue18538  opened by maker

#18539: Arguments tooltip wrong if def contains fractional default val
http://bugs.python.org/issue18539  opened by ariel_bruner

#18540: imaplib.IMAP4() ends with "Name or service not known" on Fedor
http://bugs.python.org/issue18540  opened by sYnfo

#18543: urllib.parse.urlopen shouldn't ignore installed opener when ca
http://bugs.python.org/issue18543  opened by moritzs

#18544: subprocess.Popen support for redirection of arbitrary file des
http://bugs.python.org/issue18544  opened by ropoctorix

#18546: ssl.get_server_certificate like addition for cert chain
http://bugs.python.org/issue18546  opened by underrun

#18548: In unittest doc change WidgetTestCase to SimpleWidgetTestCase 
http://bugs.python.org/issue18548  opened by py.user

#18550: internal_setblocking() doesn't check return value of fcntl()
http://bugs.python.org/issue18550  opened by christian.heimes

#18551: child_exec() doesn't check return value of fcntl()
http://bugs.python.org/issue18551  opened by christian.heimes

#18552: obj2ast_object() doesn't check return value of PyArena_AddPyOb
http://bugs.python.org/issue18552  opened by christian.heimes

#18553: os.isatty() is not Unix only
http://bugs.python.org/issue18553  opened by techtonik

#18554: os.__all__ is incomplete
http://bugs.python.org/issue18554  opened by ronaldoussoren

#18555: type_set_bases() doesn't check return value of  PyArg_UnpackTu
http://bugs.python.org/issue18555  opened by christian.heimes

#18558: Iterable glossary entry needs clarification
http://bugs.python.org/issue18558  opened by Zero

#18559: _pickle: NULL ptr dereference when PyLong_FromSsize_t() fails
http://bugs.python.org/issue18559  opened by christian.heimes

#18560: builtin_sum() doesn't check return value of PyLong_FromLong()
http://bugs.python.org/issue18560  opened by christian.heimes

#18561: ctypes _build_callargs() doesn't check name for NULL
http://bugs.python.org/issue18561  opened by christian.heimes



Most recent 15 issues with no replies (15)
==

#18561: ctypes _build_callargs() doesn't check name for NULL
http://bugs.python.org/issue18561

#18560: builtin_sum() doesn't check return value of PyLong_FromLong()
http://bugs.python.org/issue18560

#18559: _pickle: NULL ptr dereference when PyLong_FromSs

Re: [Python-Dev] Daemon creation code in the standard library

2013-07-26 Thread Antoine Pitrou 
On Fri, 26 Jul 2013 09:38:10 +0200
Ronald Oussoren  wrote:
> 
> On 25 Jul, 2013, at 4:18, Ben Finney  wrote:
> 
> > Ben Finney  writes:
> > 
> >> Work continues on the PEP 3143-compatible ‘python-daemon’, porting it to
> >> Python 3 and aiming for inclusion in the standard library.
> 
> At first glance the library appears to close all open files, with an option
> to exclude some specific file descriptors (that is, you need to pass a list
> of files that shouldn't be closed). 

Indeed, it's annoying when you want to setup logging before
daemonization starts. I had to hack my way through logging handlers to
find the fd I had to keep open.

> That makes it a lot harder to do some initialization before daemonizing.
> I prefer to perform at least some initialization early in program startup to
> be able to give sensible error messages. I've had too many initscripts that
> claimed to have started a daemon sucessfully, only to have that daemon stop
> right away because it noticed a problem right after it detached itself.

Agreed.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Coverity Scan

2013-07-26 Thread Christian Heimes 
Am 26.07.2013 16:29, schrieb Brett Cannon:
> Have you tried asking for help from Coverity? They have been rather nice
> so far and they may be willing to just give us free help in getting the
> modeling file set up properly.

Yes, I'm in contact with Dakshesh. I was able to figure out one model
for a false positive on my own. Dakshesh is helping me with another.

Christian

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-07-26 Thread Victor Stinner 
2013/7/26 Antoine Pitrou :
>> The main drawback is the additionnal syscalls: on some platforms, 2
>> additional syscalls are need to make a file descriptor non-inheritable
>> for each creation of file descriptor. According to my benchmark on the
>> implementation of the PEP 433: the overhead of making a file
>> descriptor non-inheritable is between 1% and 3% (7.8 µs => 7.9 or 8.0
>> µs) on Linux 3.6.
>
> 1% and 3% of what?
> You're telling us there's a 0.1µs overhead. It's positively tiny.

Copy-paste of the link:

"""
On Linux, setting the close-on-flag has a low overhead on
performances. Results of bench_cloexec.py on Linux 3.6:

- close-on-flag not set: 7.8 us
- O_CLOEXEC: 1% slower (7.9 us)
- ioctl(): 3% slower (8.0 us)
- fcntl(): 3% slower (8.0 us)
"""

The overhead is between 0.1 and 0.2 µs (100 and 200 ns) according to
my micro-benchmark.

"python -c pass" takes 19,000 µs (0.019 sec) on my PC. It uses 207
syscalls creating file descriptors (open() and openat()): 67 are
successful, 140 are failing with ENOENT. The estimated overhead on
"python -c pass" is 0.2*67=13.4 µs (0.07%).

Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-07-26 Thread Antoine Pitrou 
On Fri, 26 Jul 2013 22:17:47 +0200
Victor Stinner  wrote:

> 2013/7/26 Antoine Pitrou :
> >> The main drawback is the additionnal syscalls: on some platforms, 2
> >> additional syscalls are need to make a file descriptor non-inheritable
> >> for each creation of file descriptor. According to my benchmark on the
> >> implementation of the PEP 433: the overhead of making a file
> >> descriptor non-inheritable is between 1% and 3% (7.8 µs => 7.9 or 8.0
> >> µs) on Linux 3.6.
> >
> > 1% and 3% of what?
> > You're telling us there's a 0.1µs overhead. It's positively tiny.
> 
> Copy-paste of the link:
> 
> """
> On Linux, setting the close-on-flag has a low overhead on
> performances. Results of bench_cloexec.py on Linux 3.6:
> 
> - close-on-flag not set: 7.8 us
> - O_CLOEXEC: 1% slower (7.9 us)
> - ioctl(): 3% slower (8.0 us)
> - fcntl(): 3% slower (8.0 us)
> """

You aren't answering my question: slower than what?
Benchmarking is useless if you aren't telling us what exactly you are
benchmarking.

> The overhead is between 0.1 and 0.2 µs (100 and 200 ns) according to
> my micro-benchmark.

That's what I figured out (see above). It's tiny.

Antoine.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-07-26 Thread Guido van Rossum 
On Fri, Jul 26, 2013 at 5:08 AM, Victor Stinner
 wrote:
> After having written 2 PEP on the topic, I slowly agree that make all
> file descriptors non-inheritable is the best *compromise*. It solves
> most, or all, issues.

Right.

> The main drawback is the additionnal syscalls: on some platforms, 2
> additional syscalls are need to make a file descriptor non-inheritable
> for each creation of file descriptor. According to my benchmark on the
> implementation of the PEP 433: the overhead of making a file
> descriptor non-inheritable is between 1% and 3% (7.8 µs => 7.9 or 8.0
> µs) on Linux 3.6.
> http://www.python.org/dev/peps/pep-0433/#performances

Remember that this is going to be Python 3.4 and newer. AFAICT setting
O_CLOEXEC on open works on OSX (at least the man page on OSX 10.8 has
it), on newer Linuxes, and the equivalent on Windows. So even if it
does cost an extra syscall on older systems, those systems will be
obsolete before Python 3.4 becomes mainstream there. And it does look
like the syscalls are pretty cheap. I also don't think I'm not
particularly worried about the cost of syscalls for making a socket
(non)blocking -- although we should probably avoid the second fcntl()
call if the first call shows the flag is already set the way we want
it.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-07-26 Thread Victor Stinner 
2013/7/26 Antoine Pitrou :
> On Fri, 26 Jul 2013 22:17:47 +0200
>> """
>> On Linux, setting the close-on-flag has a low overhead on
>> performances. Results of bench_cloexec.py on Linux 3.6:
>>
>> - close-on-flag not set: 7.8 us
>> - O_CLOEXEC: 1% slower (7.9 us)
>> - ioctl(): 3% slower (8.0 us)
>> - fcntl(): 3% slower (8.0 us)
>> """
>
> You aren't answering my question: slower than what?

Ah, you didn't understand the labels. bench_cloexec.py runs a
benchmark on os.open(path, os.O_RDONLY, cloexec=False) and
os.open(path, os.O_RDONLY, cloexec=True) with different implementation
of making the file descriptor non-inheritable.

close-on-flag not set: 7.8 us
=> C code: open(path, O_RDONLY)

O_CLOEXEC: 1% slower (7.9 us)
=> C code: open(path, O_RDONLY|CLOEXEC)
=> 1% slower than open(path, O_RDONLY)

ioctl(): 3% slower (8.0 us)
=> C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0)
=> 3% slower than open(path, O_RDONLY)

fcntl(): 3% slower (8.0 us)
=> C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD);
fcntl(fd, F_SETFD, flags | FD_CLOEXEC)
=> 3% slower than open(path, O_RDONLY)

Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-07-26 Thread Antoine Pitrou 
On Sat, 27 Jul 2013 00:18:40 +0200
Victor Stinner  wrote:
> 2013/7/26 Antoine Pitrou :
> > On Fri, 26 Jul 2013 22:17:47 +0200
> >> """
> >> On Linux, setting the close-on-flag has a low overhead on
> >> performances. Results of bench_cloexec.py on Linux 3.6:
> >>
> >> - close-on-flag not set: 7.8 us
> >> - O_CLOEXEC: 1% slower (7.9 us)
> >> - ioctl(): 3% slower (8.0 us)
> >> - fcntl(): 3% slower (8.0 us)
> >> """
> >
> > You aren't answering my question: slower than what?
> 
> Ah, you didn't understand the labels. bench_cloexec.py runs a
> benchmark on os.open(path, os.O_RDONLY, cloexec=False) and
> os.open(path, os.O_RDONLY, cloexec=True) with different implementation
> of making the file descriptor non-inheritable.
> 
> close-on-flag not set: 7.8 us
> => C code: open(path, O_RDONLY)
> 
> O_CLOEXEC: 1% slower (7.9 us)
> => C code: open(path, O_RDONLY|CLOEXEC)
> => 1% slower than open(path, O_RDONLY)
> 
> ioctl(): 3% slower (8.0 us)
> => C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0)
> => 3% slower than open(path, O_RDONLY)
> 
> fcntl(): 3% slower (8.0 us)
> => C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD);
> fcntl(fd, F_SETFD, flags | FD_CLOEXEC)
> => 3% slower than open(path, O_RDONLY)

Ok, so I think this it is a totally reasonable compromise.

People who bother about a 3% slowdown when calling os.open() can
optimize the hell out of their code using Cython for all I care :-)

Regards

Antoine.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-07-26 Thread Gregory P. Smith 
On Fri, Jul 26, 2013 at 3:23 PM, Antoine Pitrou  wrote:

> On Sat, 27 Jul 2013 00:18:40 +0200
> Victor Stinner  wrote:
> > 2013/7/26 Antoine Pitrou :
> > > On Fri, 26 Jul 2013 22:17:47 +0200
> > >> """
> > >> On Linux, setting the close-on-flag has a low overhead on
> > >> performances. Results of bench_cloexec.py on Linux 3.6:
> > >>
> > >> - close-on-flag not set: 7.8 us
> > >> - O_CLOEXEC: 1% slower (7.9 us)
> > >> - ioctl(): 3% slower (8.0 us)
> > >> - fcntl(): 3% slower (8.0 us)
> > >> """
> > >
> > > You aren't answering my question: slower than what?
> >
> > Ah, you didn't understand the labels. bench_cloexec.py runs a
> > benchmark on os.open(path, os.O_RDONLY, cloexec=False) and
> > os.open(path, os.O_RDONLY, cloexec=True) with different implementation
> > of making the file descriptor non-inheritable.
> >
> > close-on-flag not set: 7.8 us
> > => C code: open(path, O_RDONLY)
> >
> > O_CLOEXEC: 1% slower (7.9 us)
> > => C code: open(path, O_RDONLY|CLOEXEC)
> > => 1% slower than open(path, O_RDONLY)
> >
> > ioctl(): 3% slower (8.0 us)
> > => C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0)
> > => 3% slower than open(path, O_RDONLY)
> >
> > fcntl(): 3% slower (8.0 us)
> > => C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD);
> > fcntl(fd, F_SETFD, flags | FD_CLOEXEC)
> > => 3% slower than open(path, O_RDONLY)
>
> Ok, so I think this it is a totally reasonable compromise.
>
> People who bother about a 3% slowdown when calling os.open() can
> optimize the hell out of their code using Cython for all I care :-)
>
>
+1  ;)

and +1 for making the sane default of noinherit / cloexec /
whatever-others-call-it by default for all fds/handles ever opened by
Python. It stops ignoring the issue (ie: the status quo of matching the
default behavior of C as defined in the 1970s)... That is a GOOD thing. :)

-gps
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-07-26 Thread Guido van Rossum 
On Fri, Jul 26, 2013 at 9:26 PM, Gregory P. Smith  wrote:
>
> On Fri, Jul 26, 2013 at 3:23 PM, Antoine Pitrou  wrote:
>>
>> On Sat, 27 Jul 2013 00:18:40 +0200
>> Victor Stinner  wrote:
>> > 2013/7/26 Antoine Pitrou :
>> > > On Fri, 26 Jul 2013 22:17:47 +0200
>> > >> """
>> > >> On Linux, setting the close-on-flag has a low overhead on
>> > >> performances. Results of bench_cloexec.py on Linux 3.6:
>> > >>
>> > >> - close-on-flag not set: 7.8 us
>> > >> - O_CLOEXEC: 1% slower (7.9 us)
>> > >> - ioctl(): 3% slower (8.0 us)
>> > >> - fcntl(): 3% slower (8.0 us)
>> > >> """
>> > >
>> > > You aren't answering my question: slower than what?
>> >
>> > Ah, you didn't understand the labels. bench_cloexec.py runs a
>> > benchmark on os.open(path, os.O_RDONLY, cloexec=False) and
>> > os.open(path, os.O_RDONLY, cloexec=True) with different implementation
>> > of making the file descriptor non-inheritable.
>> >
>> > close-on-flag not set: 7.8 us
>> > => C code: open(path, O_RDONLY)
>> >
>> > O_CLOEXEC: 1% slower (7.9 us)
>> > => C code: open(path, O_RDONLY|CLOEXEC)
>> > => 1% slower than open(path, O_RDONLY)
>> >
>> > ioctl(): 3% slower (8.0 us)
>> > => C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0)
>> > => 3% slower than open(path, O_RDONLY)
>> >
>> > fcntl(): 3% slower (8.0 us)
>> > => C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD);
>> > fcntl(fd, F_SETFD, flags | FD_CLOEXEC)
>> > => 3% slower than open(path, O_RDONLY)
>>
>> Ok, so I think this it is a totally reasonable compromise.
>>
>> People who bother about a 3% slowdown when calling os.open() can
>> optimize the hell out of their code using Cython for all I care :-)
>>
>
> +1  ;)
>
> and +1 for making the sane default of noinherit / cloexec /
> whatever-others-call-it by default for all fds/handles ever opened by
> Python. It stops ignoring the issue (ie: the status quo of matching the
> default behavior of C as defined in the 1970s)... That is a GOOD thing. :)

Do we even need a new PEP, or should we just do it? Or can we adapt
Victor's PEP 446?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-07-26 Thread Nick Coghlan 
On 27 July 2013 14:36, Guido van Rossum  wrote:
> On Fri, Jul 26, 2013 at 9:26 PM, Gregory P. Smith  wrote:
>>
>> On Fri, Jul 26, 2013 at 3:23 PM, Antoine Pitrou  wrote:
>>>
>>> On Sat, 27 Jul 2013 00:18:40 +0200
>>> Victor Stinner  wrote:
>>> > 2013/7/26 Antoine Pitrou :
>>> > > On Fri, 26 Jul 2013 22:17:47 +0200
>>> > >> """
>>> > >> On Linux, setting the close-on-flag has a low overhead on
>>> > >> performances. Results of bench_cloexec.py on Linux 3.6:
>>> > >>
>>> > >> - close-on-flag not set: 7.8 us
>>> > >> - O_CLOEXEC: 1% slower (7.9 us)
>>> > >> - ioctl(): 3% slower (8.0 us)
>>> > >> - fcntl(): 3% slower (8.0 us)
>>> > >> """
>>> > >
>>> > > You aren't answering my question: slower than what?
>>> >
>>> > Ah, you didn't understand the labels. bench_cloexec.py runs a
>>> > benchmark on os.open(path, os.O_RDONLY, cloexec=False) and
>>> > os.open(path, os.O_RDONLY, cloexec=True) with different implementation
>>> > of making the file descriptor non-inheritable.
>>> >
>>> > close-on-flag not set: 7.8 us
>>> > => C code: open(path, O_RDONLY)
>>> >
>>> > O_CLOEXEC: 1% slower (7.9 us)
>>> > => C code: open(path, O_RDONLY|CLOEXEC)
>>> > => 1% slower than open(path, O_RDONLY)
>>> >
>>> > ioctl(): 3% slower (8.0 us)
>>> > => C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0)
>>> > => 3% slower than open(path, O_RDONLY)
>>> >
>>> > fcntl(): 3% slower (8.0 us)
>>> > => C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD);
>>> > fcntl(fd, F_SETFD, flags | FD_CLOEXEC)
>>> > => 3% slower than open(path, O_RDONLY)
>>>
>>> Ok, so I think this it is a totally reasonable compromise.
>>>
>>> People who bother about a 3% slowdown when calling os.open() can
>>> optimize the hell out of their code using Cython for all I care :-)
>>>
>>
>> +1  ;)
>>
>> and +1 for making the sane default of noinherit / cloexec /
>> whatever-others-call-it by default for all fds/handles ever opened by
>> Python. It stops ignoring the issue (ie: the status quo of matching the
>> default behavior of C as defined in the 1970s)... That is a GOOD thing. :)
>
> Do we even need a new PEP, or should we just do it? Or can we adapt
> Victor's PEP 446?

Adapting the PEP sounds good - while I agree with switching to a sane
default, I think the daemonisation thread suggests there may need to
be a supporting API to help force FDs created by nominated logging
handlers to be inherited.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com