[Python-Dev] Strange behavior in Python 2.5a0 (trunk) --- possible error in AST?

[Travis E. Oliphant]

I'm seeing strange behavior in the Python 2.5a0 trunk that is causing 
the tests for numpy to fail.  Apparently obj[...] = 1 is not calling 
PyObject_SetItem

Here is a minimal example to show the error.  Does anyone else see this?

class temp(object):
def __setitem__(self, obj, v):
print obj, v

mine = temp()
mine[...] = 1
mine[0] = 1


Output from Python 2.4.2:
Ellipsis 1
0 1


Output from Python 2.5a0:
0 1


In other words, the first line does nothing in Python 2.5a0.


Does anybody else see this?

Thanks,

-Travis

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Strange behavior in Python 2.5a0 (trunk) --- possible error in AST?

[Michael Hudson]

"Travis E. Oliphant" <[EMAIL PROTECTED]> writes:

> I'm seeing strange behavior in the Python 2.5a0 trunk that is causing 
> the tests for numpy to fail.  Apparently obj[...] = 1 is not calling 
> PyObject_SetItem
>
> Here is a minimal example to show the error.  Does anyone else see this?
>
> class temp(object):
> def __setitem__(self, obj, v):
> print obj, v
>
> mine = temp()
> mine[...] = 1

It's a compiler problem:

>>> dis.dis(compile("mine[...] = 1", '', 'single'))
  1   0 LOAD_CONST   0 (1)
  3 LOAD_NAME0 (mine)
  6 LOAD_CONST   1 (Ellipsis)
  9 LOAD_CONST   2 (None)
 12 RETURN_VALUE

Cheers,
mwh

-- 
6. The code definitely is not portable - it will produce incorrect 
   results if run from the surface of Mars.
   -- James Bonfield, http://www.ioccc.org/2000/rince.hint
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Strange behavior in Python 2.5a0 (trunk) --- possible error in AST?

[Nick Coghlan]

Michael Hudson wrote:
> "Travis E. Oliphant" <[EMAIL PROTECTED]> writes:
> 
>> I'm seeing strange behavior in the Python 2.5a0 trunk that is causing 
>> the tests for numpy to fail.  Apparently obj[...] = 1 is not calling 
>> PyObject_SetItem
>>
>> Here is a minimal example to show the error.  Does anyone else see this?
>>
>> class temp(object):
>> def __setitem__(self, obj, v):
>> print obj, v
>>
>> mine = temp()
>> mine[...] = 1
> 
> It's a compiler problem:
> 
 dis.dis(compile("mine[...] = 1", '', 'single'))
>   1   0 LOAD_CONST   0 (1)
>   3 LOAD_NAME0 (mine)
>   6 LOAD_CONST   1 (Ellipsis)
>   9 LOAD_CONST   2 (None)
>  12 RETURN_VALUE

And how...

   case Ellipsis_kind:
 ADDOP_O(c, LOAD_CONST, Py_Ellipsis, consts)
 break;

Just a couple of minor details missing, like, oh, compiling the actual 
subscript operation :)

Bug here: http://www.python.org/sf/1448804

(assigned to myself, since I already wrote a test for it and worked out where 
to fix it)

Cheers,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
 http://www.boredomandlaziness.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Coverity Open Source Defect Scan of Python

[Michael Chermside]

Dennis Allison writes:
> I'd also encourage Coventry to explain their business model a bit more
> clearly.

Ben Chelf writes:
> Of course it's no surprise that I see open
> source projects everywhere -- as part of infrastructure or part of code
> bases that people are developing. So from a Coverity perspective,
> clearly we want to provide source code analysis for the projects that
> our customers care about
   [...]
> I really just want every developer to use source code analysis
> while they write code
   [...]
> We got a lot of the
> good publicity in the research lab because there existed this big open
> source OS that we could test our theories on.
   [...]
> I think it makes sense for Coverity to have a strong relationship with
> the open source community since that community has been helping us
> pretty much since day 1


I work for a business... and we pay full price for the tools that we
use in our work. I am aware of and follow the work that I and my
colleagues do -- when someone has a good idea, I tend to learn from
that and introduce the same idea in future projects. I also am aware
of and follow the work of quite a few open source projects (to
different degrees depending on the project). In fact, I see far more
open source projects than I do other projects. I learn a lot of good
ideas from these projects also, and I use them in my paid work. For
example, it was my experience with open source projects that convinced
me that extensive unit tests that are expected to always pass was a
feasable and useful idea. Many of the ways that open source projects
are managed are also beginning to work their way into my professional
life also. It's just that I see far more open source projects than
others, and frankly the open source projects are almost always better
run, with higher standards.

I doubt I'm the only one... I think open source will be leading the
way in software development standards for some time now. So I think
offering a software development tool FREE to open source projects in
hopes of selling it for money to comercial projects is a WONDERFUL
business model. Good luck!

-- Michael Chermside

(PS: too bad I can't buy stock in Coverity. How come all the GOOD
companies are private? I had to wait around 6 years before I could
buy stock in Google.)

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] About "Coverity Study Ranks LAMP Code Quality"

[fermigier]

http://www.internetnews.com/dev-news/article.php/3589361

"Perl had a defect density of only 0.186. In comparison Python had a
defect density of 0.372 and PHP was actually above both the baseline and
LAMP averages at 0.474."

This is of course a PR stunt. But I'm wondering if the actual "bugs"
list was transmitted to Python developers, and verified / acted upon.

Because according to
http://www.washingtontechnology.com/news/1_1/daily_news/28134-1.html :

"The maintainers of the source codes can register with Coverity to see
the full results. (End users cannot see the bug lists themselves; they
will be able to see how buggy a particular program may be.)"

Regards,

  S.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About "Coverity Study Ranks LAMP Code Quality"

[Fredrik Lundh]

"fermigier" <[EMAIL PROTECTED]> wrote:

> "Perl had a defect density of only 0.186. In comparison Python had a
> defect density of 0.372 and PHP was actually above both the baseline and
> LAMP averages at 0.474."
>
> This is of course a PR stunt. But I'm wondering if the actual "bugs"
> list was transmitted to Python developers, and verified / acted upon.

according to

http://scan.coverity.com

perl is currently at 0.151 and Python 2.5 trunk is at 0.055.  looks like the
perlheads decided to write press releases rather than fixing their core...

(given how Coverity works, the "raw" unreviewed defect density figure is
relatively useless, and anyone using that for marketing is a complete idiot.
I've no time to read the article, but I do hope it's not some Coverity guy
you've quoted...)

(also note that most true Python bugs found by Coverity are of the "may
leak or misbehave under extreme pressure, in the unlikely event that you
get here under such circumstances...")





___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About "Coverity Study Ranks LAMP Code Quality"

[Fredrik Lundh]

"fermigier" wrote:

> But I'm wondering if the actual "bugs" list was transmitted to Python 
> developers,
> and verified / acted upon.

and in case it wasn't clear from my previous post, the answer to
your specific question is "yes" ;-)





___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Strange behavior in Python 2.5a0 (trunk) --- possible error in AST?

[Travis E. Oliphant]

Nick Coghlan wrote:
> 
> And how...
> 
>case Ellipsis_kind:
>  ADDOP_O(c, LOAD_CONST, Py_Ellipsis, consts)
>  break;
> 
> Just a couple of minor details missing, like, oh, compiling the actual 
> subscript operation :)
> 
> Bug here: http://www.python.org/sf/1448804
> 
> (assigned to myself, since I already wrote a test for it and worked out where 
> to fix it)

Fabulous!  The fix committed to SVN seems to work.

Now, all of numpy's unit tests are passing with Python 2.5a0.

Great work, thank you.

-Travis

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Coverity Open Source Defect Scan of Python

[Scott David Daniels]

Michael Chermside wrote:
> (PS: too bad I can't buy stock in Coverity. How come all the GOOD
> companies are private? I had to wait around 6 years before I could
> buy stock in Google.)

Maybe because the companies whose stock is available early are companies
bent on producing stock profits, rather than a solid value proposition.
Trying to satisfy the profit-lust of angels has redirected more than one
company.

--Scott David Daniels
[EMAIL PROTECTED]

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Still looking for volunteer to run Windows buildbot

[Trent Mick]

[Tim Peters wrote]
> This touches on something we (including Martin) should think about: 
> it's very painful to build a full Python on Windows because of these
> external packages...

Yup. That is part of what I meant by updating PCBuild\readme.txt below:
to improve the instructions so the build-dance for all these external
packages is a little more cut n' paste.

> Something we might be able to do instead is have just one person per
> external project endure the pain of building it, and then have them
> check in the whole post-compilation post-test project directory tree. 
> Everyone else (presumably including Windows buildbot slaves) using the
> same compiler could then just check out the result.

If people don't mind having prebuilt binaries checked in, yes I suppose
we could do this. For some projects here at work we have a "prebuilt"
dir at the root of the source tree:

prebuilt/
win32-x86/
foo/# Windows prebuilt bits for package foo
linux-x86/
foo/
solaris8-sparc/
foo/
...


Whoa, someone is way ahead of me here. :)


> What's worse is when checkins cause both trunk and branch rebuilds. 
> That can bring my zippy box to its knees, particularly when two of the
> disk-intensive tests (like test_largefile) run at the same time.

That would require some kind of Scheduler work in the build master
config that knows how to queue up scheduled builds from multiple sources
to only allow one per machine at a time. I'd find this very useful for
other projects at work and I'm sure other buildbot users would as well.
Maybe buildbot may already have something like this?


> > Remaining TODOs:
> > - make sure the python24-maint branch one works
> 
> Works for me, although (as the Wiki page says) you also need to copy
> zlib-1.2.3 into the branch build area.

Yup. Looks like my build worked. Another TODO now though:

- Figure out why usage of:
winsound.PlaySound(, winsound.SND_ALIAS)
  fails on my Win2k box. This is why the test suite fails on that box.


> > - get the build slaves running as a Windows service
> 
> Mark Hammond has a patch to the buildbot project toward this end:
> 
> 

Cool. I'll look into that.


Trent

-- 
Trent Mick
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Still looking for volunteer to run Windows buildbot

[Thomas Heller]

Trent Mick wrote:

> Yup. Looks like my build worked. Another TODO now though:
> 
> - Figure out why usage of:
> winsound.PlaySound(, winsound.SND_ALIAS)
>   fails on my Win2k box. This is why the test suite fails on that box.
> 

Doesn't that always fail when there is no soundcard in the machine?

Thomas

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Still looking for volunteer to run Windows buildbot

[Trent Mick]

[Thomas Heller wrote]
> Trent Mick wrote:
> 
> > Yup. Looks like my build worked. Another TODO now though:
> > 
> > - Figure out why usage of:
> > winsound.PlaySound(, winsound.SND_ALIAS)
> >   fails on my Win2k box. This is why the test suite fails on that box.
> > 
> 
> Doesn't that always fail when there is no soundcard in the machine?

I don't know. The *other* winsound tests work and I *can* hear that
rising tone goop coming out of the box's tinny speakers when the test
suite is run.

http://www.python.org/dev/buildbot/all/x86%20W2k%202.4/builds/5/step-test/0


http://msdn.microsoft.com/library/default.asp?url=/library/en-us/multimed/htm/_win32_playsound.asp
PlaySound
...
SND_ALIAS   The pszSound parameter is a system-event alias in the
registry or the WIN.INI file.
...

These aliases are apparently supposed to be in the registry or WIN.INI,
but the docs don't say where. Perhaps my Win2k box is missing some
registry entries. No, I think it is the config of the box -- read on.

I do have a sound card in that box, however, the "Sounds and Multimedia
Properties" dialog (off Control Panel) says that there are "No Playback
Devices" for Sound Playback. So I guess that is it. Maybe the sound card
in that box is not hooked up. Grrr. I certainly don't care about the
sound card for that box but I don't want the test suite to keep
reporting a spurious failure.

Trent

-- 
Trent Mick
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] decorator module patch

[Jim Jewett]

Raymond Hettinger wrote:

> In contrast, a name like "functional" suggests that
> some of these tools don't quite fit.

The original intent was that the functional module
become the home of typical utilities for functional style
programming.  partial was there were patches (such
as 1412451) providing quite a few additional tools and
improved documentation on the more functional builtins
(such as map and reduce).

That particular patch was closed to spend time as
an independent module first, but there is a fair
chance that parts of it will eventually be added.
(OTOH, it might be just a few parts which fit well
under functools, rather tha just "functional")

-jJ
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Making builtins more efficient

[Jim Jewett]

Paul Moore wrote:

> Is there any practical way of detecting and flagging
> constructs like the above (remotely shadowing a
> builtin in another module)?

Phillip J. Eby wrote:
> the patch ended up being backed out ... too strict
> of a check to be accepted for Python 2.4.

http://svn.python.org/view/python/trunk/Objects/moduleobject.c

It was revision 33054, backed out in 33084.

The patch warned about any shadowing of builtins, which
probably is too strict.  An alternative might be to warn
only about modifying *another* module's globals.  (And
perhaps not just when they shadow builtins?)

For example, modules could grow a __sealed__ attribute
which gets set at the end of the import; instead of using
PyObject_GenericSetAttr directly, the tp_setattro slot
would check the __sealed__ attribute (and maybe squawk)
before deferring.

-jJ
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making builtins more efficient

[Samuele Pedroni]

Phillip J. Eby wrote:
> At 02:47 PM 3/13/2006 -0500, Jim Jewett wrote:
> 
>>Paul Moore wrote:
>>
>>
>>>Is there any practical way of detecting and flagging
>>>constructs like the above (remotely shadowing a
>>>builtin in another module)?
>>
>>Phillip J. Eby wrote:
>>
>>>the patch ended up being backed out ... too strict
>>>of a check to be accepted for Python 2.4.
>>
>>http://svn.python.org/view/python/trunk/Objects/moduleobject.c
>>
>>It was revision 33054, backed out in 33084.
>>
>>The patch warned about any shadowing of builtins, which
>>probably is too strict.
> 
> 
> I'm curious: what actually happened?  What things were causing warnings?

http://mail.python.org/pipermail/python-dev/2003-July/036879.html


> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/pedronis%40strakt.com

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Why are so many built-in types inheritable?

[Fabiano Sidler]

Hi folks!

Let me explain the above question:
For debugging purpose I tried this:

--- snip ---
def foo(): pass
function = type(foo)

class PrintingFunction(function):
  def __init__(self, func):
self.func = func
  def __call__(self, *args, **kwargs):
print args, kwargs
return function.__call__(self, args, kwargs)

class DebugMeta(type):
  def __new__(self, name, bases, dict):
for name in dict:
  if type(dict[name]) is function:
dict[name] = PrintingFunction(dict[name])

--- snap ---

Now I tought I were able to let all methods of classes with DebugMeta
as metaclass print out their arguments.  But I got the following sad
error:

TypeError: Error when calling the metaclass bases
type 'function' is not an acceptable base type

That's a pity, isn't it?
What could I do to get the above code to work? (No, I don't want to
reimplement  without this unpleasant behaviour in
Python.

Greetings,
F. Sidler
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making builtins more efficient

[Phillip J. Eby]

At 02:47 PM 3/13/2006 -0500, Jim Jewett wrote:
>Paul Moore wrote:
>
> > Is there any practical way of detecting and flagging
> > constructs like the above (remotely shadowing a
> > builtin in another module)?
>
>Phillip J. Eby wrote:
> > the patch ended up being backed out ... too strict
> > of a check to be accepted for Python 2.4.
>
>http://svn.python.org/view/python/trunk/Objects/moduleobject.c
>
>It was revision 33054, backed out in 33084.
>
>The patch warned about any shadowing of builtins, which
>probably is too strict.

I'm curious: what actually happened?  What things were causing warnings?

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Strange behavior in Python 2.5a0 (trunk) --- possible error in AST?

[Nick Coghlan]

Travis E. Oliphant wrote:
> Nick Coghlan wrote:
>> And how...
>>
>>case Ellipsis_kind:
>>  ADDOP_O(c, LOAD_CONST, Py_Ellipsis, consts)
>>  break;
>>
>> Just a couple of minor details missing, like, oh, compiling the actual 
>> subscript operation :)
>>
>> Bug here: http://www.python.org/sf/1448804
>>
>> (assigned to myself, since I already wrote a test for it and worked out 
>> where 
>> to fix it)
> 
> Fabulous!  The fix committed to SVN seems to work.
> 
> Now, all of numpy's unit tests are passing with Python 2.5a0.
> 
> Great work, thank you.

Unfortunately my new test case breaks test_compiler. I didn't notice because I 
didn't use -uall before checking it in :(

If no-one else gets to it, I'll try to sort it out tonight.

Cheers,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
 http://www.boredomandlaziness.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Still looking for volunteer to run Windows buildbot

[Martin v. Löwis]

Trent Mick wrote:
>>This touches on something we (including Martin) should think about: 
>>it's very painful to build a full Python on Windows because of these
>>external packages...
> 
> 
> Yup. That is part of what I meant by updating PCBuild\readme.txt below:
> to improve the instructions so the build-dance for all these external
> packages is a little more cut n' paste.

I took an approach with a little more automation:
Tools/buildbot/external.bat gradually learns how to fetch and build
the necessary prerequisites; to avoid moving URLs, these come from
the external/ directory of the projects svn (in case of bsddb,
this already has the patches, and the VS project files converted).

> If people don't mind having prebuilt binaries checked in

Well, I do mind having prebuilt binaries checked in. They take up space
for little gain, plus they might increase the maintenance burden.

> That would require some kind of Scheduler work in the build master
> config that knows how to queue up scheduled builds from multiple sources
> to only allow one per machine at a time. I'd find this very useful for
> other projects at work and I'm sure other buildbot users would as well.
> Maybe buildbot may already have something like this?

I could not easily find it. There is the AnyBranchScheduler, but it
seems to serve a different purpose.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Still looking for volunteer to run Windows buildbot

[Trent Mick]

[Martin v. Loewis wrote]
> I took an approach with a little more automation:
> Tools/buildbot/external.bat gradually learns how to fetch and build
> the necessary prerequisites; to avoid moving URLs, these come from
> the external/ directory of the projects svn (in case of bsddb,
> this already has the patches, and the VS project files converted).
> 
> > If people don't mind having prebuilt binaries checked in
> 
> Well, I do mind having prebuilt binaries checked in. They take up space
> for little gain, plus they might increase the maintenance burden.

I like you approach better. Very nice.


Trent

-- 
Trent Mick
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Still looking for volunteer to run Windows buildbot

[Martin v. Löwis]

Trent Mick wrote:
> I do have a sound card in that box, however, the "Sounds and Multimedia
> Properties" dialog (off Control Panel) says that there are "No Playback
> Devices" for Sound Playback. So I guess that is it. Maybe the sound card
> in that box is not hooked up. Grrr. I certainly don't care about the
> sound card for that box but I don't want the test suite to keep
> reporting a spurious failure.

Now, if there was a reliable check whether a soundcard is present, that
check could be run as a prerequisite, then raising TestSkipped if no
soundcard is present.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About "Coverity Study Ranks LAMP Code Quality"

[Greg Ewing]

Fredrik Lundh wrote:

> > But I'm wondering if the actual "bugs" list was transmitted to Python 
> > developers,
> > and verified / acted upon.
> 
> and in case it wasn't clear from my previous post, the answer to
> your specific question is "yes" ;-)

Could whoever did this perhaps post a brief description
of what sort of information their tool produces, and
how good a hit rate they have of finding real problems?

Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About "Coverity Study Ranks LAMP Code Quality"

[Jeff Epler]

On Mon, Mar 13, 2006 at 03:05:55PM +, fermigier wrote:
> Because according to
> http://www.washingtontechnology.com/news/1_1/daily_news/28134-1.html :
> 
> "The maintainers of the source codes can register with Coverity to see
> the full results. (End users cannot see the bug lists themselves; they
> will be able to see how buggy a particular program may be.)"

This distinction tweaks me a bit.  One strength of Open Source software
is that any "end user" may choose to be a developer.  But it sounds like
I have little chance of seeing Coverity's scan results on Open Source
software, because I'm not a "maintainer" of any project they've chosen
to scan.  This despite the fact that over the years I've made
(admittedly minor) contributions to a handful of the projects on their
list, and might choose to fix some of the scan-discovered defects if
only I could know what they were.

Jeff
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About "Coverity Study Ranks LAMP Code Quality"

[Martin v. Löwis]

Jeff Epler wrote:
>>Because according to
>>http://www.washingtontechnology.com/news/1_1/daily_news/28134-1.html :
>>
>>"The maintainers of the source codes can register with Coverity to see
>>the full results. (End users cannot see the bug lists themselves; they
>>will be able to see how buggy a particular program may be.)"
> 
> 
> This distinction tweaks me a bit.

I can understand that position. The bugs they find include potential
security flaws, for which exploits could be created if the results are
freely available. While its clearly impossible to keep that information
only with trusted people, they need to make a faithful attempt to
restrict it.

If you have contributed to open source projects, you should ask the
maintainers of these projects to copy you the reports they produced.
If these maintainers consider you trustworthy, they will reveal it to
you.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] checkin r43015

[Jeff Epler]

After the recent discussion about Coverity, I took a look at one of the
checkins made, apparently based on output from their tool.

http://svn.python.org/view/python/branches/release24-maint/Objects/object.c?&r1=43015&r2=43014&rev=43015&view=diff&diff_format=l

This change, a backport of a similar change made to HEAD, doesn't seem
to fix the flaw:  the PyUnicode_CheckExact() call is now guarded against
a NULL return, but the subsequent PyUnicode_Check() and PyString_Check()
calls don't seem to be.

I'm not 100% sure what's going on here, but it still looks a bit fishy.
The API reference says that PyObject_AsUnicode may return NULL, so why
doesn't the function just call PyErr_BadInternalCall() and return NULL?

Jeff
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Still looking for volunteer to run Windows buildbot

[Trent Mick]

[Martin v. Loewis wrote]
> Trent Mick wrote:
> > I do have a sound card in that box, however, the "Sounds and Multimedia
> > Properties" dialog (off Control Panel) says that there are "No Playback
> > Devices" for Sound Playback. So I guess that is it. Maybe the sound card
> > in that box is not hooked up. Grrr. I certainly don't care about the
> > sound card for that box but I don't want the test suite to keep
> > reporting a spurious failure.
> 
> Now, if there was a reliable check whether a soundcard is present, that
> check could be run as a prerequisite, then raising TestSkipped if no
> soundcard is present.

Roger on python-win32 had an answer which works for me:

[Roger Upole wrote]
> WMI can list sound devices.
>
> import win32com.client
> wmi=win32com.client.GetObject('winmgmts:')
> scs=wmi.InstancesOf('win32_sounddevice')
> for sc in scs:
>   print  sc.Properties_('Name'), sc.Properties_('Status')

However, that requires PyWin32 so can't really use that for
test_winsound.py. My understanding of ctypes is that it can NOT replace
win32com, but I'd be happy to be wrong here. Thomas?

Trent

-- 
Trent Mick
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Still looking for volunteer to run Windows buildbot

[Mark Hammond]

> Roger on python-win32 had an answer which works for me:
>
> [Roger Upole wrote]
> > WMI can list sound devices.
> >
> > import win32com.client
> > wmi=win32com.client.GetObject('winmgmts:')
> > scs=wmi.InstancesOf('win32_sounddevice')
> > for sc in scs:
> >   print  sc.Properties_('Name'), sc.Properties_('Status')
>
> However, that requires PyWin32 so can't really use that for
> test_winsound.py. My understanding of ctypes is that it can NOT replace
> win32com, but I'd be happy to be wrong here. Thomas?

Maybe the following VBScript "port" of the above will work:

-- check_soundcard.vbs
rem Check for a working sound-card - exit with 0 if OK, 1 otherwise.
set wmi = GetObject("winmgmts:")
set scs = wmi.InstancesOf("win32_sounddevice")
for each sc in scs
set status = sc.Properties_("Status")
wscript.Echo(sc.Properties_("Name") + "/" + status)
if status = "OK" then
wscript.Quit 0 rem normal exit
end if
next
rem No sound card found - exit with status code of 1
wscript.Quit 1

-- eof

Running "cscript.exe check_soundcard.vbs" and checking the return code
should work.  cscript.exe comes with all modern Windows variants, and
although there may be ways to install Windows without it, I think we can
safely assume it exists for these purposes.

Cheers,

Mark

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Topic suggestions from the PyCon feedback

[A.M. Kuchling]

Those of you on python-checkins will have noticed the recent fiddling
around with the feedback from PyCon.  

I'd like to draw the attention of the python-dev readership to the
answers for the question "What 3 topics should have been covered at
PyCon?"  I split out core Python and web-related topics into their own
lists.   Here's the core Python list:

Usage of new/advanced python features  (12)
Python Best Practices  (3)
Design Patterns in Python  (3)
Anything Alex Martelli wants to talk about.(3)
Advanced Python: metaclasses, descriptors, decorators  (3)
Strategies for integrating C and Python  (2)
Problems with python  (2)
Iterators and Generators  (2)
Ways to improve performance of Python code  (1)
Standard library gems/ unsung modules  (1)
Python Language perspective from non-Guido PEP members  (1)
Py3K  (1)
More on Python internals  (1)
Language howtos (I really enjoyed Alex Martelli's talk last year on itertools)  
(1)
Getting started hacking python source  (1)
Decimal for Experts  (1)
Better Development Practices with Python  (1)
Being more productive with Python  (1)

In the first entry, "new/advanced features" = a fuzzy set containing
generators, iterators, metaclasses, __slots__, and decorators.  When
2.5 is released, this set will probably grow to include the 'with'
statement's context managers and coroutines.  Some of the other
entries in the above list overlap with the first entry.

The conclusion I draw from these results: intermediate- or
advanced-level topics of program design are not covered enough,
whether in the Python documentation, in published books and articles,
or in PyCon talks.  Please feel free to mine the above list, or the
rest of the PyCon feedback, for topic ideas.

In particular: if you're going to attend PyCon 2007, EuroPython, or
some other conference (even a non-Python one), please consider
submitting a talk proposal covering one of the above topics.  Such
presentations would find a receptive audience, I think.

--amk
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Threading idea -- exposing a global thread lock

[Raymond Hettinger]

A user on comp.lang.python has twisted himself into knots writing 
multi-threaded 
code that avoids locks and queues but fails when running code with non-atomic 
access to a shared resource. While his specific design is somewhat flawed, it 
does suggest that we could offer an easy way to make a block of code atomic 
without the complexity of other synchronization tools:

   gil.acquire()
   try:
  #do some transaction that needs to be atomic
   finally:
  gil.release()

The idea is to temporarily suspend thread switches (either using the GIL or a 
global variable in the eval-loop).  Think of it as "non-cooperative" 
multi-threading. While this is a somewhat rough approach, it is dramatically 
simpler than the alternatives (i.e. wrapping locks around every access to a 
resource or feeding all resource requests to a separate thread via a Queue).

While I haven't tried it yet, I think the implementation is likely to be 
trivial.

FWIW, the new with-statement makes the above fragment even more readable:

with atomic_transaction():
# do a series of steps without interruption


Raymond

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] checkin r43015

[Hye-Shik Chang]

On 3/14/06, Jeff Epler <[EMAIL PROTECTED]> wrote:
> After the recent discussion about Coverity, I took a look at one of the
> checkins made, apparently based on output from their tool.
>
> http://svn.python.org/view/python/branches/release24-maint/Objects/object.c?&r1=43015&r2=43014&rev=43015&view=diff&diff_format=l
>
> This change, a backport of a similar change made to HEAD, doesn't seem
> to fix the flaw:  the PyUnicode_CheckExact() call is now guarded against
> a NULL return, but the subsequent PyUnicode_Check() and PyString_Check()
> calls don't seem to be.

Agreed. That change doesn't fix the real problem. I bet it'll still segfault
for PyObject_Unicode(NULL).

In fact, I and Neal talked about the problem and have a correct patch.
But the patch looks bit messy so we hated it. :-)
http://python.org/sf/1444030

> I'm not 100% sure what's going on here, but it still looks a bit fishy.
> The API reference says that PyObject_AsUnicode may return NULL, so why
> doesn't the function just call PyErr_BadInternalCall() and return NULL?

For the consistency with PyObject_String(NULL) which returns
a string "".

Hye-Shik
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why are so many built-in types inheritable?

[Alex Martelli]

On Mar 13, 2006, at 12:29 PM, Fabiano Sidler wrote:

> Hi folks!

Hello Fabiano!  The proper venue for your interesting issues is  
comp.lang.python (or the equivalent mailing list), where all sorts of  
people will be able to "hear" you, discuss things, and help out.   
python-dev is strictly for developers of Python itself, so please re- 
post your issues to comp.lang.python.


Thanks,

Alex


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Still looking for volunteer to run Windows buildbot

[Trent Mick]

[Mark Hammond wrote]
> Maybe the following VBScript "port" of the above will work:
> ...

Cool, yes that works.

> Running "cscript.exe check_soundcard.vbs" and checking the return code
> should work.  cscript.exe comes with all modern Windows variants, and
> although there may be ways to install Windows without it, I think we can
> safely assume it exists for these purposes.

I have a patch in the works that defaults to "yes, this machine does
have a soundcard" if cscript.exe cannot be found on the PATH.

However, one wrinkle: test_winsound.py is made up of three test cases:
BeepTest
MessageBeepTest
PlaySoundTest
only the last need be skipped if there is not soundcard. However,
TestSkipped only works add the module level. So, which is better:

1. Use TestSkipped and skip all three test cases if there is not sound
   card. Running the test suite will actually show that something is
   being skipped.

2. Conditionally define class PlaySoundTest only if there is a
   soundcard. Running the test suite on a machine without a soundcard
   will not show that something is being skipped but *will* run the Beep
   tests.

3. Break out test_winsound.py into two test modules: one with the beep
   tests and one with PlaySoundTest (the latter using TestSkipped).

Trent

-- 
Trent Mick
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Topic suggestions from the PyCon feedback

[Alex Martelli]

On Mar 13, 2006, at 7:22 PM, A.M. Kuchling wrote:
...
> Design Patterns in Python  (3)
> Anything Alex Martelli wants to talk about.(3)
...
> Language howtos (I really enjoyed Alex Martelli's talk last year on  
> itertools)  (1)

Wow, I'm blushing;-).  I promise and swear I'll do the utmost to  
attend Pycon 2007 -- this year, what between work-related issues AND  
the 2nd edition of the Nutshell, it was, alas, really unfeasible:-(.

The *ONE* thing I dislike about working in the US is vacations -- I  
get about half of what I would expect in Europe, and that's with my  
employer being reasonably generous... in practice, given I NEED some  
time to go visit family and friends back in Italy, this means I can't  
really take vacations to do conferences, but rather I must convince  
my boss that conference X is worth my time (basically, this means  
that by attending I can hope to help *HIRE* somebody -- otherwise,  
I'm fighting uphill!-).

> In the first entry, "new/advanced features" = a fuzzy set containing
> generators, iterators, metaclasses, __slots__, and decorators.  When
> 2.5 is released, this set will probably grow to include the 'with'
> statement's context managers and coroutines.  Some of the other
> entries in the above list overlap with the first entry.

Wow, I'm RARIN' for a go at that -- I did the former more than once,  
and 'with' and coroutines sound right up my alley -- if I get any  
chance to practice them, I might even be able to present related  
*patterns*...;-)

> In particular: if you're going to attend PyCon 2007, EuroPython, or
> some other conference (even a non-Python one), please consider
> submitting a talk proposal covering one of the above topics.  Such
> presentations would find a receptive audience, I think.

A-yup.  Most presentations cover specific project, which is fine, but  
there's an unsatisfied demand for talks on how best to use certain  
language features, and design patterns around them -- I suspect  
that's a big reason why I was singled out by name (that kind of thing  
is generally what I address in my talks).


Alex

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Threading idea -- exposing a global thread lock

[Phillip J. Eby]

At 09:57 PM 3/13/2006 -0500, Raymond Hettinger wrote:
>FWIW, the new with-statement makes the above fragment even more readable:
>
> with atomic_transaction():
> # do a series of steps without interruption

+1 on the idea, -1000 on the name.  It's neither atomic nor a 
transaction.  I believe that "critical section" is a more common term for 
what you're proposing.

Probably the primitive could be placed in the thread or threading module, 
so that it would be something like:

 with threading.critical_section():
 # ...

It might be nice to be able to escape out of the critical section using a 
nested with: statement, as this would allow you to treat much of a program 
as single-threaded, and then selectively allow task switching.  But I'm not 
sure what you'd call that API.  :)

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About "Coverity Study Ranks LAMP Code Quality"

[Neal Norwitz]

On 3/13/06, Greg Ewing <[EMAIL PROTECTED]> wrote:
> Fredrik Lundh wrote:
>
> > > But I'm wondering if the actual "bugs" list was transmitted to Python 
> > > developers,
> > > and verified / acted upon.
> >
> > and in case it wasn't clear from my previous post, the answer to
> > your specific question is "yes" ;-)

16 people have access, 2-3 of which I haven't heard of.  6+ different
developers have reviewed at least some of the reports.

> Could whoever did this perhaps post a brief description
> of what sort of information their tool produces, and
> how good a hit rate they have of finding real problems?

I can post info, but it won't be brief. :-)

Their reports were high quality and accurate.  Of the false positives,
it was difficult for the tool to determine that the condition they are
checking really doesn't occur.  One example is doing a
PyString_Check() then using PyString_AsString() (or Size).  The tool
complained that they could return NULL/negative number, which is true
in general.  However, since we already know the object is a string
since we already verified it, we know those APIs can't return error
conditions.

Current results:
 Uninspected 12
 False Positives 13
 Ignore 24
 Resolved 3
 Total 51

For our purposes, False, Ignore, and Resolved are all pretty much the
same in that the warnings are no longer pertinent.  Actually, one of
the resolved is still outstanding as Jeff Epler? pointed out.

http://scan.coverity.com/ shows Python originally had 96 reported
issues, currently 15 are outstanding as of 2006-Mar-13 01:16:18
(presumably PST).  This does not include all C code checked in, since
their report includes code: 272,268, comment: 41,269, blank 38,496
(total: 352033) and:

$ wc -1 `find . -name '*.[ch]'` | tail -n 1
  425810

The public report says 15, but the current developer report shows 12. 
I'm not sure why there is a discrepancy.  All 12 are in ctypes which
was recently imported.

The reports provide information about what condition is violated and
where.  Usually this means there are at least 2, somtimes more,
annotations in the code to point out the originating condition and
problem area.  The types of problems reported in Python include: 
unintialized variables, resource leak, negative return values, using a
NULL pointer, dead code, use after free and some other similar
conditions.

Here's a report of the current disposition of issues:

Name Uninspected Bug False   Ignore  Pending
 ResolvedTotal
0   0   0   1   0   0   1
Everything  12  0   13  23  0   3   51
Modules 12  0   2   8   0   0   22
Modules/expat   0   0   1   0   0   0   1
Objects 0   0   6   8   0   1   15
Parser  0   0   0   3   0   0   3
Python  0   0   5   6   0   2   13
Include 0   0   0   0   0   0   0
Modules/cjkcodecs   0   0   0   0   0   0   0
usr 0   0   0   0   0   0   0
usr/X11R6   0   0   0   0   0   0   0
usr/X11R6/include   0   0   0   0   0   0   0
usr/include 0   0   0   0   0   0   0
usr/include/arpa0   0   0   0   0   0   0
usr/include/asm 0   0   0   0   0   0   0
usr/include/bits0   0   0   0   0   0   0
usr/include/bluetooth   0   0   0   0   0   0   0
usr/include/et  0   0   0   0   0   0   0
usr/include/gdbm0   0   0   0   0   0   0
usr/include/gnu 0   0   0   0   0   0   0
usr/include/linux   0   0   0   0   0   0   0
usr/include/ncurses 0   0   0   0   0   0   0
usr/include/net 0   0   0   0   0   0   0
usr/include/netinet 0   0   0   0   0   0   0
usr/include/netpacket   0   0   0   0   0   0   0
usr/include/openssl 0   0   0   0   0   0   0
usr/include/readline0   0   0   0   0   0   0
usr/include/rpc 0   0   0   0   0   0   0
usr/include/rpcsvc  0   0   0   0   0   0   0
usr/include/sys 0   0   0   0   0   0   0
usr/lib 0   0   0   0   0   0   0
usr/lib/gcc 0   0   0   0   0   0   0
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Threading idea -- exposing a global thread lock

[Guido van Rossum]

Oh, no! Please!

I just had to dissuade someone inside Google from the same idea.

IMO it's fatally flawed for several reasons: it doesn't translate
reasonably to Jython or IronPython, it's really tricky to implement,
and it's an invitation for deadlocks. The danger of this thing in the
wrong hands is too big to warrant the (rare) use case that can only be
solved elegantly using direct GIL access.

--Guido

On 3/13/06, Raymond Hettinger <[EMAIL PROTECTED]> wrote:
> A user on comp.lang.python has twisted himself into knots writing 
> multi-threaded
> code that avoids locks and queues but fails when running code with non-atomic
> access to a shared resource. While his specific design is somewhat flawed, it
> does suggest that we could offer an easy way to make a block of code atomic
> without the complexity of other synchronization tools:
>
>gil.acquire()
>try:
>   #do some transaction that needs to be atomic
>finally:
>   gil.release()
>
> The idea is to temporarily suspend thread switches (either using the GIL or a
> global variable in the eval-loop).  Think of it as "non-cooperative"
> multi-threading. While this is a somewhat rough approach, it is dramatically
> simpler than the alternatives (i.e. wrapping locks around every access to a
> resource or feeding all resource requests to a separate thread via a Queue).
>
> While I haven't tried it yet, I think the implementation is likely to be
> trivial.
>
> FWIW, the new with-statement makes the above fragment even more readable:
>
> with atomic_transaction():
> # do a series of steps without interruption
>
>
> Raymond
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Threading idea -- exposing a global thread lock

[Barry Warsaw]

On Mon, 2006-03-13 at 23:06 -0500, Phillip J. Eby wrote:

> +1 on the idea, -1000 on the name.  It's neither atomic nor a 
> transaction.  I believe that "critical section" is a more common term for 
> what you're proposing.
> 
> Probably the primitive could be placed in the thread or threading module, 
> so that it would be something like:
> 
>  with threading.critical_section():
>  # ...

Or even  threading.synchronized().  But in any event, +1 on the
idea and on sticking the primitive in threading.

> It might be nice to be able to escape out of the critical section using a 
> nested with: statement, as this would allow you to treat much of a program 
> as single-threaded, and then selectively allow task switching.  But I'm not 
> sure what you'd call that API.  :)

threading.unsynchronize()?  Yikes.
-Barry



signature.asc
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Threading idea -- exposing a global thread lock

[Raymond Hettinger]

[Guido]
> Oh, no!

Before shooting this one down, consider a simpler incarnation not involving the 
GIL.  The idea is to allow an active thread to temporarily suspend switching 
for 
a few steps:

threading.stop_switching()
# step1
# step2
# setp3
theading.resume_switching()

To me, this version is innocuous -- it doesn't violate any existing guarantees 
(no thread knows when it is going to be switched out or how long it is going to 
be suspended) and allows the user to easily create an atomic transaction (i.e. 
swapping two global variables or printing a debug value).

While it could be abused, it has the offsetting virtue of being much simpler 
than the alternatives.

I disagree that the need is rare.  My own use case is that I sometimes add some 
debugging print statements that need to execute atomically -- it is a PITA 
because PRINT_ITEM and PRINT_NEWLINE are two different opcodes and are not 
guaranteed to pair atomically.  The current RightWay(tm) is for me to create a 
separate daemon thread for printing and to send lines to it via the queue 
module 
(even that is tricky because you don't want the main thread to exit before a 
print queued item is completed).  I suggest that that is too complex for a 
simple debugging print statement.  It would be great to simply write:

with other_threads_suspended():
print value

I haven't worked through the IronPython and Jython points of view but think 
there is probably a way to implement the simpler incarnation.

Also, unlike the gil.aquire() version, there is no blocking of the currently 
active thread.  IOW, the above incarnation won't deadlock (because the main 
thread never goes into a wait mode).


Raymond






- Original Message - 
From: "Guido van Rossum" <[EMAIL PROTECTED]>
To: "Raymond Hettinger" <[EMAIL PROTECTED]>
Cc: 
Sent: Tuesday, March 14, 2006 12:06 AM
Subject: Re: [Python-Dev] Threading idea -- exposing a global thread lock


Oh, no! Please!

I just had to dissuade someone inside Google from the same idea.

IMO it's fatally flawed for several reasons: it doesn't translate
reasonably to Jython or IronPython, it's really tricky to implement,
and it's an invitation for deadlocks. The danger of this thing in the
wrong hands is too big to warrant the (rare) use case that can only be
solved elegantly using direct GIL access.

--Guido

On 3/13/06, Raymond Hettinger <[EMAIL PROTECTED]> wrote:
> A user on comp.lang.python has twisted himself into knots writing 
> multi-threaded
> code that avoids locks and queues but fails when running code with non-atomic
> access to a shared resource. While his specific design is somewhat flawed, it
> does suggest that we could offer an easy way to make a block of code atomic
> without the complexity of other synchronization tools:
>
>gil.acquire()
>try:
>   #do some transaction that needs to be atomic
>finally:
>   gil.release()
>
> The idea is to temporarily suspend thread switches (either using the GIL or a
> global variable in the eval-loop).  Think of it as "non-cooperative"
> multi-threading. While this is a somewhat rough approach, it is dramatically
> simpler than the alternatives (i.e. wrapping locks around every access to a
> resource or feeding all resource requests to a separate thread via a Queue).
>
> While I haven't tried it yet, I think the implementation is likely to be
> trivial.
>
> FWIW, the new with-statement makes the above fragment even more readable:
>
> with atomic_transaction():
> # do a series of steps without interruption
>
>
> Raymond
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


--
--Guido van Rossum (home page: http://www.python.org/~guido/) 

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Developing/patching ctypes (was: Re: integrating ctypes into python)

[Neal Norwitz]

On 3/9/06, Thomas Heller <[EMAIL PROTECTED]> wrote:
> Would it be a solution to move the 'official' ctypes development into
> Python SVN external/ctypes, or would this be considered abuse?  Another
> location in SVN could be used as well, if external is though to contain
> only vendor drops...

Thomas,

I'd be fine with the official ctypes repo being Python SVN.

The attached patch fixes all the ctypes tests so they pass on amd64. 
It also fixes several warnings.  I'm not sure what else to do with the
patch.  Let me know how you want to handle these in the future.

I'm not sure the patch is 100% correct.  You will need to decide what
can be 64 bits and what can't.  I believe
sq_{item,slice,ass_item,ass_slice} all need to use Py_ssize_t.  The
types in ctypes.h may not require all the changes I made.  I don't
know how you want to support older version, so I unconditionally
changed the types to Py_ssize_t.

n
Index: Modules/_ctypes/callproc.c
===
--- Modules/_ctypes/callproc.c	(revision 43021)
+++ Modules/_ctypes/callproc.c	(working copy)
@@ -721,8 +721,9 @@
 		   O_get), we have to call Py_DECREF because O_get has already
 		   called Py_INCREF.
 		*/
-		if (dict->getfunc == getentry("O")->getfunc)
+		if (dict->getfunc == getentry("O")->getfunc) {
 			Py_DECREF(retval);
+		}
 	} else
 		retval = CData_FromBaseObj(restype, NULL, 0, result);
 
Index: Modules/_ctypes/_ctypes.c
===
--- Modules/_ctypes/_ctypes.c	(revision 43021)
+++ Modules/_ctypes/_ctypes.c	(working copy)
@@ -335,7 +335,7 @@
 };
 
 static PyObject *
-CDataType_repeat(PyObject *self, int length)
+CDataType_repeat(PyObject *self, Py_ssize_t length)
 {
 	return CreateArrayType(self, length);
 }
@@ -695,7 +695,7 @@
 CharArray_set_raw(CDataObject *self, PyObject *value)
 {
 	char *ptr;
-	int size;
+	Py_ssize_t size;
 	if (PyBuffer_Check(value)) {
 		size = value->ob_type->tp_as_buffer->bf_getreadbuffer(value, 0, (void *)&ptr);
 		if (size < 0)
@@ -1789,13 +1789,11 @@
 {
 	char string[256]; /* XXX is that enough? */
 	char *cp = string;
-	int len;
 	*cp++ = index + '0';
 	while (target->b_base) {
 		*cp++ = target->b_index + '0';
 		target = target->b_base;
 	}
-	len = cp - string;
 	return PyString_FromStringAndSize(string, cp-string);
 }
 /* Keep a reference to 'keep' in the 'target', at index 'index' */
@@ -1806,7 +1804,7 @@
  * key int the root object's _objects dict.
  */
 static int
-KeepRef(CDataObject *target, int index, PyObject *keep)
+KeepRef(CDataObject *target, Py_ssize_t index, PyObject *keep)
 {
 	int result;
 	CDataObject *ob;
@@ -1875,7 +1873,7 @@
 	{ NULL },
 };
 
-static int CData_GetBuffer(CDataObject *self, int seg, void **pptr)
+static Py_ssize_t CData_GetBuffer(CDataObject *self, Py_ssize_t seg, void **pptr)
 {
 	if (seg != 0) {
 		/* Hm. Must this set an exception? */
@@ -1885,7 +1883,7 @@
 	return self->b_size;
 }
 
-static int CData_GetSegcount(CDataObject *self, int *lenp)
+static Py_ssize_t CData_GetSegcount(CDataObject *self, Py_ssize_t *lenp)
 {
 	if (lenp)
 		*lenp = 1;
@@ -1893,10 +1891,10 @@
 }
 
 static PyBufferProcs CData_as_buffer = {
-	(getreadbufferproc)CData_GetBuffer,
-	(getwritebufferproc)CData_GetBuffer,
-	(getsegcountproc)CData_GetSegcount,
-	(getcharbufferproc)NULL,
+	(readbufferproc)CData_GetBuffer,
+	(writebufferproc)CData_GetBuffer,
+	(segcountproc)CData_GetSegcount,
+	(charbufferproc)NULL,
 };
 
 /*
@@ -1985,7 +1983,7 @@
 }
 
 PyObject *
-CData_FromBaseObj(PyObject *type, PyObject *base, int index, char *adr)
+CData_FromBaseObj(PyObject *type, PyObject *base, Py_ssize_t index, char *adr)
 {
 	CDataObject *cmem;
 	StgDictObject *dict;
@@ -2064,7 +2062,7 @@
 
 PyObject *
 CData_get(PyObject *type, GETFUNC getfunc, PyObject *src,
-	  int index, int size, char *adr)
+	  Py_ssize_t index, Py_ssize_t size, char *adr)
 {
 	StgDictObject *dict;
 	if (getfunc)
@@ -2081,7 +2079,7 @@
 */
 static PyObject *
 _CData_set(CDataObject *dst, PyObject *type, SETFUNC setfunc, PyObject *value,
-	   int size, char *ptr)
+	   Py_ssize_t size, char *ptr)
 {
 	CDataObject *src;
 
@@ -2177,7 +2175,7 @@
  */
 int
 CData_set(PyObject *dst, PyObject *type, SETFUNC setfunc, PyObject *value,
-	  int index, int size, char *ptr)
+	  Py_ssize_t index, Py_ssize_t size, char *ptr)
 {
 	CDataObject *mem = (CDataObject *)dst;
 	PyObject *result;
@@ -3318,7 +3316,7 @@
 
 	if (kwds) {
 		PyObject *key, *value;
-		int pos = 0;
+		Py_ssize_t pos = 0;
 		while(PyDict_Next(kwds, &pos, &key, &value)) {
 			if (-1 == PyObject_SetAttr(self, key, value))
 return -1;
@@ -3471,12 +3469,12 @@
 }
 
 static PyObject *
-Array_slice(CDataObject *self, int ilow, int ihigh)
+Array_slice(CDataObject *self, Py_ssize_t ilow, Py_ssize_t ihigh)
 {
 	StgDictObject *stgdict, *itemdict;
 	PyObject *proto;
 	PyListObject *np;
-	int i, len;
+	Py_ssize_t i, len;
 
 	if (ilow < 0)
 		ilow = 0;
@@ -3587,13 +3585,13 @@
 }
 
 static PySequenceM

Re: [Python-Dev] Threading idea -- exposing a global thread lock

[Raymond Hettinger]

> it doesn't translate reasonably to Jython or IronPython, it's really tricky 
> to 
> implement,

FWIW, someone on the newsgroup suggested implementing this via a slight 
modification to sys.checkinterval().  The idea was that a None argument would 
translate to "stop-checking" and the active thread would do a few steps 
atomically and then restore the original value.  The new with-statement makes 
that a piece of cake.


Raymond

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] checkin r43015

[Neal Norwitz]

On 3/13/06, Hye-Shik Chang <[EMAIL PROTECTED]> wrote:
> On 3/14/06, Jeff Epler <[EMAIL PROTECTED]> wrote:
> > After the recent discussion about Coverity, I took a look at one of the
> > checkins made, apparently based on output from their tool.
> >
> > http://svn.python.org/view/python/branches/release24-maint/Objects/object.c?&r1=43015&r2=43014&rev=43015&view=diff&diff_format=l
> >
> > This change, a backport of a similar change made to HEAD, doesn't seem
> > to fix the flaw:  the PyUnicode_CheckExact() call is now guarded against
> > a NULL return, but the subsequent PyUnicode_Check() and PyString_Check()
> > calls don't seem to be.
>
> Agreed. That change doesn't fix the real problem. I bet it'll still segfault
> for PyObject_Unicode(NULL).
>
> In fact, I and Neal talked about the problem and have a correct patch.
> But the patch looks bit messy so we hated it. :-)
> http://python.org/sf/1444030

Right.  I realized it would just be easier to inline the code, which
is what i wound up doing.  This problem should be addressed fully now.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Threading idea -- exposing a global thread lock

[Samuele Pedroni]

Raymond Hettinger wrote:
>>it doesn't translate reasonably to Jython or IronPython, it's really tricky 
>>to 
>>implement,
> 
> 
> FWIW, someone on the newsgroup suggested implementing this via a slight 
> modification to sys.checkinterval().  The idea was that a None argument would 
> translate to "stop-checking" and the active thread would do a few steps 
> atomically and then restore the original value.  The new with-statement makes 
> that a piece of cake.
> 
>

there's no sys.checkinterval in Jython. Implementing this would need the
introduction of some kind of GIL implementation in Jython, the JVM has 
no primitive for global critical sections. A GIL is something users of 
Jython don't want. Even with that is way too easy to have non-Jython 
threads around that could manipulates Java objects you are seamlessy 
dealing with. That is of course true for CPython and C extension 
initiated threads too, but is probably harder to be oblivious
of such things in CPython context.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About "Coverity Study Ranks LAMP Code Quality"

[Fredrik Lundh]

Neal Norwitz wrote:

> Their reports were high quality and accurate.

absolutely (which is why I'm surprised that someone's using the un-
reviewed numbers are a quality measure; guess I have to go back
and read the article to see who that was...)

> Of the false positives, it was difficult for the tool to determine that
> the condition they are checking really doesn't occur.  One example
> is doing a PyString_Check() then using PyString_AsString() (or Size).
> The tool complained that they could return NULL/negative number,
> which is true in general.

one favourite was a function that used both a return value and an
output argument, and used the following combinations to represent
the different outcomes:

return=NULL; output=junk => out of memory
return=junk; output=-1 => cannot do this
return=pointer; output=value => did this, returned value bytes

i.e.

p = function(..., &out);
if (!p)
return "out of memory";
if (out == -1)
use fallback algorithm
return result;
else {
generate result
free(p);
return result;
}

Coverity pointed out that in the out == -1 case, the calling code
never released the p pointer.

while it was easy to flag this as INVALID, I agree that the design
is a bit questionable; I'd probably change the function slightly to
make sure it always returns either NULL or a valid pointer:

p = function(..., &out);
if (!p) {
if (out == -1)
use fallback algorithm
return result;
}
return "out of memory";
}
generate result
free(p);
return result;

anyway, this is of course related to why the raw coverity figures can
be grossly misleading: code that uses certain coding patterns may get
really bad figures, without being broken in any way whatsoever.

(on the other hand, you can treat the Coverity result as "even more
warnings than your (current) compiler can provide", and fix as many
false reports as you can, just for stylistic reasons).





___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Still looking for volunteer to run Windows buildbot

[Martin v. Löwis]

Trent Mick wrote:
> 1. Use TestSkipped and skip all three test cases if there is not sound
>card. Running the test suite will actually show that something is
>being skipped.

This is best. The sound tests are not that important that they
absolutely need to be run.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com