Kristján Valur Jónsson <krist...@ccpgames.com> added the comment:

You may find this hard to believe, but we do in fact embed python into other 
applications.  In this case, it is UnrealEngine, to drive a complex, console 
based game.  Yes, embedding python is much harder than it need be and I'll 
submit some patches to make that easier someday, but that's not the point of 
this.

appMalloc, is in this case, the canonical memory allocator in UnrealEngine.  
But it could be any other memory allocator so that is beside the point.

The problem at hand, however, is this memory allocator _may_ have to inquire 
about the state of Python.  It would do this, for example, to gather statistics 
about Python's memory use.  This is critically important when developing 
console based applications, where every Kilobyte counts.  Embedding python 
sometimes requires the replacement of  the standard libc malloc with something 
else.

What appMalloc is doing, in this case, is for every allocation, to get the 
python TLS pointer.  There is nothing wrong with this, this is a GIL free 
operation, and it will return either NULL or the current TLS.  And it works, 
except, when appMalloc (through python's malloc) is being invoked from the TLS 
entry creation mechanism itself.  Then it deadlocks.

Now, regardless of the above, surely it is an improvement in general if we make 
tighter use of the TLS lock.  It's not a good idea to hold this lock across 
malloc calls if we can avoid it.  The patch is harmless, might even be an 
improvement, so why object to it?  It removes yet another "gotcha" that 
embedders, or those replacing malloc, (or instrumenting python's malloc use) 
have to face.

Cheers,

K

----------

_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue9787>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Reply via email to