The new memory API discussions (and PEP) warrant a quick pyparallel
update: a couple of weeks after PyCon, I came up with a solution for
the biggest show-stopper that has been plaguing pyparallel since its
inception: being able to detect the modification of "main thread"
Python
On 19 June 2013 23:10, Trent Nelson wrote:
> So, Victor, I'm interested to hear how the new API you're proposing
> will affect this solution I've come up with for pyparallel; I'm
> going to be absolutely dependent upon the ability to lock main
> thread pages as read-only in one fel
2013/6/19 Trent Nelson :
>
> The new memory API discussions (and PEP) warrant a quick pyparallel
> update: a couple of weeks after PyCon, I came up with a solution for
> the biggest show-stopper that has been plaguing pyparallel since its
> inception: being able to detect the modifi
Le Tue, 18 Jun 2013 22:40:49 +0200,
Victor Stinner a écrit :
>
> Other changes
> -
>
[...]
>
> * Configure external libraries like zlib or OpenSSL to allocate memory
> using ``PyMem_RawMalloc()``
Why so, and is it done by default?
> Only one get/set function for block allocators
Hi Charles-François!
Good to hear from you again. It was actually your e-mail a few
months ago that acted as the initial catalyst for this memory
protection idea, so, thanks for that :-)
Answer below.
On Wed, Jun 19, 2013 at 07:01:49AM -0700, Charles-François Natali wrote:
> 201
2013/6/19 Antoine Pitrou :
> Le Tue, 18 Jun 2013 22:40:49 +0200,
> Victor Stinner a écrit :
>>
>> Other changes
>> -
>>
> [...]
>>
>> * Configure external libraries like zlib or OpenSSL to allocate memory
>> using ``PyMem_RawMalloc()``
>
> Why so, and is it done by default?
(Oh, I r
Right, think of the "ctxt" as a "this" pointer from c++.
If you have an allocator object, that you got from some c++ api, and want to
ask Python to use that, you need to be able to thunk the "this" pointer to get
at the particular allocator instance.
It used to be a common mistake when writing C
On Wed, 19 Jun 2013 17:24:21 +0200
Victor Stinner wrote:
>
> For the "track memory usage" use case, it is important to track memory
> allocated in external libraries to have accurate reports, because
> these allocations may be huge.
[...]
> Not in main(). The Python expat and zlib modules call di
2013/6/19 Antoine Pitrou :
> On Wed, 19 Jun 2013 17:24:21 +0200
>> >> Drawback: the caller has to check if the result is 0, or handle the
>> >> error.
>> >
>> > Or you can just call Py_FatalError() if the domain is invalid.
>>
>> I don't like Py_FatalError(), especially when Python is embedded. It'
> 1. All memory allocated in a parallel context is localized to a
> private heap.
How do you allocate memory in this "private" heap? Did you add new
functions to allocate memory?
Victor
___
Python-Dev mailing list
[email protected]
http
Oh, it should be public, in my opinion.
We do exactly that when we embed python into UnrealEngine. We keep pythons
internal PyObject_Mem allocator, but have it ask UnrealEngine for its arenas.
That way, we can still keep track of python's memory usage from with the larger
application, even if
On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote:
> > 1. All memory allocated in a parallel context is localized to a
> > private heap.
>
> How do you allocate memory in this "private" heap? Did you add new
> functions to allocate memory?
Yup:
_PyHeap_Malloc
2013/6/19 Kristján Valur Jónsson :
> Oh, it should be public, in my opinion.
Ok. And do you think that the PyMemMappingAllocator structure is
complete, or that we should add something to be future-proof? At
least, PyMemMappingAllocator is enough for pymalloc usage :-)
Is PyMemMappingAllocator com
On 6/19/2013 11:24 AM, Victor Stinner wrote:
2013/6/19 Antoine Pitrou :
Le Tue, 18 Jun 2013 22:40:49 +0200,
Victor Stinner a écrit :
Only one get/set function for block allocators
--
Replace the 6 functions:
* ``void PyMem_GetRawAllocator(PyMemBl
"""
So, Victor, I'm interested to hear how the new API you're proposing
will affect this solution I've come up with for pyparallel; I'm
going to be absolutely dependent upon the ability to lock main
thread pages as read-only in one fell-swoop -- am I still going to
be able to do
On Wed, 19 Jun 2013 17:49:02 +0200
Victor Stinner wrote:
> 2013/6/19 Antoine Pitrou :
> > On Wed, 19 Jun 2013 17:24:21 +0200
> >> >> Drawback: the caller has to check if the result is 0, or handle the
> >> >> error.
> >> >
> >> > Or you can just call Py_FatalError() if the domain is invalid.
> >>
On Wed, Jun 19, 2013 at 09:20:15AM -0700, Victor Stinner wrote:
> """
> So, Victor, I'm interested to hear how the new API you're proposing
> will affect this solution I've come up with for pyparallel; I'm
> going to be absolutely dependent upon the ability to lock main
> thread pag
PyMem_RawAlloc()/Realloc/Free should be part of the stable ABI. I agree
that all other new fumctions ans structures should not.
Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http
On 20 Jun 2013 02:03, "Victor Stinner" wrote:
>
> 2013/6/19 Kristján Valur Jónsson :
> > Oh, it should be public, in my opinion.
>
> Ok. And do you think that the PyMemMappingAllocator structure is
> complete, or that we should add something to be future-proof? At
> least, PyMemMappingAllocator is
Le jeudi 20 juin 2013, Nick Coghlan a écrit :
>
> > Is PyMemMappingAllocator complete enough for your usage at CCP Games?
>
> Can we go back to calling this the "Arena" allocator? Or at least
> "Mapped"? When I see "Mapping" in the context of Python I think of the
> container API, not a memory allo
20 matches
Mail list logo
