Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-07-18 Thread Matti Picus
The NEP [0] and the corresponding PR [1] have gone through another round of editing. I would like to restart the discussion here if anyone has more to add. Things that have changed since the last round: - The functions now accept a context argument - The code has been cleaned up for consistenc

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-15 Thread Eric Wieser
Note that PEP-445 which introduced `PyMemAllocatorEx` specifically rejected omitting the `ctx` argument here: https://www.python.org/dev/peps/pep-0445/#id23, which is another argument in favor of having it. I'll try to give a more thorough justification for the pyobject / capsule suggestion in ano

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-13 Thread eliaskoromilas
Eric Wieser wrote >> Yes, sorry, had been a while since I had looked it up: >> >> https://docs.python.org/3/c-api/memory.html#c.PyMemAllocatorEx > > That `PyMemAllocatorEx` looks almost exactly like one of the two variants > I > was proposing. Is there a reason for wanting to define our own struct

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-12 Thread leofang
Eric Wieser wrote >> Yes, sorry, had been a while since I had looked it up: >> >> https://docs.python.org/3/c-api/memory.html#c.PyMemAllocatorEx > > That `PyMemAllocatorEx` looks almost exactly like one of the two variants > I was proposing. Is there a reason for wanting to define our own structur

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-11 Thread Sebastian Berg
On Tue, 2021-05-11 at 09:54 +0100, Eric Wieser wrote: > > Yes, sorry, had been a while since I had looked it up: > > > > https://docs.python.org/3/c-api/memory.html#c.PyMemAllocatorEx > > That `PyMemAllocatorEx` looks almost exactly like one of the two > variants I > was proposing. Is there a rea

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-11 Thread Eric Wieser
> Yes, sorry, had been a while since I had looked it up: > > https://docs.python.org/3/c-api/memory.html#c.PyMemAllocatorEx That `PyMemAllocatorEx` looks almost exactly like one of the two variants I was proposing. Is there a reason for wanting to define our own structure vs just using that one? I

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-10 Thread Matti Picus
On 10/5/21 8:43 pm, Sebastian Berg wrote: But right now the proposal says this is static, and I honestly don't see much reason for it to be freeable? I think this is the crux of the issue. The current design is for a singly-allocated struct to be passed around since it is just an aggregate

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-10 Thread Sebastian Berg
On Mon, 2021-05-10 at 10:01 +0100, Eric Wieser wrote: > > The Python version of this does have a `void *ctx`, but I am not > > sure if > the use for this is actually valuable for the NumPy use-cases. > > Do you mean "the CPython version"? If so, can you link a reference? Yes, sorry, had been a wh

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-10 Thread Eric Wieser
> The Python version of this does have a `void *ctx`, but I am not sure if the use for this is actually valuable for the NumPy use-cases. Do you mean "the CPython version"? If so, can you link a reference? > While I like the `PyObject *` idea, I am also not sure that it helps much. If we want a

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-09 Thread Sebastian Berg
On Thu, 2021-05-06 at 13:06 +0100, Eric Wieser wrote: > Another argument for supporting stateful allocators would be > compatibility > with the stateful C++11 allocator API, such as > https://en.cppreference.com/w/cpp/memory/allocator_traits/allocate. The Python version of this does have a `void

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-06 Thread Eric Wieser
Another argument for supporting stateful allocators would be compatibility with the stateful C++11 allocator API, such as https://en.cppreference.com/w/cpp/memory/allocator_traits/allocate. Adding support for stateful allocators at a later date would almost certainly create an ABI breakage or lots

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-06 Thread Matti Picus
On 6/5/21 2:07 pm, Eric Wieser wrote: The NEP looks good, but I worry the API isn't flexible enough. My two main concerns are: ### Stateful allocators Consider an allocator that aligns to `N` bytes, where `N` is configurable from a python call in someone else's extension module. ... ### Th

Re: [Numpy-discussion] Proposal to accept NEP 49: Data allocation strategies

2021-05-06 Thread Eric Wieser
The NEP looks good, but I worry the API isn't flexible enough. My two main concerns are: ### Stateful allocators Consider an allocator that aligns to `N` bytes, where `N` is configurable from a python call in someone else's extension module. Where do they store `N`? They can hide it in `PyDataMe