My feeling is that we should advise users of the library that any slices of a ResizableBuffer become invalid after a call to Resize.
> I was thinking about something like this [0]. The point is, that the slice > user has no way of knowing if the slice can still be safely used and who > owns the memory. You can look at the Buffer parent to see if there is a parent-child relationship, which at least tells you whether you definitely do _not_ own the memory. I'm not convinced from this use case that we need to change the way that the Buffer abstraction works. If there is a need for memory ownership-nannying, that may be best handled by some other kind of abstract interface that uses Buffers for its implementation. - Wes On Wed, Apr 11, 2018 at 8:05 AM, Antoine Pitrou <anto...@python.org> wrote: > > Hi Dimitri, > > Le 11/04/2018 à 13:42, Dimitri Vorona a écrit : >> >> I was thinking about something like this [0]. The point is, that the slice >> user has no way of knowing if the slice can still be safely used and who >> owns the memory. > > I think the answer is that calling free() on something you exported to > consumers is incorrect. If you allocate buffers, you should choose a > Buffer implementation with proper ownership semantics. For example, we > have PoolBuffer, but also Python buffers and CUDA buffers. They all > (should) have proper ownership. If you want to create buffers with data > managed with malloc/free, you need to write a MallocBuffer implementation. > >> A step back is a good idea. My use case would be to return a partially >> built slice on a buffer, while continuing appending to the buffer. Think >> delta dictionaries: while a slice of the coding table can be sent, we will >> have additional data to append later on. > > I don't know anything about delta dictionaries, but I get the idea. > > Does the implementation become harder if you split the coding table into > several buffers that never get resized? > >> To build on your previous proposal: maybe some more finely grained locking >> mechanism, like the data_ being a shared_ptr<uint_8*>, slices grabbing a >> copy of it when they want to use it and releasing it afterwards? The parent >> would then check the couter of the shared_ptr (similar to the number of >> slices). > > You need an actual lock to avoid race conditions (the parent may find a > zero shared_ptr counter, but another thread would grab a data pointer > immediately after). > > I wonder if we really want such implementation complexity. Also, > everyone is now paying the price of locking. Ideally slicing and > fetching a data pointer should be cheap. I'd like to know what others > think about this. > > Regards > > Antoine.
