Hey Saul, The Buffer returned by Plasma's Create is not owning the memory, the memory management happens via reference counting through the explicit PlasmaClient::Release call. The reason why we now return a buffer instead of a plain pointer and length is so we can support both CPU (MutableBuffer) and GPU (CudaBuffer) memory with the same interface.
If you only use CPU memory, it is safe to dynamically cast the Buffer returned by Create to a MutableBuffer and get it's underlying pointer via the mutable_data() function. This behavior might change in the future however if we decide to do reference counting through the Buffer returned by PlasmaClient::Create. However if we are merging libplasma into arrow soon I think it is save to depend on the behavior as it is at the moment. -- Philipp. On Tue, Mar 27, 2018 at 1:04 PM, Saul Shanabrook <s.shanabr...@gmail.com> wrote: > Yeah, I am writing C++ code that provides a C wrapper around this. But, how > do I access either the protected mutable_data_ or the inline mutable_data > from my this wrapper? > > On Tue, Mar 27, 2018 at 10:52 AM Wes McKinney <wesmck...@gmail.com> wrote: > > > @Kou, we removed that bit, so Buffer::mutable_data is always inline now. > > > > @Saul, you will want to create a C wrapper for arrow::Buffer so you > > can manage memory lifetime from C. Depending on the kind of Buffer you > > have, it may not be as simple as calling free(...) on its internal > > data. > > > > - Wes > > > > On Tue, Mar 27, 2018 at 10:31 AM, Kouhei Sutou <k...@clear-code.com> > wrote: > > > Hi, > > > > > > It may be wrong but you may need to add -DNDEBUG flag to C > > > compiler. > > > > > > > > > Thanks, > > > -- > > > kou > > > > > > In <CANapY_LzahViR9e7e90H=wq3j0tnbZJe1Vm=j7=vvyqawnc...@mail.gmail.com > > > > > "Re: Using Plasma with xnd" on Tue, 27 Mar 2018 13:41:49 +0000, > > > Saul Shanabrook <s.shanabr...@gmail.com> wrote: > > > > > >> Ah, I see I can use garrow_buffer_get_mutable_data from c_glib. > > >> > > >> On Tue, Mar 27, 2018 at 9:29 AM Saul Shanabrook < > s.shanabr...@gmail.com > > > > > >> wrote: > > >> > > >>> I am in the process of changing over my C wrapper to work with Arrow > > >>> 0.9.0. One change is that the plasma APIs now use arrow::Buffer > > instead of > > >>> of raw pointers, in Create and Get. > > >>> > > >>> In my C wrapper for Create, I need to get the mutable data pointer > from > > >>> the arrow::Buffer, after it is created. Here is my current > > implementation: > > >>> > > https://github.com/plures/pxnd/blob/84e9403ef3811e0893e12076455d7c > bc49c39358/libplasma/plasma.cc#L56-L75 > > >>> I do something similar to the Python wrapper around plasma: > > >>> > > https://github.com/apache/arrow/blob/f9f8320339692d4134d1ef42c32cb7 > c8d547593e/python/pyarrow/plasma.pyx#L300-L307 > > >>> > > >>> > > >>> However, this fails to link, I think because the mutable_data method > is > > >>> defined in the class and so is inline. So it doesn't show up in the > > >>> libarrow.dylib/libarrow.a archive. Let me know if I am missing > > something > > >>> obvious, this is just my hypothesis for why it isn't working. > > >>> > > >>> Is there a way for a third party library to get the pointer to the > data > > >>> from a arrow::Buffer? > > >>> > > >>> On Wed, Mar 7, 2018 at 10:59 AM Wes McKinney <wesmck...@gmail.com> > > wrote: > > >>> > > >>>> Where are you going to make Plasma requests, in Python or in C? > > >>>> > > >>>> * If in Python, use the buffer protocol to access the memory in C > > >>>> https://docs.python.org/3/c-api/buffer.html#bufferobjects > > >>>> * If in C, then I would suggest adding extern "C" APIs directly in > > >>>> libplasma.so so you can connect to the Plasma store and make > requests > > >>>> directly in C. > > >>>> > > >>>> I want to make 100% sure that you need to interact with the Plasma > > >>>> client in C before I send you down that path. Here's an example of > a C > > >>>> API embedded in an otherwise C++ library > > >>>> > > https://github.com/wesm/feather/blob/master/cpp/src/feather/feather-c.h. > > >>>> > > >>>> - Wes > > >>>> > > >>>> On Wed, Mar 7, 2018 at 10:53 AM, Saul Shanabrook < > > s.shanabr...@gmail.com> > > >>>> wrote: > > >>>> > Hey Wes, > > >>>> > > > >>>> > Since all our logic for writing and reading the data is written in > > C, we > > >>>> > need to be get access to the memory in C as well. I think it makes > > sense > > >>>> > then for me to work on creating a C++ wrapper around Plasma that > > exports > > >>>> > what is necessary in C. Should that be in Arrow GLib? > > >>>> > > > >>>> > Best, > > >>>> > Saul > > >>>> > > > >>>> > On Tue, Mar 6, 2018 at 11:49 PM Wes McKinney <wesmck...@gmail.com > > > > >>>> wrote: > > >>>> > > > >>>> >> hi Saul -- I think the easiest solution here is the > > buffer/memoryview > > >>>> >> protocol. You won't have to touch the Cython or C++ API from > > pyarrow > > >>>> >> if you do this. > > >>>> >> > > >>>> >> You can interact with a Buffer object like any other Python > object > > >>>> >> implementing the buffer protocol. See numpy.frombuffer as an > > example > > >>>> >> of a function that interacts with such objects. I would suggest > > adding > > >>>> >> a method to xnd method for this. > > >>>> >> > > >>>> >> If you need to interact with Plasma from C then things will be > more > > >>>> >> complicated -- Robert or Philipp should be able to advise in this > > >>>> >> case. > > >>>> >> > > >>>> >> - Wes > > >>>> >> > > >>>> >> On Tue, Mar 6, 2018 at 4:55 PM, Saul Shanabrook < > > >>>> s.shanabr...@gmail.com> > > >>>> >> wrote: > > >>>> >> > Hey Wes, > > >>>> >> > > > >>>> >> > I don't have much experience doing C + Python + Cython > > development, > > >>>> so I > > >>>> >> am > > >>>> >> > probably missing something obvious, but reading the Cython > docs, > > >>>> >> > < > > >>>> >> > > >>>> > > https://cython.readthedocs.io/en/latest/src/userguide/ > external_C_code.html#public-declarations > > >>>> >> > > > >>>> >> > it > > >>>> >> > seems like I can only access types marked as public from C > code. > > >>>> When I > > >>>> >> > compile arrow locally, I do get some C++ headers for the plasma > > code, > > >>>> >> but I > > >>>> >> > don't think I can use them from C code either. > > >>>> >> > > > >>>> >> > Best, > > >>>> >> > Saul > > >>>> >> > > > >>>> >> > > > >>>> >> > > > >>>> >> > On Tue, Mar 6, 2018 at 3:12 PM Wes McKinney < > wesmck...@gmail.com > > > > > >>>> wrote: > > >>>> >> > > > >>>> >> >> hi Saul, > > >>>> >> >> > > >>>> >> >> Are you able to use the buffer/memoryview protocol? Instances > of > > >>>> >> >> pyarrow.Buffer, like PlasmaBuffer, support this > > >>>> >> >> > > >>>> >> >> > > >>>> >> > > >>>> > > https://github.com/apache/arrow/blob/master/python/ > pyarrow/plasma.pyx#L182 > > >>>> >> >> > > >>>> >> >> - Wes > > >>>> >> >> > > >>>> >> >> On Tue, Mar 6, 2018 at 3:09 PM, Saul Shanabrook < > > >>>> s.shanabr...@gmail.com > > >>>> >> > > > >>>> >> >> wrote: > > >>>> >> >> > I am trying to use the Plasma store to back xnd objects. > Xnd ( > > >>>> >> >> > https://xnd.readthedocs.io/en/latest/xnd/index.html) is a > > >>>> container > > >>>> >> >> library > > >>>> >> >> > in C that has Python bindings. I would like to get a pointer > > to > > >>>> the > > >>>> >> >> > allocated memory after creating or get an object in Plasma. > I > > see > > >>>> that > > >>>> >> >> this > > >>>> >> >> > is supported in the C++ API ( > > >>>> >> >> > > > >>>> >> >> > > >>>> >> > > >>>> > > https://arrow.apache.org/docs/cpp/classplasma_1_1_plasma_client.html# > ac18ab9cc792c620a97a3dcb165e0ecd7 > > >>>> >> >> ) > > >>>> >> >> > but not in the python API (as far as I can tell). Is it > > possible > > >>>> to > > >>>> >> use > > >>>> >> >> the > > >>>> >> >> > C++ Plasma API from a C project? If not, would it make sense > > to > > >>>> expose > > >>>> >> >> > pointer access on the Python API using capsules > > >>>> >> >> > https://docs.python.org/3.6/c-api/capsule.html > > >>>> >> >> > <https://docs.python.org/3.6/c-api/capsule.html#capsules>? > > >>>> >> >> > > >>>> >> > > >>>> > > >>> > > >
