> But, how do I access either the protected mutable_data_ or the inline > mutable_data from my this wrapper?
I may be missing something, but you need to use Buffer::mutable_data() somewhere in the wrapper. So either provide a function in the wrapper that calls it, or invoke it during some init step (if you are copying fields into a C struct) On Tue, Mar 27, 2018 at 4:33 PM, Philipp Moritz <pcmor...@gmail.com> wrote: > Also, if you provide me with full instructions on how to reproduce the > linking error, I'm happy to look if I can figure out what is going on. > > On Tue, Mar 27, 2018 at 1:25 PM, Philipp Moritz <pcmor...@gmail.com> wrote: > >> Hey Saul, >> >> The Buffer returned by Plasma's Create is not owning the memory, the >> memory management happens via reference counting through the explicit >> PlasmaClient::Release call. The reason why we now return a buffer instead >> of a plain pointer and length is so we can support both CPU (MutableBuffer) >> and GPU (CudaBuffer) memory with the same interface. >> >> If you only use CPU memory, it is safe to dynamically cast the Buffer >> returned by Create to a MutableBuffer and get it's underlying pointer via >> the mutable_data() function. >> >> This behavior might change in the future however if we decide to do >> reference counting through the Buffer returned by PlasmaClient::Create. >> However if we are merging libplasma into arrow soon I think it is save to >> depend on the behavior as it is at the moment. >> >> -- Philipp. >> >> On Tue, Mar 27, 2018 at 1:04 PM, Saul Shanabrook <s.shanabr...@gmail.com> >> wrote: >> >>> Yeah, I am writing C++ code that provides a C wrapper around this. But, >>> how >>> do I access either the protected mutable_data_ or the inline mutable_data >>> from my this wrapper? >>> >>> On Tue, Mar 27, 2018 at 10:52 AM Wes McKinney <wesmck...@gmail.com> >>> wrote: >>> >>> > @Kou, we removed that bit, so Buffer::mutable_data is always inline now. >>> > >>> > @Saul, you will want to create a C wrapper for arrow::Buffer so you >>> > can manage memory lifetime from C. Depending on the kind of Buffer you >>> > have, it may not be as simple as calling free(...) on its internal >>> > data. >>> > >>> > - Wes >>> > >>> > On Tue, Mar 27, 2018 at 10:31 AM, Kouhei Sutou <k...@clear-code.com> >>> wrote: >>> > > Hi, >>> > > >>> > > It may be wrong but you may need to add -DNDEBUG flag to C >>> > > compiler. >>> > > >>> > > >>> > > Thanks, >>> > > -- >>> > > kou >>> > > >>> > > In <CANapY_LzahViR9e7e90H=wq3j0tnbZJe1Vm=j7=vvyqawnc...@mail.gm >>> ail.com> >>> > > "Re: Using Plasma with xnd" on Tue, 27 Mar 2018 13:41:49 +0000, >>> > > Saul Shanabrook <s.shanabr...@gmail.com> wrote: >>> > > >>> > >> Ah, I see I can use garrow_buffer_get_mutable_data from c_glib. >>> > >> >>> > >> On Tue, Mar 27, 2018 at 9:29 AM Saul Shanabrook < >>> s.shanabr...@gmail.com >>> > > >>> > >> wrote: >>> > >> >>> > >>> I am in the process of changing over my C wrapper to work with Arrow >>> > >>> 0.9.0. One change is that the plasma APIs now use arrow::Buffer >>> > instead of >>> > >>> of raw pointers, in Create and Get. >>> > >>> >>> > >>> In my C wrapper for Create, I need to get the mutable data pointer >>> from >>> > >>> the arrow::Buffer, after it is created. Here is my current >>> > implementation: >>> > >>> >>> > https://github.com/plures/pxnd/blob/84e9403ef3811e0893e12076 >>> 455d7cbc49c39358/libplasma/plasma.cc#L56-L75 >>> > >>> I do something similar to the Python wrapper around plasma: >>> > >>> >>> > https://github.com/apache/arrow/blob/f9f8320339692d4134d1ef4 >>> 2c32cb7c8d547593e/python/pyarrow/plasma.pyx#L300-L307 >>> > >>> >>> > >>> >>> > >>> However, this fails to link, I think because the mutable_data >>> method is >>> > >>> defined in the class and so is inline. So it doesn't show up in the >>> > >>> libarrow.dylib/libarrow.a archive. Let me know if I am missing >>> > something >>> > >>> obvious, this is just my hypothesis for why it isn't working. >>> > >>> >>> > >>> Is there a way for a third party library to get the pointer to the >>> data >>> > >>> from a arrow::Buffer? >>> > >>> >>> > >>> On Wed, Mar 7, 2018 at 10:59 AM Wes McKinney <wesmck...@gmail.com> >>> > wrote: >>> > >>> >>> > >>>> Where are you going to make Plasma requests, in Python or in C? >>> > >>>> >>> > >>>> * If in Python, use the buffer protocol to access the memory in C >>> > >>>> https://docs.python.org/3/c-api/buffer.html#bufferobjects >>> > >>>> * If in C, then I would suggest adding extern "C" APIs directly in >>> > >>>> libplasma.so so you can connect to the Plasma store and make >>> requests >>> > >>>> directly in C. >>> > >>>> >>> > >>>> I want to make 100% sure that you need to interact with the Plasma >>> > >>>> client in C before I send you down that path. Here's an example of >>> a C >>> > >>>> API embedded in an otherwise C++ library >>> > >>>> >>> > https://github.com/wesm/feather/blob/master/cpp/src/feather/feather-c.h >>> . >>> > >>>> >>> > >>>> - Wes >>> > >>>> >>> > >>>> On Wed, Mar 7, 2018 at 10:53 AM, Saul Shanabrook < >>> > s.shanabr...@gmail.com> >>> > >>>> wrote: >>> > >>>> > Hey Wes, >>> > >>>> > >>> > >>>> > Since all our logic for writing and reading the data is written >>> in >>> > C, we >>> > >>>> > need to be get access to the memory in C as well. I think it >>> makes >>> > sense >>> > >>>> > then for me to work on creating a C++ wrapper around Plasma that >>> > exports >>> > >>>> > what is necessary in C. Should that be in Arrow GLib? >>> > >>>> > >>> > >>>> > Best, >>> > >>>> > Saul >>> > >>>> > >>> > >>>> > On Tue, Mar 6, 2018 at 11:49 PM Wes McKinney < >>> wesmck...@gmail.com> >>> > >>>> wrote: >>> > >>>> > >>> > >>>> >> hi Saul -- I think the easiest solution here is the >>> > buffer/memoryview >>> > >>>> >> protocol. You won't have to touch the Cython or C++ API from >>> > pyarrow >>> > >>>> >> if you do this. >>> > >>>> >> >>> > >>>> >> You can interact with a Buffer object like any other Python >>> object >>> > >>>> >> implementing the buffer protocol. See numpy.frombuffer as an >>> > example >>> > >>>> >> of a function that interacts with such objects. I would suggest >>> > adding >>> > >>>> >> a method to xnd method for this. >>> > >>>> >> >>> > >>>> >> If you need to interact with Plasma from C then things will be >>> more >>> > >>>> >> complicated -- Robert or Philipp should be able to advise in >>> this >>> > >>>> >> case. >>> > >>>> >> >>> > >>>> >> - Wes >>> > >>>> >> >>> > >>>> >> On Tue, Mar 6, 2018 at 4:55 PM, Saul Shanabrook < >>> > >>>> s.shanabr...@gmail.com> >>> > >>>> >> wrote: >>> > >>>> >> > Hey Wes, >>> > >>>> >> > >>> > >>>> >> > I don't have much experience doing C + Python + Cython >>> > development, >>> > >>>> so I >>> > >>>> >> am >>> > >>>> >> > probably missing something obvious, but reading the Cython >>> docs, >>> > >>>> >> > < >>> > >>>> >> >>> > >>>> >>> > https://cython.readthedocs.io/en/latest/src/userguide/extern >>> al_C_code.html#public-declarations >>> > >>>> >> > >>> > >>>> >> > it >>> > >>>> >> > seems like I can only access types marked as public from C >>> code. >>> > >>>> When I >>> > >>>> >> > compile arrow locally, I do get some C++ headers for the >>> plasma >>> > code, >>> > >>>> >> but I >>> > >>>> >> > don't think I can use them from C code either. >>> > >>>> >> > >>> > >>>> >> > Best, >>> > >>>> >> > Saul >>> > >>>> >> > >>> > >>>> >> > >>> > >>>> >> > >>> > >>>> >> > On Tue, Mar 6, 2018 at 3:12 PM Wes McKinney < >>> wesmck...@gmail.com >>> > > >>> > >>>> wrote: >>> > >>>> >> > >>> > >>>> >> >> hi Saul, >>> > >>>> >> >> >>> > >>>> >> >> Are you able to use the buffer/memoryview protocol? >>> Instances of >>> > >>>> >> >> pyarrow.Buffer, like PlasmaBuffer, support this >>> > >>>> >> >> >>> > >>>> >> >> >>> > >>>> >> >>> > >>>> >>> > https://github.com/apache/arrow/blob/master/python/pyarrow/p >>> lasma.pyx#L182 >>> > >>>> >> >> >>> > >>>> >> >> - Wes >>> > >>>> >> >> >>> > >>>> >> >> On Tue, Mar 6, 2018 at 3:09 PM, Saul Shanabrook < >>> > >>>> s.shanabr...@gmail.com >>> > >>>> >> > >>> > >>>> >> >> wrote: >>> > >>>> >> >> > I am trying to use the Plasma store to back xnd objects. >>> Xnd ( >>> > >>>> >> >> > https://xnd.readthedocs.io/en/latest/xnd/index.html) is a >>> > >>>> container >>> > >>>> >> >> library >>> > >>>> >> >> > in C that has Python bindings. I would like to get a >>> pointer >>> > to >>> > >>>> the >>> > >>>> >> >> > allocated memory after creating or get an object in >>> Plasma. I >>> > see >>> > >>>> that >>> > >>>> >> >> this >>> > >>>> >> >> > is supported in the C++ API ( >>> > >>>> >> >> > >>> > >>>> >> >> >>> > >>>> >> >>> > >>>> >>> > https://arrow.apache.org/docs/cpp/classplasma_1_1_plasma_cli >>> ent.html#ac18ab9cc792c620a97a3dcb165e0ecd7 >>> > >>>> >> >> ) >>> > >>>> >> >> > but not in the python API (as far as I can tell). Is it >>> > possible >>> > >>>> to >>> > >>>> >> use >>> > >>>> >> >> the >>> > >>>> >> >> > C++ Plasma API from a C project? If not, would it make >>> sense >>> > to >>> > >>>> expose >>> > >>>> >> >> > pointer access on the Python API using capsules >>> > >>>> >> >> > https://docs.python.org/3.6/c-api/capsule.html >>> > >>>> >> >> > <https://docs.python.org/3.6/c-api/capsule.html#capsules>? >>> > >>>> >> >> >>> > >>>> >> >>> > >>>> >>> > >>> >>> > >>> >> >>
