Yeah, I am writing C++ code that provides a C wrapper around this. But, how do I access either the protected mutable_data_ or the inline mutable_data from my this wrapper?
On Tue, Mar 27, 2018 at 10:52 AM Wes McKinney <wesmck...@gmail.com> wrote: > @Kou, we removed that bit, so Buffer::mutable_data is always inline now. > > @Saul, you will want to create a C wrapper for arrow::Buffer so you > can manage memory lifetime from C. Depending on the kind of Buffer you > have, it may not be as simple as calling free(...) on its internal > data. > > - Wes > > On Tue, Mar 27, 2018 at 10:31 AM, Kouhei Sutou <k...@clear-code.com> wrote: > > Hi, > > > > It may be wrong but you may need to add -DNDEBUG flag to C > > compiler. > > > > > > Thanks, > > -- > > kou > > > > In <CANapY_LzahViR9e7e90H=wq3j0tnbZJe1Vm=j7=vvyqawnc...@mail.gmail.com> > > "Re: Using Plasma with xnd" on Tue, 27 Mar 2018 13:41:49 +0000, > > Saul Shanabrook <s.shanabr...@gmail.com> wrote: > > > >> Ah, I see I can use garrow_buffer_get_mutable_data from c_glib. > >> > >> On Tue, Mar 27, 2018 at 9:29 AM Saul Shanabrook <s.shanabr...@gmail.com > > > >> wrote: > >> > >>> I am in the process of changing over my C wrapper to work with Arrow > >>> 0.9.0. One change is that the plasma APIs now use arrow::Buffer > instead of > >>> of raw pointers, in Create and Get. > >>> > >>> In my C wrapper for Create, I need to get the mutable data pointer from > >>> the arrow::Buffer, after it is created. Here is my current > implementation: > >>> > https://github.com/plures/pxnd/blob/84e9403ef3811e0893e12076455d7cbc49c39358/libplasma/plasma.cc#L56-L75 > >>> I do something similar to the Python wrapper around plasma: > >>> > https://github.com/apache/arrow/blob/f9f8320339692d4134d1ef42c32cb7c8d547593e/python/pyarrow/plasma.pyx#L300-L307 > >>> > >>> > >>> However, this fails to link, I think because the mutable_data method is > >>> defined in the class and so is inline. So it doesn't show up in the > >>> libarrow.dylib/libarrow.a archive. Let me know if I am missing > something > >>> obvious, this is just my hypothesis for why it isn't working. > >>> > >>> Is there a way for a third party library to get the pointer to the data > >>> from a arrow::Buffer? > >>> > >>> On Wed, Mar 7, 2018 at 10:59 AM Wes McKinney <wesmck...@gmail.com> > wrote: > >>> > >>>> Where are you going to make Plasma requests, in Python or in C? > >>>> > >>>> * If in Python, use the buffer protocol to access the memory in C > >>>> https://docs.python.org/3/c-api/buffer.html#bufferobjects > >>>> * If in C, then I would suggest adding extern "C" APIs directly in > >>>> libplasma.so so you can connect to the Plasma store and make requests > >>>> directly in C. > >>>> > >>>> I want to make 100% sure that you need to interact with the Plasma > >>>> client in C before I send you down that path. Here's an example of a C > >>>> API embedded in an otherwise C++ library > >>>> > https://github.com/wesm/feather/blob/master/cpp/src/feather/feather-c.h. > >>>> > >>>> - Wes > >>>> > >>>> On Wed, Mar 7, 2018 at 10:53 AM, Saul Shanabrook < > s.shanabr...@gmail.com> > >>>> wrote: > >>>> > Hey Wes, > >>>> > > >>>> > Since all our logic for writing and reading the data is written in > C, we > >>>> > need to be get access to the memory in C as well. I think it makes > sense > >>>> > then for me to work on creating a C++ wrapper around Plasma that > exports > >>>> > what is necessary in C. Should that be in Arrow GLib? > >>>> > > >>>> > Best, > >>>> > Saul > >>>> > > >>>> > On Tue, Mar 6, 2018 at 11:49 PM Wes McKinney <wesmck...@gmail.com> > >>>> wrote: > >>>> > > >>>> >> hi Saul -- I think the easiest solution here is the > buffer/memoryview > >>>> >> protocol. You won't have to touch the Cython or C++ API from > pyarrow > >>>> >> if you do this. > >>>> >> > >>>> >> You can interact with a Buffer object like any other Python object > >>>> >> implementing the buffer protocol. See numpy.frombuffer as an > example > >>>> >> of a function that interacts with such objects. I would suggest > adding > >>>> >> a method to xnd method for this. > >>>> >> > >>>> >> If you need to interact with Plasma from C then things will be more > >>>> >> complicated -- Robert or Philipp should be able to advise in this > >>>> >> case. > >>>> >> > >>>> >> - Wes > >>>> >> > >>>> >> On Tue, Mar 6, 2018 at 4:55 PM, Saul Shanabrook < > >>>> s.shanabr...@gmail.com> > >>>> >> wrote: > >>>> >> > Hey Wes, > >>>> >> > > >>>> >> > I don't have much experience doing C + Python + Cython > development, > >>>> so I > >>>> >> am > >>>> >> > probably missing something obvious, but reading the Cython docs, > >>>> >> > < > >>>> >> > >>>> > https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#public-declarations > >>>> >> > > >>>> >> > it > >>>> >> > seems like I can only access types marked as public from C code. > >>>> When I > >>>> >> > compile arrow locally, I do get some C++ headers for the plasma > code, > >>>> >> but I > >>>> >> > don't think I can use them from C code either. > >>>> >> > > >>>> >> > Best, > >>>> >> > Saul > >>>> >> > > >>>> >> > > >>>> >> > > >>>> >> > On Tue, Mar 6, 2018 at 3:12 PM Wes McKinney <wesmck...@gmail.com > > > >>>> wrote: > >>>> >> > > >>>> >> >> hi Saul, > >>>> >> >> > >>>> >> >> Are you able to use the buffer/memoryview protocol? Instances of > >>>> >> >> pyarrow.Buffer, like PlasmaBuffer, support this > >>>> >> >> > >>>> >> >> > >>>> >> > >>>> > https://github.com/apache/arrow/blob/master/python/pyarrow/plasma.pyx#L182 > >>>> >> >> > >>>> >> >> - Wes > >>>> >> >> > >>>> >> >> On Tue, Mar 6, 2018 at 3:09 PM, Saul Shanabrook < > >>>> s.shanabr...@gmail.com > >>>> >> > > >>>> >> >> wrote: > >>>> >> >> > I am trying to use the Plasma store to back xnd objects. Xnd ( > >>>> >> >> > https://xnd.readthedocs.io/en/latest/xnd/index.html) is a > >>>> container > >>>> >> >> library > >>>> >> >> > in C that has Python bindings. I would like to get a pointer > to > >>>> the > >>>> >> >> > allocated memory after creating or get an object in Plasma. I > see > >>>> that > >>>> >> >> this > >>>> >> >> > is supported in the C++ API ( > >>>> >> >> > > >>>> >> >> > >>>> >> > >>>> > https://arrow.apache.org/docs/cpp/classplasma_1_1_plasma_client.html#ac18ab9cc792c620a97a3dcb165e0ecd7 > >>>> >> >> ) > >>>> >> >> > but not in the python API (as far as I can tell). Is it > possible > >>>> to > >>>> >> use > >>>> >> >> the > >>>> >> >> > C++ Plasma API from a C project? If not, would it make sense > to > >>>> expose > >>>> >> >> > pointer access on the Python API using capsules > >>>> >> >> > https://docs.python.org/3.6/c-api/capsule.html > >>>> >> >> > <https://docs.python.org/3.6/c-api/capsule.html#capsules>? > >>>> >> >> > >>>> >> > >>>> > >>> >
