Where are you going to make Plasma requests, in Python or in C? * If in Python, use the buffer protocol to access the memory in C https://docs.python.org/3/c-api/buffer.html#bufferobjects * If in C, then I would suggest adding extern "C" APIs directly in libplasma.so so you can connect to the Plasma store and make requests directly in C.
I want to make 100% sure that you need to interact with the Plasma client in C before I send you down that path. Here's an example of a C API embedded in an otherwise C++ library https://github.com/wesm/feather/blob/master/cpp/src/feather/feather-c.h. - Wes On Wed, Mar 7, 2018 at 10:53 AM, Saul Shanabrook <s.shanabr...@gmail.com> wrote: > Hey Wes, > > Since all our logic for writing and reading the data is written in C, we > need to be get access to the memory in C as well. I think it makes sense > then for me to work on creating a C++ wrapper around Plasma that exports > what is necessary in C. Should that be in Arrow GLib? > > Best, > Saul > > On Tue, Mar 6, 2018 at 11:49 PM Wes McKinney <wesmck...@gmail.com> wrote: > >> hi Saul -- I think the easiest solution here is the buffer/memoryview >> protocol. You won't have to touch the Cython or C++ API from pyarrow >> if you do this. >> >> You can interact with a Buffer object like any other Python object >> implementing the buffer protocol. See numpy.frombuffer as an example >> of a function that interacts with such objects. I would suggest adding >> a method to xnd method for this. >> >> If you need to interact with Plasma from C then things will be more >> complicated -- Robert or Philipp should be able to advise in this >> case. >> >> - Wes >> >> On Tue, Mar 6, 2018 at 4:55 PM, Saul Shanabrook <s.shanabr...@gmail.com> >> wrote: >> > Hey Wes, >> > >> > I don't have much experience doing C + Python + Cython development, so I >> am >> > probably missing something obvious, but reading the Cython docs, >> > < >> https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#public-declarations >> > >> > it >> > seems like I can only access types marked as public from C code. When I >> > compile arrow locally, I do get some C++ headers for the plasma code, >> but I >> > don't think I can use them from C code either. >> > >> > Best, >> > Saul >> > >> > >> > >> > On Tue, Mar 6, 2018 at 3:12 PM Wes McKinney <wesmck...@gmail.com> wrote: >> > >> >> hi Saul, >> >> >> >> Are you able to use the buffer/memoryview protocol? Instances of >> >> pyarrow.Buffer, like PlasmaBuffer, support this >> >> >> >> >> https://github.com/apache/arrow/blob/master/python/pyarrow/plasma.pyx#L182 >> >> >> >> - Wes >> >> >> >> On Tue, Mar 6, 2018 at 3:09 PM, Saul Shanabrook <s.shanabr...@gmail.com >> > >> >> wrote: >> >> > I am trying to use the Plasma store to back xnd objects. Xnd ( >> >> > https://xnd.readthedocs.io/en/latest/xnd/index.html) is a container >> >> library >> >> > in C that has Python bindings. I would like to get a pointer to the >> >> > allocated memory after creating or get an object in Plasma. I see that >> >> this >> >> > is supported in the C++ API ( >> >> > >> >> >> https://arrow.apache.org/docs/cpp/classplasma_1_1_plasma_client.html#ac18ab9cc792c620a97a3dcb165e0ecd7 >> >> ) >> >> > but not in the python API (as far as I can tell). Is it possible to >> use >> >> the >> >> > C++ Plasma API from a C project? If not, would it make sense to expose >> >> > pointer access on the Python API using capsules >> >> > https://docs.python.org/3.6/c-api/capsule.html >> >> > <https://docs.python.org/3.6/c-api/capsule.html#capsules>? >> >> >>
