[
https://issues.apache.org/jira/browse/ARROW-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386229#comment-16386229
]
Antoine Pitrou commented on ARROW-2195:
---------------------------------------
The detailed explanation here is that the C++ shared_ptr is passed around, not
the Python wrapper. So {{PlasmaBuffer.__dealloc__}} is called even though there
are still C++ exports of the underlying data, particularly in the
{{BatchRecord}} object.
Incidentally, it seems {{ForeignBuffer}} will have the same problem: when the
C++ shared_ptr is passed around, the original Python object won't be kept
alive. Instead, perhaps we need to be able to add destructors to an arbitrary
C++ Buffer object, for example:
{code:c++}
Status Buffer::AddDestructor(std::function<void()>&& destructor);
{code}
Then it will be possible to keep a Python object alive as long as the
corresponding C++ buffer is alive:
{code:c++}
Status AddBufferPayload(std::shared_ptr<Buffer> buffer, PyObject* ref)
{
RETURN_NOT_OK(buffer->AddDestructor([&ref]() { Py_DECREF(ref); }));
Py_INCREF(ref);
return Status::OK();
}
{code}
{code:python}
def foreign_buffer(addr, size, base):
cdef shared_ptr[CBuffer] buffer = make_shared[CBuffer](
<uint8_t*>c_addr, c_size)
AddBufferPayload(buffer, base)
return pyarrow_wrap_buffer(buffer)
{code}
{code:python}
cdef class _PlasmaObjectKeepAlive:
cdef:
ObjectID object_id
PlasmaClient client
def __cinit__(self, ObjectID object_id, PlasmaClient client):
self.object_id = object_id
self.client = client
def __dealloc__(self):
"""
Notify Plasma that the object is no longer needed.
If the plasma client has been shut down, then don't do anything.
"""
self.client.release(self.object_id)
cdef class PlasmaClient:
cdef _make_plasma_buffer(self, ObjectID object_id,
shared_ptr[CBuffer] buffer, int64_t size):
AddBufferPaylod(buffer, _PlasmaObjectKeepAlive(object_id, self)
return pyarrow_wrap_buffer(buffer)
{code}
cc [~xhochy]
> [Plasma] Segfault when retrieving RecordBatch from plasma store
> ---------------------------------------------------------------
>
> Key: ARROW-2195
> URL: https://issues.apache.org/jira/browse/ARROW-2195
> Project: Apache Arrow
> Issue Type: Bug
> Reporter: Philipp Moritz
> Priority: Major
> Fix For: 0.9.0
>
>
> It can be reproduced with the following script:
> {code:python}
> import pyarrow as pa
> import pyarrow.plasma as plasma
> def retrieve1():
> client = plasma.connect('test', "", 0)
> key = "keynumber1keynumber1"
> pid = plasma.ObjectID(bytearray(key,'UTF-8'))
> [buff] = client .get_buffers([pid])
> batch = pa.RecordBatchStreamReader(buff).read_next_batch()
> print(batch)
> print(batch.schema)
> print(batch[0])
> return batch
> client = plasma.connect('test', "", 0)
> test1 = [1, 12, 23, 3, 21, 34]
> test1 = pa.array(test1, pa.int32())
> batch = pa.RecordBatch.from_arrays([test1], ['FIELD1'])
> key = "keynumber1keynumber1"
> pid = plasma.ObjectID(bytearray(key,'UTF-8'))
> sink = pa.MockOutputStream()
> stream_writer = pa.RecordBatchStreamWriter(sink, batch.schema)
> stream_writer.write_batch(batch)
> stream_writer.close()
> bff = client.create(pid, sink.size())
> stream = pa.FixedSizeBufferWriter(bff)
> writer = pa.RecordBatchStreamWriter(stream, batch.schema)
> writer.write_batch(batch)
> client.seal(pid)
> batch = retrieve1()
> print(batch)
> print(batch.schema)
> print(batch[0])
> {code}
>
> Preliminary backtrace:
>
> {code}
> CESS (code=1, address=0x111138158)
> frame #0: 0x000000010e6457fc
> lib.so`__pyx_pw_7pyarrow_3lib_10Int32Value_1as_py(_object*, _object*) + 28
> lib.so`__pyx_pw_7pyarrow_3lib_10Int32Value_1as_py:
> -> 0x10e6457fc <+28>: movslq (%rdx,%rcx,4), %rdi
> 0x10e645800 <+32>: callq 0x10e698170 ; symbol stub for:
> PyInt_FromLong
> 0x10e645805 <+37>: testq %rax, %rax
> 0x10e645808 <+40>: je 0x10e64580c ; <+44>
> (lldb) bt
> * thread #1: tid = 0xf1378e, 0x000000010e6457fc
> lib.so`__pyx_pw_7pyarrow_3lib_10Int32Value_1as_py(_object*, _object*) + 28,
> queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1,
> address=0x111138158)
> * frame #0: 0x000000010e6457fc
> lib.so`__pyx_pw_7pyarrow_3lib_10Int32Value_1as_py(_object*, _object*) + 28
> frame #1: 0x000000010e5ccd35 lib.so`__Pyx_PyObject_CallNoArg(_object*) +
> 133
> frame #2: 0x000000010e613b25
> lib.so`__pyx_pw_7pyarrow_3lib_10ArrayValue_3__repr__(_object*) + 933
> frame #3: 0x000000010c2f83bc libpython2.7.dylib`PyObject_Repr + 60
> frame #4: 0x000000010c35f651 libpython2.7.dylib`PyEval_EvalFrameEx + 22305
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
