[
https://issues.apache.org/jira/browse/ARROW-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881990#comment-16881990
]
lidavidm edited comment on ARROW-5610 at 7/10/19 12:21 PM:
-----------------------------------------------------------
Right now, if you define an extension type in Java whose type name is not
"arrow.py_extension_type", you have no way of writing the Python equivalent. I
think what's needed is a C++ extension type whose implementation dispatches to
Python callbacks, which can be instantiated and registered with an arbitrary
name. Basically, what Joris suggests with the extension type that can be
parameterized with a name.
I think the implementation would be similar to Flight, where you have a C++
subclass that contains a set of function pointers and a Python object, and
invokes those functions by passing them the Python object and the C++
arguments. The functions would be defined in Cython and take care of bridging
between the two.
I don't think there needs to be a Python-specific registry, just a way to hook
arbitrary Python into the extension type metadata (de)serialization. Right now,
the C++ subclass calls a specific classmethod that tries to unpickle the
metadata, but there's no reason why it has to be pickle.
was (Author: lidavidm):
Right now, if you define an extension type in Java whose type name is not
"arrow.py_extension_type", you have no way of writing the Python equivalent. I
think what's needed is a C++ extension type whose implementation dispatches to
Python callbacks, which can be instantiated and registered with an arbitrary
name. Basically, what Joris suggests with the extension type that can be
parameterized with a name.
> [Python] Define extension type API in Python to "receive" or "send" a foreign
> extension type
> --------------------------------------------------------------------------------------------
>
> Key: ARROW-5610
> URL: https://issues.apache.org/jira/browse/ARROW-5610
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Reporter: Wes McKinney
> Priority: Major
> Fix For: 1.0.0
>
>
> In work in ARROW-840, a static {{arrow.py_extension_type}} name is used.
> There will be cases where an extension type is coming from another
> programming language (e.g. Java), so it would be useful to be able to "plug
> in" a Python extension type subclass that will be used to deserialize the
> extension type coming over the wire. This has some different API requirements
> since the serialized representation of the type will not have knowledge of
> Python pickling, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
