I looked into the Arrow build system some more. It is possible to get the
Python classes generated by adding "--python-out" flag (set to a directory
created for it) to the `${ARROW_PROTOBUF_PROTOC}` command under
`macro(build_substrait)` in `cpp/cmake_modules/ThirdpartyToolchain.cmake`.
However, this makes them available only in the Arrow C++ build whereas for
the current purpose they need to be available in the PyArrow build. The
PyArrow build calls `cmake` on `python/CMakeLists.txt`, which AFAICS has
access to `cpp/cmake_modules`. So, one solution could be to pull
`macro(build_substrait)` into `python/CMakeLists.txt` and call it to
generate the Python protobuf classes under `python/`, making them available
for import by PyArrow code. This would probably be cleaner with some macro
parameters to distinguish between C++ and Python generation.
Does this sound like a reasonable approach?
Yaron.
________________________________
From: Yaron Gvili <rt...@hotmail.com>
Sent: Saturday, July 2, 2022 8:55 AM
To: dev@arrow.apache.org <dev@arrow.apache.org>; Phillip Cloud <
cpcl...@gmail.com>
Subject: Re: accessing Substrait protobuf Python classes from PyArrow
I'm somewhat confused by this answer because I think resolving the issue I
raised does not require any change outside PyArrow. I'll try to explain the
issue differently.
First, let me describe the current situation with Substrait protobuf in
Arrow C++. The Arrow C++ build system handles external projects in
`cpp/cmake_modules/ThirdpartyToolchain.cmake`, and one of these external
projects is "substrait". By default, the build system takes the source code
for "substrait" from `
https://github.com/substrait-io/substrait/archive/${ARROW_SUBSTRAIT_BUILD_VERSION}.tar.gz
` where `ARROW_SUBSTRAIT_BUILD_VERSION` is set in
`cpp/thirdparty/versions.txt`. The source code is check-summed and unpacked
in `substrait_ep-prefix` under the build directory and from this the
protobuf C++ classes are generated in `*.pb.{h,cc}` files in
`substrait_ep-generated` under the build directory. The build system makes
a library using the `*.cc` files and makes the `*.h` files available for
other C++ modules to use.
Setting up the above mechanism did not require any change in the
`substrait-io/substrait` repo, nor any coordination with its authors. What
I'm looking for is a similar build mechanism for PyArrow that builds
Substrait protobuf Python classes and makes them available for use by other
PyArrow modules. I believe this PyArrow build mechanism does not exist
currently and that setting up one would not require any changes outside
PyArrow. I'm asking (1) whether that's indeed the case, (2) whether others
agree this mechanism is needed at least due to the problem I ran into that
I previously described, and (3) for any thoughts about how to set up this
mechanism assuming it is needed.
Weston, perhaps your thinking was that the Substrait protobuf Python
classes need to be built by a repo in the substrait-io space and made
available as a binary+headers package? This can be done but will require
involving Substrait people and appears to be inconsistent with current
patterns in the Arrow build system. Note that for my purposes here, the
Substrait protobuf Python classes will be used for composing or
interpreting a Substrait plan, not for transforming it by an optimizer,
though a Python-based optimizer is a valid use case for them.
Yaron.
________________________________
From: Weston Pace <weston.p...@gmail.com>
Sent: Friday, July 1, 2022 12:42 PM
To: dev@arrow.apache.org <dev@arrow.apache.org>; Phillip Cloud <
cpcl...@gmail.com>
Subject: Re: accessing Substrait protobuf Python classes from PyArrow
Given that Acero does not do any planner / optimizer type tasks I'm
not sure you will find anything like this in arrow-cpp or pyarrow.
What you are describing I sometimes refer to as "plan slicing and
dicing". I have wondered if we will someday need this in Acero but I
fear it is a slippery slope between "a little bit of plan
manipulation" and "a full blown planner" so I've shied away from it.
My first spot to look would be a substrait-python repository which
would belong here: https://github.com/substrait-io
However, it does not appear that such a repository exists. If you're
willing to create one then a quick ask on the Substrait Slack instance
should be enough to get the repository created. Perhaps there is some
genesis of this library in Ibis although I think Ibis would use its
own representation for slicing and dicing and only use Substrait for
serialization.
Once that repository is created pyarrow could probably import it but
unless this plan manipulation makes sense purely from a pyarrow
perspective I would rather prefer that the user application import
both pyarrow and substrait-python independently.
Perhaps @Phillip Cloud or someone from the Ibis space might have some
ideas on where this might be found.
-Weston
On Thu, Jun 30, 2022 at 10:06 AM Yaron Gvili <rt...@hotmail.com> wrote:
Hi,
Is there support for accessing Substrait protobuf Python classes (such
as Plan) from PyArrow? If not, how should such support be added? For
example, should the PyArrow build system pull in the Substrait repo as an
external project and build its protobuf Python classes, in a manner similar
to how Arrow C++ does it?
I'm pondering these questions after running into an issue with code I'm
writing under PyArrow that parses a Substrait plan represented as a
dictionary. The current (and kind of shaky) parsing operation in this code
uses json.dumps() on the dictionary, which results in a string that is
passed to a Cython API that handles it using Arrow C++ code that has access
to Substrait protobuf C++ classes. But when the Substrait plan contains a
bytes-type, json.dump() no longer works and fails with "TypeError: Object
of type bytes is not JSON serializable". A fix for this, and a better way
to parse, is using google.protobuf.json_format.ParseDict() [1] on the
dictionary. However, this invocation requires a second argument, namely a
protobuf message instance to merge with. The class of this message (such as
Plan) is a Substrait protobuf Python class, hence the need to access such
classes from PyArrow.
[1]
https://googleapis.dev/python/protobuf/latest/google/protobuf/json_format.html
Yaron.
