It looks like the main decision to make is whether accessing Substrait protobuf
Python classes from PyArrow is what we want here, because it would require some
effort to implement as discussed.
> These are two reasons to need the protobuf in python. However, I still don't
> fully understand wh
Hi Yaron,
Le 07/07/2022 à 10:48, Yaron Gvili a écrit :
It looks like the main decision to make is whether accessing Substrait protobuf
Python classes from PyArrow is what we want here, because it would require some
effort to implement as discussed.
I'm not the best person to comment on thi
> However, from my understanding, the implementation effort is quite
> manageable. Please correct me if I'm wrong,..
Yes, this was discussed earlier in this thread. The protobuf compiler would
need to be ran by the PyArrow build; currently, only the Arrow C++ build runs
it. There would be a bit
I'll go ahead and set up arrow-nanoarrow for convenience.
In the medium term we should think about whether arrow-adbc and arrow-nanoarrow
should be folded back into the arrow monorepo, in order to potentially reduce
the release/CI maintenance burden, or document why we've chosen to split those
Now at https://github.com/apache/arrow-nanoarrow
Dewey: you can use .asf.yaml to enable issues and such:
https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-GitHubsettings
On Thu, Jul 7, 2022, at 09:06, David Li wrote:
> I'll go ahead and set up arrow-
Hello,
I wonder if we have nightly source tarball published somewhere?
Li
Does it work to just use the HEAD [1]?
Otherwise, the nightly CI jobs do generate releases, but I don't think there's
much of a guarantee around the naming/format being stable [2].
I'm not sure if there's an explicit/dedicated "nightly source release" (I'm not
aware of one), or how that would d
Thanks David - I am currently using [1]. A slight issue is that I don't
have the corresponding sdist for "pyarrow" so I'm wondering if there is an
easy way for me to grab matching [1] source and pyarrow source tarball.
Not a big issue, I can create the pyarrow dist from [1]
On Thu, Jul 7, 2022 at
Ah.
Hmm, while the Crossbow releases do include wheels, they don't include the
source (the source link there is for the CI scripts, not Arrow itself). I'm not
sure if there's anything else that packages the Python sources.
On Thu, Jul 7, 2022, at 11:10, Li Jin wrote:
> Thanks David - I am curr
We do upload nightly wheels to an alternative PyPI index (see
https://arrow.apache.org/docs/python/install.html#installing-nightly-packages),
at https://pypi.fury.io/arrow-nightlies/pyarrow, and it seems we actually
also upload an sdist there.
(it could still be more reliable to used HEAD, though,
Thanks both!
This looks useful - I wonder if the version between the crossbow tarball
and pypi dist matches (i.e. are they published together)? i.e. the latest
crossbow source is "Nightly-test-2022-07-07" and the pypi fury is
"pyarrow-9.0.0.dev341.tar.gz"
On Thu, Jul 7, 2022 at 11:19 AM Joris V
Hello Li,
The jobs that create the wheels and sdist a part of the nightly packaging
builds [1].
Within the log [2] you can check the exact sha the wheels/sdist are built
with and use https://github.com/apache/arrow/tarball/ to get the
matching source tarball.
Best
Jacob
[1]:
https://github.com/u
The crossbow tarballs do not contain the arrow source, they only contain
the crossbow source (aka a few yaml files).
On Thu, Jul 7, 2022 at 5:29 PM Li Jin wrote:
> Thanks both!
>
> This looks useful - I wonder if the version between the crossbow tarball
> and pypi dist matches (i.e. are they pub
Thanks! This is very helpful.
On Thu, Jul 7, 2022 at 11:33 AM Jacob Wujciak wrote:
> The crossbow tarballs do not contain the arrow source, they only contain
> the crossbow source (aka a few yaml files).
>
> On Thu, Jul 7, 2022 at 5:29 PM Li Jin wrote:
>
> > Thanks both!
> >
> > This looks use
Hello,
I am trying to build Arrow/Pyarrow with our internal build system (cmake
based) and encounter and error when running pyarrow test:
ImportError while importing test module
'/home/ljin/vats/add-arrowpython-master/ext/public/python/pyarrow/master/dist/lib/python3.9/pyarrow/tests/test_table.py
I don't think you need anything more on the PyArrow side, but you need
to (re)compile Arrow C++ with ARROW_COMPUTE enabled, is that the case?
Le 07/07/2022 à 22:16, Li Jin a écrit :
Hello,
I am trying to build Arrow/Pyarrow with our internal build system (cmake
based) and encounter and e
Thanks Antoine! I got this fixed (although it is not the ARROW_COMPUTE flag
but some other error I made)
Li
On Thu, Jul 7, 2022 at 4:23 PM Antoine Pitrou wrote:
>
> I don't think you need anything more on the PyArrow side, but you need
> to (re)compile Arrow C++ with ARROW_COMPUTE enabled, is t
Hi,
How about using arrow::ipc::StreamDecoder instead of
arrow::ipc::StreamReader?
class MyListener : public arrow::ipc::Listener {
public:
arrow::Status
OnRecordBatchDecoded(std::shared_ptr record_batch)
override {
ArrowFilter arrow_filter = ArrowFilter(record_batch);
Hi,
Yes. It's zero-copy.
See also the documentation of
arrow::ipc::StreamDecoder::next_required_size():
https://arrow.apache.org/docs/cpp/api/ipc.html#_CPPv4NK5arrow3ipc13StreamDecoder18next_required_sizeEv
Thanks,
--
kou
In
"StreamDecoder zero-copy (?) for pre-framed contiguous Messages" o
Hi,
It seems that there are no objections for these patch releases.
I'll initiate these patch releases. Here is my plan. If
there are any objections/concerns, please share them.
* We'll release 6.0.2, 7.0.1 and 8.0.1
* These releases only include
https://github.com/apache/arrow/pull/1332
20 matches
Mail list logo
