Re: [ANNOUNCE] New Arrow committers: Dewey Dunnington, Alenka Frim, and Rok Mihevc

2022-06-23 Thread Alenka Frim
Thank you so much! And congrats to my fellow committers ❤ V čet., 23. jun. 2022 02:36 je oseba Wang Xudong napisala: > Congratulations!! > > Vibhatha Abeykoon 于2022年6月23日周四 07:36写道: > > > Congratulations Dewey, Alenka and Rok!!! > > > > On Thu, Jun 23, 2022 at 4:30 AM Matt Topol > wrote: > > >

Re: Arrow FunctionRegsitry usage in Python

2022-06-23 Thread Vibhatha Abeykoon
Hello Mura, You can also take a look at the compute registration example [1], [2] in the Arrow source code to get an idea how to write a custom function and use it in C++. Assuming your objective is to add a new function to C++ permanently and use it in Python, you may need to take a look at the e

Re: user-defined Python-based data-sources in Arrow

2022-06-23 Thread Weston Pace
This seems reasonable to me. A very similar interface is the RecordBatchReader[1] which is roughly (glossing over details)... ``` class RecordBatchReader { virtual std::shared_ptr schema() const = 0; virtual Result> Next() = 0; virtual Status Close() = 0; }; ``` This seems pretty close to

Re: [question] Arrow C GLib conda package

2022-06-23 Thread Sutou Kouhei
Hi, No. The Apache Arrow C GLib package doesn't exist on conda-forge. I think that we need to create https://github.com/conda-forge/arrow-c-glib-feedstock or something like https://github.com/conda-forge/arrow-cpp-feedstock for it. Uwe will help you. Thanks, -- kou In "[question] Arrow C

[question] Arrow C GLib conda package

2022-06-23 Thread Ivan Ogasawara
Hi everyone, Is the Arrow C GLib (https://github.com/apache/arrow/tree/master/c_glib) available on conda-forge? If not, I can contribute to add this package to conda-forge. I would just need some guidance about where it would be the better place to have it. Thanks for the attention, Ivan

Re: Arrow FunctionRegsitry usage in Python

2022-06-23 Thread Aldrin
Hi Mura, The short answer is: yes, it is possible. Naively, to expose such a function something may need to be added in cython. I am learning more about how to add new compute functions, so I am not sure yet if functions in the function registry are automatically exposed to other languages (e.g. R

Re: vectorized processing for arrow::take()

2022-06-23 Thread Aldrin
Without knowing implementation details of the Take function, I know that Arrow uses xsimd and does try to enable the compiler to vectorize code where it can. To clarify, are you asking how to improve the performance you're seeing, or are you asking how to check if the compiled code is using vector

Re: vectorized processing for arrow::take()

2022-06-23 Thread Chak-Pong Chung
correction: not clang, I meant the Vectorizers from LLVM https://llvm.org/docs/Vectorizers.html if we can use it with arrow array On Thu, Jun 23, 2022 at 3:35 PM Chak-Pong Chung wrote: > I asked a question here about vectorized processing. > > > https://stackoverflow.com/questions/72735678/how

vectorized processing for arrow::take()

2022-06-23 Thread Chak-Pong Chung
I asked a question here about vectorized processing. https://stackoverflow.com/questions/72735678/how-to-vectorize-arrowcomputetake Any idea? I am also open to the approaches like Intel MKL, xsimd, clang and so on. -- Regards, Chak-Pong

Re: What is the lib file for Acero/Arrow compute?

2022-06-23 Thread Li Jin
Gotcha. Thanks David! On Thu, Jun 23, 2022 at 11:54 AM David Li wrote: > Yeah, it's just part of libarrow.so. There's a long-standing issue to try > to split out (most of) the compute kernels into a separate library to slim > down the main one, but the issue is that we probably want to keep cast

Re: What is the lib file for Acero/Arrow compute?

2022-06-23 Thread David Li
Yeah, it's just part of libarrow.so. There's a long-standing issue to try to split out (most of) the compute kernels into a separate library to slim down the main one, but the issue is that we probably want to keep casts (and perhaps a few other kernels) in the main library. AIUI, we may also wa

Re: Rust / Arrow sync Call - June 23 2022

2022-06-23 Thread Andy Grove
Thanks for sending out the reminder Andrew. I will try and remember to do that for the upcoming calls. I updated the agenda document with notes on topics discussed. Andy. On Wed, Jun 22, 2022 at 11:12 AM Andrew Lamb wrote: > Andy Grove is hosting the next Rust / Arrow sync call tomorrow. More >

What is the lib file for Acero/Arrow compute?

2022-06-23 Thread Li Jin
Hi, I just noticed there is no specific lib file for Acero/Arrow compute when I have BUILD_COMPUTE=ON - is it included in the libarrow.so? Thanks! Li