[NIGHTLY] Arrow Build Report for Job nightly-2021-05-14-0

2021-05-14 Thread Crossbow
Arrow Build Report for Job nightly-2021-05-14-0 All tasks: https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-05-14-0 Failed Tasks: - conda-osx-clang-py37-r40: URL: https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-05-14-0-azure-conda-osx-clang

[C++][DISCUSS] Implementing interpreted (non-compiled) tests for compute functions

2021-05-14 Thread Wes McKinney
hi folks, As we build more functions (kernels) in the project, I note that the amount of hand-coded C++ code relating to testing function correctness is growing significantly. Many of these tests are quite simple and could be expressed in a text format that can be parsed and evaluated. Thoughts a

Re: [C++][DISCUSS] Implementing interpreted (non-compiled) tests for compute functions

2021-05-14 Thread Antoine Pitrou
Le 14/05/2021 à 15:30, Wes McKinney a écrit : hi folks, As we build more functions (kernels) in the project, I note that the amount of hand-coded C++ code relating to testing function correctness is growing significantly. Many of these tests are quite simple and could be expressed in a text fo

Re: [C++][DISCUSS] Implementing interpreted (non-compiled) tests for compute functions

2021-05-14 Thread Julian Hyde
Do these any of these compute functions have analogs in other implementations of Arrow (e.g. Rust)? I believe that as much as possible of Arrow’s compute functionality should be cross-language. Perhaps there are language-specific differences in how functions are invoked, but the basic functiona

Re: [C++][DISCUSS] Implementing interpreted (non-compiled) tests for compute functions

2021-05-14 Thread Weston Pace
With that in mind it seems the somewhat recurring discussion on coming up with a language independent standard for logical query plans (https://lists.apache.org/thread.html/rfab15e09c97a8fb961d6c5db8b2093824c58d11a51981a40f40cc2c0%40%3Cdev.arrow.apache.org%3E) would be relevant. Each test case wou

Re: [C++][DISCUSS] Implementing interpreted (non-compiled) tests for compute functions

2021-05-14 Thread Wes McKinney
Regardless of the amount of work involved with writing a test harness/driver (which could be mostly in Python as one possibility), I find it hard to believe that writing functional tests that look like this->CheckUnary("utf8_reverse", "[\"ɑɽⱤæÆ&\", null, \"\", \"bbb\"]", this->type(), "[\

Re: [C++][DISCUSS] Implementing interpreted (non-compiled) tests for compute functions

2021-05-14 Thread Eduardo Ponce
Another aspect to keep in mind is that some tests require internal options to be changed before executing the compute functions (e.g., check overflow, allow NaN comparisons, change validity bits, etc.). Also, there are tests that take randomized inputs and others make use of the min/max values for

Re: [C++][DISCUSS] Implementing interpreted (non-compiled) tests for compute functions

2021-05-14 Thread David Li
I think even if it's not (easily) generalizable across languages, it'd still be a win for C++ (and hopefully languages that bind to C++). Also, I don't think they're meant to completely replace language-specific tests, but rather complement them, and make it easier to add and maintain tests in the

Re: [C++][DISCUSS] Implementing interpreted (non-compiled) tests for compute functions

2021-05-14 Thread Jorge Cardoso Leitão
Hi, (this problem also exists in Rust, btw) Couldn't we use something like we do for our integration tests? Create a separate binary that would allow us to call e.g. test-compute --method equal --json-file --arg "column1" --arg "column2" --expected "column3" (or simply pass the input via stdin

Re: [DISCUSS] 4.0.1 patch release?

2021-05-14 Thread Jorge Cardoso Leitão
Just to make sure: the goal is to cherry-pick all changes targeted for 4.0.1 into a branch and release from there? If that is the case, then I will create a branch and start cherry-picking the changes in order they were merged in master. I see 5 issues on the list still open. I subscribed to them

Re: [C++][DISCUSS] Implementing interpreted (non-compiled) tests for compute functions

2021-05-14 Thread Wes McKinney
In C++, we have the "ArrayFromJSON" function which is an even simpler way of specifying input data compared with the integration tests. That's one possible starting point. The "interpreted tests" could be all specified and driven by minimal dependency Python code, as one possible way to approach t

Re: [DISCUSS] 4.0.1 patch release?

2021-05-14 Thread Neal Richardson
Thanks for taking this on. Krisztián can confirm the details (or point you to where this is documented), but based on past patch releases, I believe you would make a `maint-4.0.x` branch off of the existing `release-4.0.0` branch, cherry-pick the commits associated with the JIRAs tagged for 4.0.1 (

Re: [DISCUSS] 4.0.1 patch release?

2021-05-14 Thread Jorge Cardoso Leitão
Hi, I have started collecting commits to the maint branch [1]. The exact commands I used: git clone g...@github.com:apache/arrow.git cd arrow/dev python3 -m venv venv source venv/bin/activate pip install -e archery pip install GitPython jira semver jinja2 archery release cherry-pick 4.0.1 # ran t