[jira] [Created] (ARROW-4982) [GLib][CI] Run tests on AppVeyor

2019-03-20 Thread Kouhei Sutou (JIRA)
Kouhei Sutou created ARROW-4982: --- Summary: [GLib][CI] Run tests on AppVeyor Key: ARROW-4982 URL: https://issues.apache.org/jira/browse/ARROW-4982 Project: Apache Arrow Issue Type: Test

[jira] [Created] (ARROW-4981) [Ruby] Add support for CSV data encoding conversion

2019-03-20 Thread Kouhei Sutou (JIRA)
Kouhei Sutou created ARROW-4981: --- Summary: [Ruby] Add support for CSV data encoding conversion Key: ARROW-4981 URL: https://issues.apache.org/jira/browse/ARROW-4981 Project: Apache Arrow Issue

[jira] [Created] (ARROW-4980) [GLib] Use GInputStream as the parent of GArrowInputStream

2019-03-20 Thread Kouhei Sutou (JIRA)
Kouhei Sutou created ARROW-4980: --- Summary: [GLib] Use GInputStream as the parent of GArrowInputStream Key: ARROW-4980 URL: https://issues.apache.org/jira/browse/ARROW-4980 Project: Apache Arrow

[jira] [Created] (ARROW-4979) [GLib] Add missing lock to garrow::GIOInputStream

2019-03-20 Thread Kouhei Sutou (JIRA)
Kouhei Sutou created ARROW-4979: --- Summary: [GLib] Add missing lock to garrow::GIOInputStream Key: ARROW-4979 URL: https://issues.apache.org/jira/browse/ARROW-4979 Project: Apache Arrow Issue Ty

Re: MemoryPool in Arrow libraries

2019-03-20 Thread Micah Kornfield
Hi Eric, I'm not an expert on most of this, but hopefully this will add a little bit of context. My sense is that the Java implementation is farther along for "real world" use-cases because it is used by Dremio and Drill in distributed analytics engines, where they've had to solve find grained mem

[jira] [Created] (ARROW-4978) [Ruby] Fix wrong internal variable name for table data

2019-03-20 Thread Kouhei Sutou (JIRA)
Kouhei Sutou created ARROW-4978: --- Summary: [Ruby] Fix wrong internal variable name for table data Key: ARROW-4978 URL: https://issues.apache.org/jira/browse/ARROW-4978 Project: Apache Arrow Iss

Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC0

2019-03-20 Thread Brian Hulette
+1 (non-binding) Ran js-verify-release-candidate.sh on Archlinux w/ node v11.12.0 Thanks Krisztian! Brian On Wed, Mar 20, 2019 at 5:40 PM Paul Taylor wrote: > +1 non-binding > > Ran `dev/release/js-verify-release-candidate.sh 0.4.1 0` on MacOS High > Sierra w/ node v11.6.0 > > > On Wed, Mar 20

Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC0

2019-03-20 Thread Paul Taylor
+1 non-binding Ran `dev/release/js-verify-release-candidate.sh 0.4.1 0` on MacOS High Sierra w/ node v11.6.0 On Wed, Mar 20, 2019 at 5:21 PM Kouhei Sutou wrote: > +1 (binding) > > I ran the followings on Debian GNU/Linux sid: > > * dev/release/js-verify-release-candidate.sh 0.4.1 0 > > with:

Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC0

2019-03-20 Thread Kouhei Sutou
+1 (binding) I ran the followings on Debian GNU/Linux sid: * dev/release/js-verify-release-candidate.sh 0.4.1 0 with: * Node.js v11.12.0 Thanks, -- kou In "[VOTE] Release Apache Arrow JS 0.4.1 - RC0" on Thu, 21 Mar 2019 00:09:54 +0100, Krisztián Szűcs wrote: > Hello all, > > I wo

[VOTE] Release Apache Arrow JS 0.4.1 - RC0

2019-03-20 Thread Krisztián Szűcs
Hello all, I would like to propose the following release candidate (rc0) of Apache Arrow JavaScript version 0.4.1. The source release rc0 is hosted at [1]. This release candidate is based on commit f55542eeb59dde8ff4512c707b9eca1b43b62073 Please download, verify checksums and signatures, run th

Re: [DISCUSS] Cutting a JavaScript 0.4.1 bugfix release

2019-03-20 Thread Krisztián Szűcs
No worries, it's fairly straightforward. RC0 is ready, please verify. On Wed, Mar 20, 2019 at 11:35 PM Brian Hulette wrote: > Thanks Wes. > > Krisztian - Uwe cut 0.4.0 for us and said he was pretty comfortable with > the process, so you may be able to defer to him if you don't have time. > > On

Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC0

2019-03-20 Thread Krisztián Szűcs
+1 (binding) I usually don't like to vote on my own release, but ran `dev/release/js-verify-release-candidate.sh 0.4.1 0` with Node v11.12.0 on OSX 10.14.3 and it looks good. On Thu, Mar 21, 2019 at 12:09 AM Krisztián Szűcs wrote: > Hello all, > > I would like to propose the following release c

Re: [DISCUSS] Cutting a JavaScript 0.4.1 bugfix release

2019-03-20 Thread Brian Hulette
Thanks Wes. Krisztian - Uwe cut 0.4.0 for us and said he was pretty comfortable with the process, so you may be able to defer to him if you don't have time. On Wed, Mar 20, 2019 at 3:26 PM Wes McKinney wrote: > It seems based on [1] that we are overdue in cutting a bugfix JS > release because o

[DISCUSS] Cutting a JavaScript 0.4.1 bugfix release

2019-03-20 Thread Wes McKinney
It seems based on [1] that we are overdue in cutting a bugfix JS release because of a problem with the 0.4.0 release on NPM If there are no objections to this I suggest we call a vote right away and close the vote as soon as we have requisite PMC votes. Krisztian, would you be able to help with th

[jira] [Created] (ARROW-4977) [Ruby] Add support for building on Windows

2019-03-20 Thread Kouhei Sutou (JIRA)
Kouhei Sutou created ARROW-4977: --- Summary: [Ruby] Add support for building on Windows Key: ARROW-4977 URL: https://issues.apache.org/jira/browse/ARROW-4977 Project: Apache Arrow Issue Type: Imp

[jira] [Created] (ARROW-4976) [JS] RecordBatchReader should reset its Node/DOM streams

2019-03-20 Thread Paul Taylor (JIRA)
Paul Taylor created ARROW-4976: -- Summary: [JS] RecordBatchReader should reset its Node/DOM streams Key: ARROW-4976 URL: https://issues.apache.org/jira/browse/ARROW-4976 Project: Apache Arrow Iss

[jira] [Created] (ARROW-4975) [C++] Support concatenation of UnionArrays

2019-03-20 Thread Benjamin Kietzman (JIRA)
Benjamin Kietzman created ARROW-4975: Summary: [C++] Support concatenation of UnionArrays Key: ARROW-4975 URL: https://issues.apache.org/jira/browse/ARROW-4975 Project: Apache Arrow Issue

Re: Arrow sync call today 12:00 EDT

2019-03-20 Thread Bryan Cutler
Attendees: Praveen Kumar Siddharth Teotia Bryan Cutler Since there were not many participants this time, we concluded early. On Wed, Mar 20, 2019 at 8:37 AM Wes McKinney wrote: > I'm unable to join today due to a scheduling conflict > > https://meet.google.com/vtm-teks-phx >

Re: Timeline for 0.13 Arrow release

2019-03-20 Thread Brian Hulette
I think that makes sense. I would really like to make JS part of the mainstream releases, but we already have JS-0.4.1 ready to go [1] with primarily bugfixes for JS-0.4.0. I think we should just cut that and integrate JS in 0.14. [1] https://issues.apache.org/jira/projects/ARROW/versions/12344961

Arrow sync call today 12:00 EDT

2019-03-20 Thread Wes McKinney
I'm unable to join today due to a scheduling conflict https://meet.google.com/vtm-teks-phx

Re: Timeline for 0.13 Arrow release

2019-03-20 Thread Wes McKinney
In light of the discussion on https://github.com/apache/arrow/pull/3630 I think we should wait until we have a "not broken" JavaScript-only release on NPM and have confidence that we can respond to the community's needs On Tue, Mar 19, 2019 at 11:24 PM Paul Taylor wrote: > > I agree, the JS has m

[jira] [Created] (ARROW-4974) Array approx equality

2019-03-20 Thread Alexandre Crayssac (JIRA)
Alexandre Crayssac created ARROW-4974: - Summary: Array approx equality Key: ARROW-4974 URL: https://issues.apache.org/jira/browse/ARROW-4974 Project: Apache Arrow Issue Type: Sub-task

[jira] [Created] (ARROW-4971) DataType equality

2019-03-20 Thread Alexandre Crayssac (JIRA)
Alexandre Crayssac created ARROW-4971: - Summary: DataType equality Key: ARROW-4971 URL: https://issues.apache.org/jira/browse/ARROW-4971 Project: Apache Arrow Issue Type: Sub-task

[jira] [Created] (ARROW-4973) Slice Array equality

2019-03-20 Thread Alexandre Crayssac (JIRA)
Alexandre Crayssac created ARROW-4973: - Summary: Slice Array equality Key: ARROW-4973 URL: https://issues.apache.org/jira/browse/ARROW-4973 Project: Apache Arrow Issue Type: Sub-task

[jira] [Created] (ARROW-4972) Array equality

2019-03-20 Thread Alexandre Crayssac (JIRA)
Alexandre Crayssac created ARROW-4972: - Summary: Array equality Key: ARROW-4972 URL: https://issues.apache.org/jira/browse/ARROW-4972 Project: Apache Arrow Issue Type: Sub-task

Re: Page-aligned Data Buffers in Java

2019-03-20 Thread Micah Kornfield
Ahh, sorry that was in the title. I think this might become easier once https://issues.apache.org/jira/browse/ARROW-3191 is complete, which I think is being actively worked on, but others who are more familiar with the Java implementation can chime in. On Wednesday, March 20, 2019, Dimitris Lekka

Re: Building Arrow + Gandiva on Ubuntu Trusty

2019-03-20 Thread Krisztián Szűcs
Hi Praveen, On Wed, Mar 20, 2019 at 1:26 PM Praveen Kumar wrote: > Hi Folks, > > Post the CMake Refactor, we are running into issues in the Gandiva crossbow > jobs. > > We are using conda to setup the dependencies currently. The following are > the issues that we ran into > > 1. I am not sure if

Re: Building Arrow + Gandiva on Ubuntu Trusty

2019-03-20 Thread Praveen Kumar
Sounds great Wes. Thanks a ton. I think we have run into enough issues that moving to the devtoolset version is the right option now. Will try this out and report back soon. On Wed, Mar 20, 2019 at 6:08 PM Wes McKinney wrote: > hi Praveen, > > On Wed, Mar 20, 2019 at 7:26 AM Praveen Kumar wro

[jira] [Created] (ARROW-4970) [C++][Parquet] Implement parquet::FileMetaData::Equals

2019-03-20 Thread Krisztian Szucs (JIRA)
Krisztian Szucs created ARROW-4970: -- Summary: [C++][Parquet] Implement parquet::FileMetaData::Equals Key: ARROW-4970 URL: https://issues.apache.org/jira/browse/ARROW-4970 Project: Apache Arrow

Re: Building Arrow + Gandiva on Ubuntu Trusty

2019-03-20 Thread Wes McKinney
hi Praveen, On Wed, Mar 20, 2019 at 7:26 AM Praveen Kumar wrote: > > Hi Folks, > > Post the CMake Refactor, we are running into issues in the Gandiva crossbow > jobs. > > We are using conda to setup the dependencies currently. The following are > the issues that we ran into > > 1. I am not sure i

Building Arrow + Gandiva on Ubuntu Trusty

2019-03-20 Thread Praveen Kumar
Hi Folks, Post the CMake Refactor, we are running into issues in the Gandiva crossbow jobs. We are using conda to setup the dependencies currently. The following are the issues that we ran into 1. I am not sure if the conda forge packages are compatible in trusty (I was getting issues linking to

[jira] [Created] (ARROW-4969) [C++] Set RPATH in correct order for test executables on OSX

2019-03-20 Thread Krisztian Szucs (JIRA)
Krisztian Szucs created ARROW-4969: -- Summary: [C++] Set RPATH in correct order for test executables on OSX Key: ARROW-4969 URL: https://issues.apache.org/jira/browse/ARROW-4969 Project: Apache Arrow

Re: Page-aligned Data Buffers in Java

2019-03-20 Thread Dimitris Lekkas
This refers to the Java implementation. Our backend is primarily written in Java, thus we decided to prototype our integration in Java. To elucidate, let's assume that we want to create a shared file in memory (e.g /dev/shm) and FPGAs will later DMA this shared memory area. By utilizing the current

Re: Page-aligned Data Buffers in Java

2019-03-20 Thread Micah Kornfield
Hi Dimitris, This sounds interesting. Is this for the C++ implementation? Could you go into a little bit more detail? How would this differ then using a MemoryPool implementation that always aligns to 4KB boundaries (instead of the 64 byte boundaries the default one does today [1])? Thanks, Mic

Page-aligned Data Buffers in Java

2019-03-20 Thread Dimitris Lekkas
Hello folks, I am working at Inaccel where we utilize FPGAs to accelerate machine learning workloads . Recently, we wanted to integrate our platform with Arrow and we stumbled upon the non-alignment of data-buffers to page boundaries (4KB). We implemented the option to supply per-column metadata t

Re: Union typeIds width

2019-03-20 Thread Micah Kornfield
Hi Paul, TL;DR; I think the the typeIds field you referenced is not the offset for dense vectors mentioned by the spec. I believe (but lack the historical context) that it is an outgrowth of the Java implementation that might be useful in other contexts. The requirement is that typeIDs field you

Union typeIds width

2019-03-20 Thread Paul Taylor
I noticed the the DenseUnion docs[1] says the typeIds buffer is 8-bit signed integers, but in the flatbuffer schema[2] it's typed as int (and flatc generates a function that returns an Int32Array). How are the other implementations treating this buffer, and should we update the docs or the fla