Re: Adding CudaBuffer to pyarrow

2018-08-28 Thread Wes McKinney
hi Pearu, Sounds good to me; I had wanted to do it already but got caught up working on other things. You'll probably want to create libarrow_gpu.pxd since this is an optional library. Otherwise it should be reasonably straightforward. some relevant issues - https://issues.apache.org/jira/browse

Assembling an initial development roadmap for Arrow in R

2018-08-28 Thread Wes McKinney
hi folks, In light of Romain's progress on setting up an R Arrow library (https://github.com/apache/arrow/pull/2489), I started putting together a Google document to sketch out a general scaffold for R interfaces to the Arrow ecosystem and some of the initial deliverables that would be of high val

[jira] [Created] (ARROW-3138) 'Couldn't deserialize thrift' error when reading large binary column

2018-08-28 Thread Jeremy Heffner (JIRA)
Jeremy Heffner created ARROW-3138: - Summary: 'Couldn't deserialize thrift' error when reading large binary column Key: ARROW-3138 URL: https://issues.apache.org/jira/browse/ARROW-3138 Project: Apache

Adding CudaBuffer to pyarrow

2018-08-28 Thread Pearu Peterson
Hi, Is there any interest in implementing CudaBuffer support in pyarrow? Are there any obstacles that would prevent to do this? I would be willing to give it a try. As I understand, the entry point for this task would be to patch pyarrow/includes/libarrow.pxd . Any hints for what to take into ac

[jira] [Created] (ARROW-3137) pyarrow 0.10 requires newer version of numpy than specified in requirements

2018-08-28 Thread James Campbell (JIRA)
James Campbell created ARROW-3137: - Summary: pyarrow 0.10 requires newer version of numpy than specified in requirements Key: ARROW-3137 URL: https://issues.apache.org/jira/browse/ARROW-3137 Project:

[jira] [Created] (ARROW-3136) [C++] Clean up arrow:: public API

2018-08-28 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-3136: --- Summary: [C++] Clean up arrow:: public API Key: ARROW-3136 URL: https://issues.apache.org/jira/browse/ARROW-3136 Project: Apache Arrow Issue Type: Improvement

Re: [DISCUSS] Standardize Java style

2018-08-28 Thread Julian Hyde
My two cents: it’s much, much more important to have a standard style (enforced automatically) than what that style is. People should come into this expecting to compromise their personal preferences. > On Aug 28, 2018, at 10:29 AM, Bryan Cutler wrote: > > Sounds good Li. I just wanted to make

Re: [DISCUSS] Standardize Java style

2018-08-28 Thread Bryan Cutler
Sounds good Li. I just wanted to make sure we don't get locked into some style choices just because google does it that way. It's a good starting point though, so lets see how it looks applying the rules as part of ARROW-1688 and I'll raise anything that seems like it needs some discussion. On Tue

[jira] [Created] (ARROW-3135) [C++] Add helper functions for validity bitmap propagation in kernel context

2018-08-28 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-3135: --- Summary: [C++] Add helper functions for validity bitmap propagation in kernel context Key: ARROW-3135 URL: https://issues.apache.org/jira/browse/ARROW-3135 Project: Apa

[jira] [Created] (ARROW-3134) [C++] Implement n-ary iterator for a collection of chunked arrays with possibly different chunking layouts

2018-08-28 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-3134: --- Summary: [C++] Implement n-ary iterator for a collection of chunked arrays with possibly different chunking layouts Key: ARROW-3134 URL: https://issues.apache.org/jira/browse/ARROW-

[jira] [Created] (ARROW-3133) [C++] Logical boolean kernels in kernels/boolean.cc cannot write into preallocated memory

2018-08-28 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-3133: --- Summary: [C++] Logical boolean kernels in kernels/boolean.cc cannot write into preallocated memory Key: ARROW-3133 URL: https://issues.apache.org/jira/browse/ARROW-3133

[jira] [Created] (ARROW-3132) Regenerate 0.10.0 changelog

2018-08-28 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-3132: --- Summary: Regenerate 0.10.0 changelog Key: ARROW-3132 URL: https://issues.apache.org/jira/browse/ARROW-3132 Project: Apache Arrow Issue Type: Bug Comp

Re: How to concatenate RecordBatches into a single RecordBatch?

2018-08-28 Thread Uwe L. Korn
Hello Jacob, while not optimal, you could try to use https://docs.python.org/3/library/io.html#io.BufferedReader together with a much larger buffer_size than the default. This might not be the best way possible as we have to cross the Python/C++ boundary more often but should improve on the cu

Re: [DISCUSS] Standardize Java style

2018-08-28 Thread Li Jin
Bryan, I agree that the some every google style rules might not be the best fit for Arrow project (e.g. import order rules). Like you said, we can use google style as a base line and tweak specific rules if necessary. Li On Mon, Aug 27, 2018 at 9:15 PM Bryan Cutler wrote: > Thanks for bringing

[jira] [Created] (ARROW-3131) [Go] add test for Go-1.11

2018-08-28 Thread Sebastien Binet (JIRA)
Sebastien Binet created ARROW-3131: -- Summary: [Go] add test for Go-1.11 Key: ARROW-3131 URL: https://issues.apache.org/jira/browse/ARROW-3131 Project: Apache Arrow Issue Type: Improvement

[jira] [Created] (ARROW-3130) [Go] add initial support for Go modules

2018-08-28 Thread Sebastien Binet (JIRA)
Sebastien Binet created ARROW-3130: -- Summary: [Go] add initial support for Go modules Key: ARROW-3130 URL: https://issues.apache.org/jira/browse/ARROW-3130 Project: Apache Arrow Issue Type: