[jira] [Created] (ARROW-1962) [Java] Add reset() to ValueVector interface

2018-01-02 Thread Bryan Cutler (JIRA)
Bryan Cutler created ARROW-1962: --- Summary: [Java] Add reset() to ValueVector interface Key: ARROW-1962 URL: https://issues.apache.org/jira/browse/ARROW-1962 Project: Apache Arrow Issue Type: Im

Re: [DRAFT] Apache Arrow board report

2018-01-02 Thread Siddharth Teotia
+1. Thanks, Wes. On Tue, Jan 2, 2018 at 12:10 PM, Holden Karau wrote: > Would it make sense to mention the other Apache projects using/planning to > use Arrow? > > On Tue, Jan 2, 2018 at 11:31 AM Li Jin wrote: > > > +1. Thanks Wes! > > > > On Tue, Jan 2, 2018 at 11:19 AM, Uwe L. Korn wrote: >

Re: [DRAFT] Apache Arrow board report

2018-01-02 Thread Holden Karau
Would it make sense to mention the other Apache projects using/planning to use Arrow? On Tue, Jan 2, 2018 at 11:31 AM Li Jin wrote: > +1. Thanks Wes! > > On Tue, Jan 2, 2018 at 11:19 AM, Uwe L. Korn wrote: > > > +1 > > > > On Tue, Jan 2, 2018, at 4:21 PM, Wes McKinney wrote: > > > Here is a dra

Re: [DRAFT] Apache Arrow board report

2018-01-02 Thread Li Jin
+1. Thanks Wes! On Tue, Jan 2, 2018 at 11:19 AM, Uwe L. Korn wrote: > +1 > > On Tue, Jan 2, 2018, at 4:21 PM, Wes McKinney wrote: > > Here is a draft for this quarter's ASF board report. The Activity / > > Health sections are a bit light on detail, if others would like to add > > some things fee

[jira] [Created] (ARROW-1961) [Python] Writing Parquet file with flavor='spark' loses pandas schema metadata

2018-01-02 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-1961: --- Summary: [Python] Writing Parquet file with flavor='spark' loses pandas schema metadata Key: ARROW-1961 URL: https://issues.apache.org/jira/browse/ARROW-1961 Project: A

Re: Is there any way to get data of numeric array like maximum/minimum value?

2018-01-02 Thread Wes McKinney
We are still in the early stages of defining such APIs. See https://github.com/apache/arrow/tree/master/cpp/src/arrow/compute where this code is being written. We would be glad of any additional work or input on the design for the computation APIs - Wes On Tue, Jan 2, 2018 at 10:43 AM, Jin Hai

[jira] [Created] (ARROW-1960) [Python] Pre-emptively import TensorFlow if it is available

2018-01-02 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-1960: --- Summary: [Python] Pre-emptively import TensorFlow if it is available Key: ARROW-1960 URL: https://issues.apache.org/jira/browse/ARROW-1960 Project: Apache Arrow

Re: Have you considered using unified memory in the memory allocation?

2018-01-02 Thread Wes McKinney
hi Jin -- the idea of using unified memory is interesting. My initial thinking was to make the semantics of host vs. device memory explicit, but it's hard to tell until we see how more GPU users will be using the Arrow libraries. As an initial first step, we could create a Buffer implementation (li

Re: [DRAFT] Apache Arrow board report

2018-01-02 Thread Uwe L. Korn
+1 On Tue, Jan 2, 2018, at 4:21 PM, Wes McKinney wrote: > Here is a draft for this quarter's ASF board report. The Activity / > Health sections are a bit light on detail, if others would like to add > some things feel free to send them along. > > thanks > Wes > > ## Description: > > Apache Arro

[jira] [Created] (ARROW-1959) [Python] Add option for "lossy" conversions (overflow -> null) from timestamps to datetime.datetime / pandas.Timestamp

2018-01-02 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-1959: --- Summary: [Python] Add option for "lossy" conversions (overflow -> null) from timestamps to datetime.datetime / pandas.Timestamp Key: ARROW-1959 URL: https://issues.apache.org/jira/b

Re: Is there any way to get data of numeric array like maximum/minimum value?

2018-01-02 Thread Jin Hai
Cool, do you have any idea how to implement the function in Arrow? Such as create a JIRA ticket and describe the idea? > On 2 Jan 2018, at 11:37 PM, Wes McKinney wrote: > > > Hi Jin, > > If you are reading Parquet files, you can look at the column > statistics for a row group using the C++ A

Re: arrow read/write examples in Java

2018-01-02 Thread Wes McKinney
I'll take a look at updating the site docs today. Thanks for pointing this out! On Wed, Dec 27, 2017 at 4:57 AM, Animesh Trivedi wrote: > Hello everyone, > > I solved the issue with my writer. Now everything is working fine, > including HDFS file reads and writes. I also wrote a parquet to arrow

Re: Is there any way to get data of numeric array like maximum/minimum value?

2018-01-02 Thread Wes McKinney
Hi Jin, If you are reading Parquet files, you can look at the column statistics for a row group using the C++ API. We definitely plan to implement functions to compute reduction-based statistics like minimum and maximum from data already in an Arrow array in C++, but it has not been implemented ye

[DRAFT] Apache Arrow board report

2018-01-02 Thread Wes McKinney
Here is a draft for this quarter's ASF board report. The Activity / Health sections are a bit light on detail, if others would like to add some things feel free to send them along. thanks Wes ## Description: Apache Arrow is a cross-language development platform for in-memory data. It specifies a