Re: [Discuss][Format][Java] Finalizing Union Types

2019-05-24 Thread Micah Kornfield
Thanks for the responses, I've clipped the questions and provided responses inline. is the proposal that both cpp & java will support only option 2 ? > I guess 1 is a subset of 2 anyway. CPP already supports option 2. I would like to make CPP and java compatible, in a way that this acceptable fo

Re: Should EOS be mandatory for IPC File format?

2019-05-24 Thread John Muehlhausen
I will restructure the existing PR and create new ones (with JIRA) for JS/Java. Just haven't gotten around to it yet. On Wed, May 22, 2019 at 9:10 PM Wes McKinney wrote: > Yes, I think separate JIRA issues for Java and JS would be best. I'd > recommend having one patch for each, so maybe we can

Re: memory mapped IPC File of RecordBatches?

2019-05-24 Thread John Muehlhausen
I'm not seeing it as a bug at this point. I was only using it to convince myself that the batch was zero-copy. As you know I'd like to draw up a proposal for fully pre-allocated but only partially populated batches in order to facilitate low-latency streaming appends. It may come up then? I'm m

[Python] Any reason to exclude __lt__ from ArrayValue ?

2019-05-24 Thread John Muehlhausen
We have __eq__ leaning on as_py() already ... any reason not to have __lt__ ? This makes it possible to use bisect to find slices in ordered data without a __getitem__ wrapper: 1176.0 key=pa.array(['AAPL']) 110.0 print(bisect.bisect_left(batch[3],key[0])) 64.0 print(bisect.bisec

Re: Java/Scala: efficient reading of Parquet into Arrow?

2019-05-24 Thread Arvin
In my opinion, parquet support only read part of the columns, I use alike code below. I think parquet reader only read the corresponding page of specific column of the data block according to meta. If parquet file reader read all the block data, then the meta info is unnecessary and it must filter

doput without ticket?

2019-05-24 Thread ming zhang
hi wonder how to understand current doput() call. the doget() call pass the flightinfo as the parameter. does it makes sense to also pass flightinfo as part of the doput() call? thanks ming

[jira] [Created] (ARROW-5418) [CI][R] Run code coverage and report to codecov.io

2019-05-24 Thread Neal Richardson (JIRA)
Neal Richardson created ARROW-5418: -- Summary: [CI][R] Run code coverage and report to codecov.io Key: ARROW-5418 URL: https://issues.apache.org/jira/browse/ARROW-5418 Project: Apache Arrow I

[jira] [Created] (ARROW-5417) [Website] http://arrow.apache.org doesn't redirect to https

2019-05-24 Thread Neal Richardson (JIRA)
Neal Richardson created ARROW-5417: -- Summary: [Website] http://arrow.apache.org doesn't redirect to https Key: ARROW-5417 URL: https://issues.apache.org/jira/browse/ARROW-5417 Project: Apache Arrow

[jira] [Created] (ARROW-5416) [Website] Add Homebrew to project installation page

2019-05-24 Thread Neal Richardson (JIRA)
Neal Richardson created ARROW-5416: -- Summary: [Website] Add Homebrew to project installation page Key: ARROW-5416 URL: https://issues.apache.org/jira/browse/ARROW-5416 Project: Apache Arrow

[jira] [Created] (ARROW-5415) [Release] Release script should update R version everywhere

2019-05-24 Thread Neal Richardson (JIRA)
Neal Richardson created ARROW-5415: -- Summary: [Release] Release script should update R version everywhere Key: ARROW-5415 URL: https://issues.apache.org/jira/browse/ARROW-5415 Project: Apache Arrow

[jira] [Created] (ARROW-5414) [C++] Using "Ninja" build system generator overrides default Release build type on Windows

2019-05-24 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-5414: --- Summary: [C++] Using "Ninja" build system generator overrides default Release build type on Windows Key: ARROW-5414 URL: https://issues.apache.org/jira/browse/ARROW-5414

[jira] [Created] (ARROW-5413) [C++] CSV reader doesn't remove BOM

2019-05-24 Thread Igor Yastrebov (JIRA)
Igor Yastrebov created ARROW-5413: - Summary: [C++] CSV reader doesn't remove BOM Key: ARROW-5413 URL: https://issues.apache.org/jira/browse/ARROW-5413 Project: Apache Arrow Issue Type: Improv

[jira] [Created] (ARROW-5412) [Java] Integration test fails with UnsupportedOperationException

2019-05-24 Thread Benjamin Kietzman (JIRA)
Benjamin Kietzman created ARROW-5412: Summary: [Java] Integration test fails with UnsupportedOperationException Key: ARROW-5412 URL: https://issues.apache.org/jira/browse/ARROW-5412 Project: Apach

Re: [Discuss][Format][Java] Finalizing Union Types

2019-05-24 Thread Antoine Pitrou
I don't understand the limitation to different types, so +1 for generalized unions. That said, I don't think it's high-priority either. Regards Antoine. Le 24/05/2019 à 04:17, Micah Kornfield a écrit : > I'd like to bump this thread, to see if anyone has any comments. If nobody > objects I

Re: [Discuss][Format][Java] Finalizing Union Types

2019-05-24 Thread Ravindra Pindikura
Micah, Couple of questions inline : On Tue, May 21, 2019 at 10:21 AM Micah Kornfield wrote: > In the past [1] there hasn't been agreement on the final requirements for > union types. > > Briefly the two approaches that are currently advocated: > 1. Limit unions to only contain one field of e