[jira] [Updated] (ARROW-64) Add zsh support to C++ build scripts

2016-03-19 Thread Jacques Nadeau (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-64?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacques Nadeau updated ARROW-64: Assignee: Uwe L. Korn > Add zsh support to C++ build scripts > >

[jira] [Commented] (ARROW-55) Python: fix legacy Python (2.7) tests and add to Travis CI

2016-03-19 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-55?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198291#comment-15198291 ] Wes McKinney commented on ARROW-55: --- [~danrobinson...@gmail.com] i'm unable to assign this

Re: Understanding "shared" memory implications

2016-03-19 Thread Jacques Nadeau
@Todd: agree entirely on prototyping design. My goal is throw out some ideas and some POC code and then we can explore from there. My main thoughts have initially been around lifecycle management. I've done some work previously where a consistently sized shared buffer using mmap has improved perfo

Re: [jira] [Created] (ARROW-68) Update setup_build_env and third-party script to be more userfriendly

2016-03-19 Thread Wes McKinney
hello, You can unsubscribe from dev@arrow.apache.org by sending an email to dev-unsubscr...@apache.arrow.org However: if you are not interested in keeping an eye on the engineering work that is being done on the project (the JIRA updates), consider setting up and email filter matching on "[jira]"

Re: [jira] [Created] (ARROW-68) Update setup_build_env and third-party script to be more userfriendly

2016-03-19 Thread Wes McKinney
Sorry, that is dev-unsubscr...@arrow.apache.org On Thu, Mar 17, 2016 at 9:12 AM, Wes McKinney wrote: > hello, > > You can unsubscribe from dev@arrow.apache.org by sending an email to > dev-unsubscr...@apache.arrow.org > > However: if you are not interested in keeping an eye on the > engineering w

Re: Understanding "shared" memory implications

2016-03-19 Thread Julian Hyde
This is all very interesting stuff, but just so we’re clear: it is not Arrow’s responsibility to provide an RPC/IPC/LPC mechanism, nor facilities for resource management. If we DID decide to make this Arrow’s responsibility it would overlap with other components which specialize in such stuff.

[jira] [Resolved] (ARROW-64) Add zsh support to C++ build scripts

2016-03-19 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-64?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney resolved ARROW-64. --- Resolution: Fixed Issue resolved by pull request 24 [https://github.com/apache/arrow/pull/24] > Add zsh

Re: Understanding "shared" memory implications

2016-03-19 Thread Reynold Xin
I always thought Arrow was just an in-memory format, and it is the responsibility of whoever else that want to use it to carry that responsibilities out, because depending on workloads, different frameworks might pick very different applications. Otherwise it seems to be doing too much and having t

[jira] [Created] (ARROW-67) C++: Draft type metadata conversion to/from IPC representation

2016-03-19 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-67: - Summary: C++: Draft type metadata conversion to/from IPC representation Key: ARROW-67 URL: https://issues.apache.org/jira/browse/ARROW-67 Project: Apache Arrow Is

[jira] [Updated] (ARROW-66) Maybe some missing steps in installation guide

2016-03-19 Thread Qian Xu (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-66?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qian Xu updated ARROW-66: - Attachment: (was: ARROW-66.patch) > Maybe some missing steps in installation guide > --

Re: Understanding "shared" memory implications

2016-03-19 Thread Zhe Zhang
I have similar concerns as Todd stated below. With an mmap-based approach, we are treating shared memory objects like files. This brings in all filesystem related considerations like ACL and lifecycle mgmt. Stepping back a little, the shared-memory work isn't really specific to Arrow. A few questi

Re: Understanding "shared" memory implications

2016-03-19 Thread Julian Hyde
Calcite is a salutary example if what happens if you *don’t* figure out early enough what is core and what is not. You’re hardly the biggest fan of the bundled default execution implementation. At your bidding, we’ve been trying for almost 2 years to get that stuff out of core. Arrow is, at its

RE: Understanding "shared" memory implications

2016-03-19 Thread Zheng, Kai
Sounds good to have all these compatible, modular goals and changes, for Apache Arrow, in the early stage. On the other hand, not being afraid of changes keep evolving towards the core goals and the well-defined initiatives, which is also important. Parquet is relevant and also a good example.

[jira] [Commented] (ARROW-64) Add zsh support to C++ build scripts

2016-03-19 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-64?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198295#comment-15198295 ] Wes McKinney commented on ARROW-64: --- [~xhochy] I am unable to assign this to you, so feel

[jira] [Resolved] (ARROW-55) Python: fix legacy Python (2.7) tests and add to Travis CI

2016-03-19 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-55?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney resolved ARROW-55. --- Resolution: Fixed Issue resolved by pull request 25 [https://github.com/apache/arrow/pull/25] > Python:

Re: Understanding "shared" memory implications

2016-03-19 Thread Jacques Nadeau
>>You’re hardly the biggest fan of the bundled default execution implementation. At your bidding, we’ve been trying for almost 2 years to get that stuff out of core. Great point. As you stated, I think there are at least two lessons with Calcite: 1. Make sure to have an easy to use out of the box

[jira] [Commented] (ARROW-28) C++: Add google/benchmark to the 3rd-party build toolchain

2016-03-19 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-28?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198504#comment-15198504 ] Wes McKinney commented on ARROW-28: --- Please do! We'll probably want to start a little benc

[jira] [Commented] (ARROW-64) Add zsh support to C++ build scripts

2016-03-19 Thread Jacques Nadeau (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-64?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198301#comment-15198301 ] Jacques Nadeau commented on ARROW-64: - [~wesmckinn]: people need to have the role of con

[jira] [Updated] (ARROW-69) Change permissions for assignable users

2016-03-19 Thread Dan Robinson (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-69?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Robinson updated ARROW-69: -- Description: Per comments from [~wesmckinn] and [~jnadeau] on https://issues.apache.org/jira/browse/ARROW

Re: Understanding "shared" memory implications

2016-03-19 Thread Jacques Nadeau
I think it is okay for a project to be different things to different people. I think it is really important as a library that we have enough supporting examples that people can get started quickly. In some sense I'm modeling this after what Julian did with Calcite. For example he provides a defau

Re: Understanding "shared" memory implications

2016-03-19 Thread Leif Walsh
Seems to me IPC/LPC/RPC focuses on the wrong distinction. I think the right one is between async message-passing (over a socket), where the receiver decides when to handle the message, and synchronous/direct memory manipulation (shared mmap, rdma), where the "client" manipulates the "server's" (rat

Re: Understanding "shared" memory implications

2016-03-19 Thread Wes McKinney
It has always been the expectation that no system would be required to use a particular piece of Arrow software to "use Arrow" (hence the importance of having a well-defined specification for memory and metadata). However, we should also not expect all systems to create their own implementations of

Re: [jira] [Created] (ARROW-68) Update setup_build_env and third-party script to be more userfriendly

2016-03-19 Thread EL MEHDI EL JAIR
Hi, I want to unsubscribe to this forum, help please, Thanks, 2016-03-17 13:35 GMT+00:00 J. Albert Bowden : > how do i unsubscribe from this? > > sorry, went to issues.apache.org > > says i don't have an account, can't recover with this email address > > not sure how i got on here, but i would

Re: Contribution etiquette?

2016-03-19 Thread Wes McKinney
hi Micah, welcome! There is no set process for assigning the JIRAs. 1) If you want to do something (or already did something) with no JIRA, feel free to open one and self-assign 2) If there is an unassigned JIRA, feel free to claim it 3) If you wish to work on an assigned JIRA that someone else

Re: Understanding "shared" memory implications

2016-03-19 Thread Wes McKinney
On Wed, Mar 16, 2016 at 2:33 PM, Jacques Nadeau wrote: > > For Arrow, let's make sure that we do our best to accomplish both (1) and > (2). They seem like entirely compatible goals. > > For my part on the C++ side, I plan to proceed with a hub-and-spoke model. A minimal small core library with "l

Re: Understanding "shared" memory implications

2016-03-19 Thread Corey Nolet
I've been under the impression that exposing memory to be shared directly and not copied WAS, in fact, the responsibility of Arrow. In fact, I read this in [1] and this is turned me on to Arrow in the first place. [1] http://www.datanami.com/2016/02/17/arrow-aims-to-defrag-big-in-memory-analytics

[jira] [Commented] (ARROW-55) Python: fix legacy Python (2.7) tests and add to Travis CI

2016-03-19 Thread Dan Robinson (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-55?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198122#comment-15198122 ] Dan Robinson commented on ARROW-55: --- Added tests to Travis too! https://github.com/apache/

[jira] [Resolved] (ARROW-68) Update setup_build_env and third-party script to be more userfriendly

2016-03-19 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-68?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney resolved ARROW-68. --- Resolution: Fixed Issue resolved by pull request 27 [https://github.com/apache/arrow/pull/27] > Update

[jira] [Commented] (ARROW-64) Add zsh support to C++ build scripts

2016-03-19 Thread Jacques Nadeau (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-64?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198404#comment-15198404 ] Jacques Nadeau commented on ARROW-64: - Sure. Do you want to open a JIRA to have INFRA se

RE: Understanding "shared" memory implications

2016-03-19 Thread Zheng, Kai
Hi Wes, Thanks for the helpful clarifying and am glad to know you're also well moving on the project. Yes I will look closely to the project, raising my specific comments in the mailing list. Regards, Kai -Original Message- From: Wes McKinney [mailto:w...@cloudera.com] Sent: Friday,

[jira] [Commented] (ARROW-28) C++: Add google/benchmark to the 3rd-party build toolchain

2016-03-19 Thread Micah Kornfield (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-28?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198493#comment-15198493 ] Micah Kornfield commented on ARROW-28: -- I'd be happy to take this, if no one is current

Re: Understanding "shared" memory implications

2016-03-19 Thread Wes McKinney
hi Kai, This sounds like it might merit a separate thread to discuss the growth of Arrow as a modular ecosystem of libraries in different programming languages and related tools (we've discussed shared memory data access and metadata representation, but not questions around ownership and managemen

Re: [jira] [Created] (ARROW-68) Update setup_build_env and third-party script to be more userfriendly

2016-03-19 Thread J. Albert Bowden
how do i unsubscribe from this? sorry, went to issues.apache.org says i don't have an account, can't recover with this email address not sure how i got on here, but i would like to get off sorry again, thanks On Wed, Mar 16, 2016 at 7:58 PM, Micah Kornfield (JIRA) wrote: > Micah Kornfield cr

[jira] [Created] (ARROW-70) C++: Add "lite" DCHECK macros used in parquet-cpp

2016-03-19 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-70: - Summary: C++: Add "lite" DCHECK macros used in parquet-cpp Key: ARROW-70 URL: https://issues.apache.org/jira/browse/ARROW-70 Project: Apache Arrow Issue Type: New

[jira] [Created] (ARROW-68) Update setup_build_env and third-party script to be more userfriendly

2016-03-19 Thread Micah Kornfield (JIRA)
Micah Kornfield created ARROW-68: Summary: Update setup_build_env and third-party script to be more userfriendly Key: ARROW-68 URL: https://issues.apache.org/jira/browse/ARROW-68 Project: Apache Arrow

Re: Contribution etiquette?

2016-03-19 Thread Wes McKinney
Also, if you do have bandwidth and interest in doing code reviews, this is also super helpful, so feel free to chime in on GitHub if you want to do a code review (either on outstanding or merged patches while we are in commit-then-review mode). On Wed, Mar 16, 2016 at 3:08 PM, Wes McKinney wrote: