Re: I setup a slack team to have a live channel to discuss Arrow

2016-03-11 Thread Venkat Krishnamurthy
Can you please add ven...@cray.com as well? Thanks On Friday, March 11, 2016, Chao Sun wrote: > sunc...@apache.org - please invite. > > On Fri, Mar 11, 2016 at 3:42 PM, Zhe Zhang > > wrote: > > > Please invite z...@apache.org . Thanks. > > > > On Fri, Mar 11, 2016 at 1:25 PM Yiannis Gkoufas >

Distributed arrow?

2016-03-04 Thread Venkat Krishnamurthy
All I've been following along with great interest, and have a n00b question. What happens when any of the arrow-capable processing tools needs to work with a data set or structure that is bigger than a single node's memory capacity? Does arrow itself handle the distribution of the resulting colum

Re: Intel CPU architecture

2016-03-02 Thread Venkat Krishnamurthy
Is JICS the Joint Institute for Comp Sciences at ORNL/UT? If so, is one of the target platforms Titan@ORNL? On Wed, Mar 2, 2016 at 2:56 PM, Edmon Begoli wrote: > Would you guys be interested in perhaps having a Hangout with my team from > JICS/NICS? > > We have some major experts and research th

Re: Comparing with Parquet

2016-02-25 Thread Venkat Krishnamurthy
I think they're fundamentally orthogonal. Tachyon provides a full-fledged storage system that uses memory as one of its tiers, which leaves it up to applications as to how they represent their data structures. When applications want to use Tachyon, it's up to them to decide how to serialize their

RDMA, shared memory etc

2016-02-25 Thread Venkat Krishnamurthy
All We're very interested in exploring how Arrow can be used in traditional scientific computing installations. I've read several initial overviews, and am particularly interested in the mentions of RDMA where available, since that is a standard capability on HPC platforms. Are there specific de