Re: Tungsten off heap memory access for C++ libraries

2015-09-01 Thread Paul Weiss
https://issues.apache.org/jira/browse/SPARK-10399 Is the jira to track. On Sep 1, 2015 5:32 PM, "Paul Wais" wrote: > Paul: I've worked on running C++ code on Spark at scale before (via JNA, > ~200 > cores) and am working on something more contribution-oriented now (via > JNI). > A few comments:

Re: Tungsten off heap memory access for C++ libraries

2015-08-31 Thread Paul Weiss
ve a native (e.g. > C++) API for data access shipped with Spark. There are a lot of questions > (e.g. build, portability) that need to be answered. > > On Mon, Aug 31, 2015 at 1:12 AM, Reynold Xin wrote: > >> >> On Sun, Aug 30, 2015 at 5:58 AM, Paul Weiss >> wro

Re: Tungsten off heap memory access for C++ libraries

2015-08-30 Thread Paul Weiss
lso adhere to the >> access contracts as well. >> >> Tim >> >> On Sat, Aug 29, 2015 at 12:17 PM, Paul Weiss >> wrote: >> > Hi, >> > >> > Would the benefits of project tungsten be available for access by >> non-JVM >> > progr

Tungsten off heap memory access for C++ libraries

2015-08-29 Thread Paul Weiss
Hi, Would the benefits of project tungsten be available for access by non-JVM programs directly into the off-heap memory? Spark using dataframes w/ the tungsten improvements will definitely help analytics within the JVM world but accessing outside 3rd party c++ libraries is a challenge especially