Spark 2.0?
Even if the above answers my first question, I'd still like to know if the
new Spark API will allow RDDs to be /filled/ from the C++ side, as a data
source, rather than a derived dataset.
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nab
breaking consideration and I'll start using it for
consistency, maybe even interoperability.
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/Tungsten-off-heap-memory-access-for-C-libraries-tp13898p17387.html
Sent from the Apache Spark
opbox/djinni/tree/master/example/localhost
For the long deets, see:
https://github.com/dropbox/djinni/pull/140
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/Tungsten-off-heap-memory-access-for-C-libraries-tp13898p14427.html
Sent from the Apache
It's a very messy problem :)
>
> Was there indeed a JIRA started to track this issue? Can't find it at the
> moment ...
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Tungsten-off-heap-memory-
dive into messing with (standard) Java String <->
std::string using JNI. It's a very messy problem :)
Was there indeed a JIRA started to track this issue? Can't find it at the
moment ...
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.co
Please do. Thanks.
On Mon, Aug 31, 2015 at 5:00 AM, Paul Weiss wrote:
> Sounds good, want me to create a jira and link it to SPARK-9697? Will put
> down some ideas to start.
> On Aug 31, 2015 4:14 AM, "Reynold Xin" wrote:
>
>> BTW if you are interested in this, we could definitely get some help
Sounds good, want me to create a jira and link it to SPARK-9697? Will put
down some ideas to start.
On Aug 31, 2015 4:14 AM, "Reynold Xin" wrote:
> BTW if you are interested in this, we could definitely get some help in
> terms of prototyping the feasibility, i.e. how we can have a native (e.g.
>
BTW if you are interested in this, we could definitely get some help in
terms of prototyping the feasibility, i.e. how we can have a native (e.g.
C++) API for data access shipped with Spark. There are a lot of questions
(e.g. build, portability) that need to be answered.
On Mon, Aug 31, 2015 at 1:
On Sun, Aug 30, 2015 at 5:58 AM, Paul Weiss wrote:
>
> Also, is this work being done on a branch I could look into further and
> try out?
>
>
We don't have a branch yet -- because there is no code nor design for this
yet. As I said, it is one of the motivations behind Tungsten, but it is
fairly e
Reynold,
That is great to hear. Definitely interested in how 2. is being
implemented and how it will be exposed in C++. One important aspect of
leveraging the off heap memory is how the data is organized as well as
being able to easily access it from the C++ side. For example how would
you stor
Supporting non-JVM code without memory copying and serialization is
actually one of the motivations behind Tungsten. We didn't talk much about
it since it is not end-user-facing and it is still too early. There are a
few challenges still:
1. Spark cannot run entirely in off-heap mode (by entirely
I would also like to see data shared off-heap to a 3rd party C++
library with JNI, I think the complications would be how to memory
manage this and make sure the 3rd party libraries also adhere to the
access contracts as well.
Tim
On Sat, Aug 29, 2015 at 12:17 PM, Paul Weiss wrote:
> Hi,
>
> Wou
Hi,
Would the benefits of project tungsten be available for access by non-JVM
programs directly into the off-heap memory? Spark using dataframes w/ the
tungsten improvements will definitely help analytics within the JVM world
but accessing outside 3rd party c++ libraries is a challenge especially
13 matches
Mail list logo
