Hello,
I recently start studying the Spark's memory management system.
More spesifically I want to understand how spark use the off-Heap memory.
Interanlly I saw, that there are two types of offHeap memory.
(offHeapExecutionMemoryPool and offHeapStorageMemoryPool).
How Spark us
Hello,
I recently start studying the Spark's memory management system. My
question is about the offHeapExecutionMemoryPool and
offHeapStorageMemoryPool.
1. How Spark use the offHeapExecutionMemoryPool ?
2. How use the offHeap memory (I understand the allocation side),
but it is no
Spark 2.0?
Even if the above answers my first question, I'd still like to know if the
new Spark API will allow RDDs to be /filled/ from the C++ side, as a data
source, rather than a derived dataset.
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nab
breaking consideration and I'll start using it for
consistency, maybe even interoperability.
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/Tungsten-off-heap-memory-access-for-C-libraries-tp13898p17387.html
Sent from the Apache Spark
that djinni
only supports deep-copies of on-JVM-heap data (and no special off-heap
features yet). However, you can in theory use Unsafe to get pointers to
off-heap memory and pass those (as ints) to native code.
So if you need a solution *today*, try checking out a small demo:
https://github.com/dr
It's a very messy problem :)
>
> Was there indeed a JIRA started to track this issue? Can't find it at the
> moment ...
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Tungsten-off-heap-memory-
dive into messing with (standard) Java String <->
std::string using JNI. It's a very messy problem :)
Was there indeed a JIRA started to track this issue? Can't find it at the
moment ...
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.co
Please do. Thanks.
On Mon, Aug 31, 2015 at 5:00 AM, Paul Weiss wrote:
> Sounds good, want me to create a jira and link it to SPARK-9697? Will put
> down some ideas to start.
> On Aug 31, 2015 4:14 AM, "Reynold Xin" wrote:
>
>> BTW if you are interested in this, we could definitely get some help
Sounds good, want me to create a jira and link it to SPARK-9697? Will put
down some ideas to start.
On Aug 31, 2015 4:14 AM, "Reynold Xin" wrote:
> BTW if you are interested in this, we could definitely get some help in
> terms of prototyping the feasibility, i.e. how we can have a native (e.g.
>
BTW if you are interested in this, we could definitely get some help in
terms of prototyping the feasibility, i.e. how we can have a native (e.g.
C++) API for data access shipped with Spark. There are a lot of questions
(e.g. build, portability) that need to be answered.
On Mon, Aug 31, 2015 at 1:
On Sun, Aug 30, 2015 at 5:58 AM, Paul Weiss wrote:
>
> Also, is this work being done on a branch I could look into further and
> try out?
>
>
We don't have a branch yet -- because there is no code nor design for this
yet. As I said, it is one of the motivations behind Tungsten, but it is
fairly e
Reynold,
That is great to hear. Definitely interested in how 2. is being
implemented and how it will be exposed in C++. One important aspect of
leveraging the off heap memory is how the data is organized as well as
being able to easily access it from the C++ side. For example how would
you
> Would the benefits of project tungsten be available for access by non-JVM
> > programs directly into the off-heap memory? Spark using dataframes w/
> the
> > tungsten improvements will definitely help analytics within the JVM world
> > but accessing outside 3rd party c+
Hi,
>
> Would the benefits of project tungsten be available for access by non-JVM
> programs directly into the off-heap memory? Spark using dataframes w/ the
> tungsten improvements will definitely help analytics within the JVM world
> but accessing outside 3rd party c++ libraries is a ch
Hi,
Would the benefits of project tungsten be available for access by non-JVM
programs directly into the off-heap memory? Spark using dataframes w/ the
tungsten improvements will definitely help analytics within the JVM world
but accessing outside 3rd party c++ libraries is a challenge
15 matches
Mail list logo
