In 2015 Reynold Xin made improvements to Spark and it was basically moving
some structures that were on the java heap and moving them off heap.
In particular it seemed like the memory did not require any
serialization/deserialization.
How was the performed? Was the data memory mapped? If it was, then to avoid
serialization/deserialization i'm assuming some sort of wrapper was
introduced to allow access to that data.
Something like this:
struct DataType
{
long pointertoData;
method1();
method2();
}
but if it was done this way there is an extra indirection, and I'm assuming
it benchmarked positively.
Just trying to learn.
