Re: Make off-heap store pluggable

Hi Alexey, SPARK-6479 is for the plugin API, and SPARK-6112 is for hdfs plugin. Thanks. Zhan Zhang On Jul 21, 2015, at 10:56 AM, Alexey Goncharuk mailto:alexey.goncha...@gmail.com>> wrote:

Re: Make off-heap store pluggable

2015-07-20 23:29 GMT-07:00 Matei Zaharia : > I agree with this -- basically, to build on Reynold's point, you should be > able to get almost the same performance by implementing either the Hadoop > FileSystem API or the Spark Data Source API over Ignite in the right way. > This would let people sa

Re: Make off-heap store pluggable

2015-07-20 21:32 GMT-07:00 Prashant Sharma : > +1 Looks like a nice idea(I do not see any harm). Would you like to work > on the patch to support it ? > > Prashant Sharma > > Yes, I would like to contribute to it once we clarify the appropriate path. --Alexey > > > On Tue, Jul 21, 2015 at 2:46

Re: Make off-heap store pluggable

2015-07-20 21:40 GMT-07:00 Reynold Xin : > I sent it prematurely. > > They are already pluggable, or at least in the process to be more > pluggable. In 1.4, instead of calling the external system's API directly, > we added an API for that. There is a patch to add support for HDFS > in-memory cach

Re: Make off-heap store pluggable

(Related, not important comment: it would also be nice to separate out the Tachyon dependency from core, as it's conceptually pluggable but is still hard-coded into several places in the code, and a lot of the comments/docs in the code.) On Tue, Jul 21, 2015 at 5:40 AM, Reynold Xin wrote: > I se

Re: Make off-heap store pluggable

I agree with this -- basically, to build on Reynold's point, you should be able to get almost the same performance by implementing either the Hadoop FileSystem API or the Spark Data Source API over Ignite in the right way. This would let people save data persistently in Ignite in addition to usi

Re: Make off-heap store pluggable

I sent it prematurely. They are already pluggable, or at least in the process to be more pluggable. In 1.4, instead of calling the external system's API directly, we added an API for that. There is a patch to add support for HDFS in-memory cache. Somewhat orthogonal to this, longer term, I am no

Re: Make off-heap store pluggable

They are already pluggable. On Mon, Jul 20, 2015 at 9:32 PM, Prashant Sharma wrote: > +1 Looks like a nice idea(I do not see any harm). Would you like to work > on the patch to support it ? > > Prashant Sharma > > > > On Tue, Jul 21, 2015 at 2:46 AM, Alexey Goncharuk < > alexey.goncha...@gmail.

Re: Make off-heap store pluggable

+1 Looks like a nice idea(I do not see any harm). Would you like to work on the patch to support it ? Prashant Sharma On Tue, Jul 21, 2015 at 2:46 AM, Alexey Goncharuk < alexey.goncha...@gmail.com> wrote: > Hello Spark community, > > I was looking through the code in order to understand better

Make off-heap store pluggable

Hello Spark community, I was looking through the code in order to understand better how RDD is persisted to Tachyon off-heap filesystem. It looks like that the Tachyon filesystem is hard-coded and there is no way to switch to another in-memory filesystem. I think it would be great if the implement