rxin wrote > There is some work to create an off-heap storage API for Spark. I think > with it, it will be easier to support different storage backends. > > https://issues.apache.org/jira/browse/SPARK-6479 > > With that API in place, rest of the integration should probably just live > outside of Spark in Ignite or as a 3rd party package.
Actually Apache Ignite already comes with a very easy to use off-heap store based on JCache (JSR107) API. I think you may be able to reuse it for the Spark off-heap implementation. More information can be found here: http://apacheignite.readme.io/v1.0/docs/off-heap-memory I will comment in the ticket as well. rxin wrote > On Fri, Apr 10, 2015 at 5:21 AM, Devl Devel < > devl.development@ > > > wrote: > >> Hi >> >> Having evaluated the ignite project and it's relationship to spark: >> Ignite >> vs. Spark >> >> https://wiki.apache.org/incubator/IgniteProposal >> >> The scope of the project are defined. However are there any current or >> future projects planned to have an Ignite file system RDD (like the >> Hadoop >> RDD) so that Spark can leverage the in-memory file system as an >> alternative/complement to Tachyon for example? >> >> The idea being to use MLLib and other Spark components but with the >> option >> to use ignite fs. >> >> I'm not an expert on Ignite FS but would appreciate any comments on the >> matter. >> >> Thanks >> >> Devl >> -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Integration-with-Apache-Ignite-tp11520p12177.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
