Hello, guys.
Which version of Spark do we want to use?
1. Currently, Ignite depends on Spark 2.1.0.
* Can be run on JDK 7.
* Still supported: 2.1.2 will be released soon.
2. Latest Spark version is 2.2.0.
* Can be run only on JDK 8+
* Released Jul 11, 2017.
* Already supported by huge vendors(Amazon for example).
Note that in IGNITE-3084 I implement some internal Spark API.
So It will take some effort to switch between Spark 2.1 and 2.2
2017-09-27 2:20 GMT+03:00 Valentin Kulichenko <valentin.kuliche...@gmail.com
>:
> I will review in the next few days.
>
> -Val
>
> On Tue, Sep 26, 2017 at 2:23 PM, Denis Magda <dma...@apache.org> wrote:
>
> > Hello Nikolay,
> >
> > This is good news. Finally this capability is coming to Ignite.
> >
> > Val, Vladimir, could you do a preliminary review?
> >
> > Answering on your questions.
> >
> > 1. Yardstick should be enough for performance measurements. As a Spark
> > user, I will be curious to know what’s the point of this integration.
> > Probably we need to compare Spark + Ignite and Spark + Hive or Spark +
> > RDBMS cases.
> >
> > 2. If Spark community is reluctant let’s include the module in
> > ignite-spark integration.
> >
> > —
> > Denis
> >
> > > On Sep 25, 2017, at 11:14 AM, Николай Ижиков <nizhikov....@gmail.com>
> > wrote:
> > >
> > > Hello, guys.
> > >
> > > Currently, I’m working on integration between Spark and Ignite [1].
> > >
> > > For now, I implement following:
> > > * Ignite DataSource implementation(IgniteRelationProvider)
> > > * DataFrame support for Ignite SQL table.
> > > * IgniteCatalog implementation for a transparent resolving of
> ignites
> > > SQL tables.
> > >
> > > Implementation of it can be found in PR [2]
> > > It would be great if someone provides feedback for a prototype.
> > >
> > > I made some examples in PR so you can see how API suppose to be used
> [3].
> > > [4].
> > >
> > > I need some advice. Can you help me?
> > >
> > > 1. How should this PR be tested?
> > >
> > > Of course, I need to provide some unit tests. But what about
> scalability
> > > tests, etc.
> > > Maybe we need some Yardstick benchmark or similar?
> > > What are your thoughts?
> > > Which scenarios should I consider in the first place?
> > >
> > > 2. Should we provide Spark Catalog implementation inside Ignite
> codebase?
> > >
> > > A current implementation of Spark Catalog based on *internal Spark
> API*.
> > > Spark community seems not interested in making Catalog API public or
> > > including Ignite Catalog in Spark code base [5], [6].
> > >
> > > *Should we include Spark internal API implementation inside Ignite code
> > > base?*
> > >
> > > Or should we consider to include Catalog implementation in some
> external
> > > module?
> > > That will be created and released outside Ignite?(we still can support
> > and
> > > develop it inside Ignite community).
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-3084
> > > [2] https://github.com/apache/ignite/pull/2742
> > > [3] https://github.com/apache/ignite/pull/2742/files#diff-
> > > f4ff509cef3018e221394474775e0905
> > > [4] https://github.com/apache/ignite/pull/2742/files#diff-
> > > f2b670497d81e780dfd5098c5dd8a89c
> > > [5] http://apache-spark-developers-list.1001551.n3.
> > > nabble.com/Spark-Core-Custom-Catalog-Integration-between-
> > > Apache-Ignite-and-Apache-Spark-td22452.html
> > > [6] https://issues.apache.org/jira/browse/SPARK-17767
> > >
> > > --
> > > Nikolay Izhikov
> > > nizhikov....@gmail.com
> >
> >
>
--
Nikolay Izhikov
nizhikov....@gmail.com
