I will review in the next few days. -Val
On Tue, Sep 26, 2017 at 2:23 PM, Denis Magda <dma...@apache.org> wrote: > Hello Nikolay, > > This is good news. Finally this capability is coming to Ignite. > > Val, Vladimir, could you do a preliminary review? > > Answering on your questions. > > 1. Yardstick should be enough for performance measurements. As a Spark > user, I will be curious to know what’s the point of this integration. > Probably we need to compare Spark + Ignite and Spark + Hive or Spark + > RDBMS cases. > > 2. If Spark community is reluctant let’s include the module in > ignite-spark integration. > > — > Denis > > > On Sep 25, 2017, at 11:14 AM, Николай Ижиков <nizhikov....@gmail.com> > wrote: > > > > Hello, guys. > > > > Currently, I’m working on integration between Spark and Ignite [1]. > > > > For now, I implement following: > > * Ignite DataSource implementation(IgniteRelationProvider) > > * DataFrame support for Ignite SQL table. > > * IgniteCatalog implementation for a transparent resolving of ignites > > SQL tables. > > > > Implementation of it can be found in PR [2] > > It would be great if someone provides feedback for a prototype. > > > > I made some examples in PR so you can see how API suppose to be used [3]. > > [4]. > > > > I need some advice. Can you help me? > > > > 1. How should this PR be tested? > > > > Of course, I need to provide some unit tests. But what about scalability > > tests, etc. > > Maybe we need some Yardstick benchmark or similar? > > What are your thoughts? > > Which scenarios should I consider in the first place? > > > > 2. Should we provide Spark Catalog implementation inside Ignite codebase? > > > > A current implementation of Spark Catalog based on *internal Spark API*. > > Spark community seems not interested in making Catalog API public or > > including Ignite Catalog in Spark code base [5], [6]. > > > > *Should we include Spark internal API implementation inside Ignite code > > base?* > > > > Or should we consider to include Catalog implementation in some external > > module? > > That will be created and released outside Ignite?(we still can support > and > > develop it inside Ignite community). > > > > [1] https://issues.apache.org/jira/browse/IGNITE-3084 > > [2] https://github.com/apache/ignite/pull/2742 > > [3] https://github.com/apache/ignite/pull/2742/files#diff- > > f4ff509cef3018e221394474775e0905 > > [4] https://github.com/apache/ignite/pull/2742/files#diff- > > f2b670497d81e780dfd5098c5dd8a89c > > [5] http://apache-spark-developers-list.1001551.n3. > > nabble.com/Spark-Core-Custom-Catalog-Integration-between- > > Apache-Ignite-and-Apache-Spark-td22452.html > > [6] https://issues.apache.org/jira/browse/SPARK-17767 > > > > -- > > Nikolay Izhikov > > nizhikov....@gmail.com > >
