Val, Thank you. I fixed issues and answered questions from comments. Please, take a look.
2017-12-13 3:28 GMT+03:00 Valentin Kulichenko <valentin.kuliche...@gmail.com >: > Hi Nikolay, > > I reviewed the code and left several comments in the ticket [1]. Please > take a look. > > [1] https://issues.apache.org/jira/browse/IGNITE-3084 > > -Val > > On Mon, Dec 4, 2017 at 3:03 PM, Valentin Kulichenko < > valentin.kuliche...@gmail.com> wrote: > > > Denis, > > > > Nikolay was doing final changes and TC stabilization. I'm planning to do > > final review this week, so hopefully we will merge the code soon. > > > > -Val > > > > On Mon, Dec 4, 2017 at 1:31 PM, Denis Magda <dma...@apache.org> wrote: > > > >> Nikolay, Val, > >> > >> Since we agreed to release the feature without the strategy support, can > >> the current integration meet the world in 2.4 release? Please chime in > this > >> conversation: > >> http://apache-ignite-developers.2346864.n4.nabble.com/Time- > >> and-scope-for-Apache-Ignite-2-4-td24987.html > >> > >> — > >> Denis > >> > >> > On Nov 28, 2017, at 5:42 PM, Valentin Kulichenko < > >> valentin.kuliche...@gmail.com> wrote: > >> > > >> > Denis, > >> > > >> > Agree. I will do the final review in next few days and merge the code. > >> > > >> > -Val > >> > > >> > On Tue, Nov 28, 2017 at 5:28 PM, Denis Magda <dma...@apache.org> > wrote: > >> > > >> >> Guys, > >> >> > >> >> Looking into the parallel discussion about the strategy support I > would > >> >> change my initial stance and support the idea of releasing the > >> integration > >> >> in its current state. Is the code ready to be merged into the master? > >> Let’s > >> >> concentrate on this first and handle the strategy support as a > separate > >> >> JIRA task. Agree? > >> >> > >> >> — > >> >> Denis > >> >> > >> >>> On Nov 27, 2017, at 3:47 PM, Valentin Kulichenko < > >> >> valentin.kuliche...@gmail.com> wrote: > >> >>> > >> >>> Nikolay, > >> >>> > >> >>> Let's estimate the strategy implementation work, and then decide > >> weather > >> >> to > >> >>> merge the code in current state or not. If anything is unclear, > please > >> >>> start a separate discussion. > >> >>> > >> >>> -Val > >> >>> > >> >>> On Fri, Nov 24, 2017 at 5:42 AM, Николай Ижиков < > >> nizhikov....@gmail.com> > >> >>> wrote: > >> >>> > >> >>>> Hello, Val, Denis. > >> >>>> > >> >>>>> Personally, I think that we should release the integration only > >> after > >> >>>> the strategy is fully supported. > >> >>>> > >> >>>> I see two major reason to propose merge of DataFrame API > >> implementation > >> >>>> without custom strategy: > >> >>>> > >> >>>> 1. My PR is relatively huge, already. From my experience of > >> interaction > >> >>>> with Ignite community - the bigger PR becomes, the more time of > >> >> commiters > >> >>>> required to review PR. > >> >>>> So, I propose to move smaller, but complete steps here. > >> >>>> > >> >>>> 2. It is not clear for me what exactly includes "custom strategy > and > >> >>>> optimization". > >> >>>> Seems, that additional discussion required. > >> >>>> I think, I can put my thoughts on the paper and start discussion > >> right > >> >>>> after basic implementation is done. > >> >>>> > >> >>>>> Custom strategy implementation is actually very important for this > >> >>>> integration. > >> >>>> > >> >>>> Understand and fully agreed. > >> >>>> I'm ready to continue work in that area. > >> >>>> > >> >>>> 23.11.2017 02:15, Denis Magda пишет: > >> >>>> > >> >>>> Val, Nikolay, > >> >>>>> > >> >>>>> Personally, I think that we should release the integration only > >> after > >> >> the > >> >>>>> strategy is fully supported. Without the strategy we don’t really > >> >> leverage > >> >>>>> from Ignite’s SQL engine and introduce redundant data movement > >> between > >> >>>>> Ignite and Spark nodes. > >> >>>>> > >> >>>>> How big is the effort to support the strategy in terms of the > >> amount of > >> >>>>> work left? 40%, 60%, 80%? > >> >>>>> > >> >>>>> — > >> >>>>> Denis > >> >>>>> > >> >>>>> On Nov 22, 2017, at 2:57 PM, Valentin Kulichenko < > >> >>>>>> valentin.kuliche...@gmail.com> wrote: > >> >>>>>> > >> >>>>>> Nikolay, > >> >>>>>> > >> >>>>>> Custom strategy implementation is actually very important for > this > >> >>>>>> integration. Basically, it will allow to create a SQL query for > >> Ignite > >> >>>>>> and > >> >>>>>> execute it directly on the cluster. Your current implementation > >> only > >> >>>>>> adds a > >> >>>>>> new DataSource which means that Spark will fetch data in its own > >> >> memory > >> >>>>>> first, and then do most of the work (like joins for example). > Does > >> it > >> >>>>>> make > >> >>>>>> sense to you? Can you please take a look at this and provide your > >> >>>>>> thoughts > >> >>>>>> on how much development is implied there? > >> >>>>>> > >> >>>>>> Current code looks good to me though and I'm OK if the strategy > is > >> >>>>>> implemented as a next step in a scope of separate ticket. I will > do > >> >> final > >> >>>>>> review early next week and will merge it if everything is OK. > >> >>>>>> > >> >>>>>> -Val > >> >>>>>> > >> >>>>>> On Thu, Oct 19, 2017 at 7:29 AM, Николай Ижиков < > >> >> nizhikov....@gmail.com> > >> >>>>>> wrote: > >> >>>>>> > >> >>>>>> Hello. > >> >>>>>>> > >> >>>>>>> 3. IgniteCatalog vs. IgniteExternalCatalog. Why do we have two > >> >> Catalog > >> >>>>>>>> > >> >>>>>>> implementations and what is the difference? > >> >>>>>>> > >> >>>>>>> IgniteCatalog removed. > >> >>>>>>> > >> >>>>>>> 5. I don't like that IgniteStrategy and IgniteOptimization have > >> to be > >> >>>>>>>> > >> >>>>>>> set manually on SQLContext each time it's created....Is there > any > >> >> way to > >> >>>>>>> automate this and improve usability? > >> >>>>>>> > >> >>>>>>> IgniteStrategy and IgniteOptimization are removed as it empty > now. > >> >>>>>>> > >> >>>>>>> Actually, I think it makes sense to create a builder similar to > >> >>>>>>>> > >> >>>>>>> SparkSession.builder()... > >> >>>>>>> > >> >>>>>>> IgniteBuilder added. > >> >>>>>>> Syntax looks like: > >> >>>>>>> > >> >>>>>>> ``` > >> >>>>>>> val igniteSession = IgniteSparkSession.builder() > >> >>>>>>> .appName("Spark Ignite catalog example") > >> >>>>>>> .master("local") > >> >>>>>>> .config("spark.executor.instances", "2") > >> >>>>>>> .igniteConfig(CONFIG) > >> >>>>>>> .getOrCreate() > >> >>>>>>> > >> >>>>>>> igniteSession.catalog.listTables().show() > >> >>>>>>> ``` > >> >>>>>>> > >> >>>>>>> Please, see updated PR - https://github.com/apache/igni > >> te/pull/2742 > >> >>>>>>> > >> >>>>>>> 2017-10-18 20:02 GMT+03:00 Николай Ижиков < > nizhikov....@gmail.com > >> >: > >> >>>>>>> > >> >>>>>>> Hello, Valentin. > >> >>>>>>>> > >> >>>>>>>> My answers is below. > >> >>>>>>>> Dmitry, do we need to move discussion to Jira? > >> >>>>>>>> > >> >>>>>>>> 1. Why do we have org.apache.spark.sql.ignite package in our > >> >> codebase? > >> >>>>>>>>> > >> >>>>>>>> > >> >>>>>>>> As I mentioned earlier, to implement and override Spark Catalog > >> one > >> >>>>>>>> have > >> >>>>>>>> to use internal(private) Spark API. > >> >>>>>>>> So I have to use package `org.spark.sql.***` to have access to > >> >> private > >> >>>>>>>> class and variables. > >> >>>>>>>> > >> >>>>>>>> For example, SharedState class that stores link to > >> ExternalCatalog > >> >>>>>>>> declared as `private[sql] class SharedState` - i.e. package > >> private. > >> >>>>>>>> > >> >>>>>>>> Can these classes reside under org.apache.ignite.spark instead? > >> >>>>>>>>> > >> >>>>>>>> > >> >>>>>>>> No, as long as we want to have our own implementation of > >> >>>>>>>> ExternalCatalog. > >> >>>>>>>> > >> >>>>>>>> 2. IgniteRelationProvider contains multiple constants which I > >> guess > >> >> are > >> >>>>>>>>> > >> >>>>>>>> some king of config options. Can you describe the purpose of > >> each of > >> >>>>>>>> them? > >> >>>>>>>> > >> >>>>>>>> I extend comments for this options. > >> >>>>>>>> Please, see my commit [1] or PR HEAD: > >> >>>>>>>> > >> >>>>>>>> 3. IgniteCatalog vs. IgniteExternalCatalog. Why do we have two > >> >> Catalog > >> >>>>>>>>> > >> >>>>>>>> implementations and what is the difference? > >> >>>>>>>> > >> >>>>>>>> Good catch, thank you! > >> >>>>>>>> After additional research I founded that only > >> IgniteExternalCatalog > >> >>>>>>>> required. > >> >>>>>>>> I will update PR with IgniteCatalog remove in a few days. > >> >>>>>>>> > >> >>>>>>>> 4. IgniteStrategy and IgniteOptimization are currently no-op. > >> What > >> >> are > >> >>>>>>>>> > >> >>>>>>>> our plans on implementing them? Also, what exactly is planned > in > >> >>>>>>>> IgniteOptimization and what is its purpose? > >> >>>>>>>> > >> >>>>>>>> Actually, this is very good question :) > >> >>>>>>>> And I need advice from experienced community members here: > >> >>>>>>>> > >> >>>>>>>> `IgniteOptimization` purpose is to modify query plan created by > >> >> Spark. > >> >>>>>>>> Currently, we have one optimization described in IGNITE-3084 > [2] > >> by > >> >>>>>>>> you, > >> >>>>>>>> Valentin :) : > >> >>>>>>>> > >> >>>>>>>> “If there are non-Ignite relations in the plan, we should fall > >> back > >> >> to > >> >>>>>>>> native Spark strategies“ > >> >>>>>>>> > >> >>>>>>>> I think we can go little further and reduce join of two Ignite > >> >> backed > >> >>>>>>>> Data Frames into single Ignite SQL query. Currently, this > >> feature is > >> >>>>>>>> unimplemented. > >> >>>>>>>> > >> >>>>>>>> *Do we need it now? Or we can postpone it and concentrates on > >> basic > >> >>>>>>>> Data > >> >>>>>>>> Frame and Catalog implementation?* > >> >>>>>>>> > >> >>>>>>>> `Strategy` purpose, as you correctly mentioned in [2], is > >> transform > >> >>>>>>>> LogicalPlan into physical operators. > >> >>>>>>>> I don’t have ideas how to use this opportunity. So I think we > >> don’t > >> >>>>>>>> need > >> >>>>>>>> IgniteStrategy. > >> >>>>>>>> > >> >>>>>>>> Can you or anyone else suggest some optimization strategy to > >> speed > >> >> up > >> >>>>>>>> SQL > >> >>>>>>>> query execution? > >> >>>>>>>> > >> >>>>>>>> 5. I don't like that IgniteStrategy and IgniteOptimization have > >> to > >> >> be > >> >>>>>>>>> > >> >>>>>>>> set manually on SQLContext each time it's created....Is there > any > >> >> way > >> >>>>>>>> to > >> >>>>>>>> automate this and improve usability? > >> >>>>>>>> > >> >>>>>>>> These classes added to `extraOptimizations` when one using > >> >>>>>>>> IgniteSparkSession. > >> >>>>>>>> As far as I know, there is no way to automatically add these > >> >> classes to > >> >>>>>>>> regular SparkSession. > >> >>>>>>>> > >> >>>>>>>> 6. What is the purpose of IgniteSparkSession? I see it's used > in > >> >>>>>>>>> > >> >>>>>>>> IgniteCatalogExample but not in IgniteDataFrameExample, which > is > >> >>>>>>>> Confusing. > >> >>>>>>>> > >> >>>>>>>> DataFrame API is *public* Spark API. So anyone can provide > >> >>>>>>>> implementation > >> >>>>>>>> and plug it into Spark. That’s why IgniteDataFrameExample > doesn’t > >> >> need > >> >>>>>>>> any > >> >>>>>>>> Ignite specific session. > >> >>>>>>>> > >> >>>>>>>> Catalog API is *internal* Spark API. There is no way to plug > >> custom > >> >>>>>>>> catalog implementation into Spark [3]. So we have to use > >> >>>>>>>> `IgniteSparkSession` that extends regular SparkSession and > >> overrides > >> >>>>>>>> links > >> >>>>>>>> to `ExternalCatalog`. > >> >>>>>>>> > >> >>>>>>>> 7. To create IgniteSparkSession we first create IgniteContext. > >> Is it > >> >>>>>>>>> > >> >>>>>>>> really needed? It looks like we can directly provide the > >> >> configuration > >> >>>>>>>> file; if IgniteSparkSession really requires IgniteContext, it > can > >> >>>>>>>> create it > >> >>>>>>>> by itself under the hood. > >> >>>>>>>> > >> >>>>>>>> Actually, IgniteContext is base class for Ignite <-> Spark > >> >> integration > >> >>>>>>>> for now. So I tried to reuse it here. I like the idea to remove > >> >>>>>>>> explicit > >> >>>>>>>> usage of IgniteContext. > >> >>>>>>>> Will implement it in a few days. > >> >>>>>>>> > >> >>>>>>>> Actually, I think it makes sense to create a builder similar to > >> >>>>>>>>> > >> >>>>>>>> SparkSession.builder()... > >> >>>>>>>> > >> >>>>>>>> Great idea! I will implement such builder in a few days. > >> >>>>>>>> > >> >>>>>>>> 9. Do I understand correctly that IgniteCacheRelation is for > the > >> >> case > >> >>>>>>>>> > >> >>>>>>>> when we don't have SQL configured on Ignite side? > >> >>>>>>>> > >> >>>>>>>> Yes, IgniteCacheRelation is Data Frame implementation for a > >> >> key-value > >> >>>>>>>> cache. > >> >>>>>>>> > >> >>>>>>>> I thought we decided not to support this, no? Or this is > >> something > >> >>>>>>>>> else? > >> >>>>>>>>> > >> >>>>>>>> > >> >>>>>>>> My understanding is following: > >> >>>>>>>> > >> >>>>>>>> 1. We can’t support automatic resolving key-value caches in > >> >>>>>>>> *ExternalCatalog*. Because there is no way to reliably detect > key > >> >> and > >> >>>>>>>> value > >> >>>>>>>> classes. > >> >>>>>>>> > >> >>>>>>>> 2. We can support key-value caches in regular Data Frame > >> >>>>>>>> implementation. > >> >>>>>>>> Because we can require user to provide key and value classes > >> >>>>>>>> explicitly. > >> >>>>>>>> > >> >>>>>>>> 8. Can you clarify the query syntax in > >> >> IgniteDataFrameExample#nativeS > >> >>>>>>>>> > >> >>>>>>>> parkSqlFromCacheExample2? > >> >>>>>>>> > >> >>>>>>>> Key-value cache: > >> >>>>>>>> > >> >>>>>>>> key - java.lang.Long, > >> >>>>>>>> value - case class Person(name: String, birthDate: > >> java.util.Date) > >> >>>>>>>> > >> >>>>>>>> Schema of data frame for cache is: > >> >>>>>>>> > >> >>>>>>>> key - long > >> >>>>>>>> value.name - string > >> >>>>>>>> value.birthDate - date > >> >>>>>>>> > >> >>>>>>>> So we can select data from data from cache: > >> >>>>>>>> > >> >>>>>>>> SELECT > >> >>>>>>>> key, `value.name`, `value.birthDate` > >> >>>>>>>> FROM > >> >>>>>>>> testCache > >> >>>>>>>> WHERE key >= 2 AND `value.name` like '%0' > >> >>>>>>>> > >> >>>>>>>> [1] https://github.com/apache/ignite/pull/2742/commits/ > faf3ed6fe > >> >>>>>>>> bf417bc59b0519156fd4d09114c8da7 > >> >>>>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-3084? > focusedCom > >> >>>>>>>> mentId=15794210&page=com.atlassian.jira.plugin.system.issuet > >> >>>>>>>> abpanels:comment-tabpanel#comment-15794210 > >> >>>>>>>> [3] https://issues.apache.org/jira/browse/SPARK-17767? > focusedCom > >> >>>>>>>> mentId=15543733&page=com.atlassian.jira.plugin.system.issuet > >> >>>>>>>> abpanels:comment-tabpanel#comment-15543733 > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> 18.10.2017 04:39, Dmitriy Setrakyan пишет: > >> >>>>>>>> > >> >>>>>>>> Val, thanks for the review. Can I ask you to add the same > >> comments > >> >> to > >> >>>>>>>> the > >> >>>>>>>> > >> >>>>>>>>> ticket? > >> >>>>>>>>> > >> >>>>>>>>> On Tue, Oct 17, 2017 at 3:20 PM, Valentin Kulichenko < > >> >>>>>>>>> valentin.kuliche...@gmail.com> wrote: > >> >>>>>>>>> > >> >>>>>>>>> Nikolay, Anton, > >> >>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>>> I did a high level review of the code. First of all, > impressive > >> >>>>>>>>>> results! > >> >>>>>>>>>> However, I have some questions/comments. > >> >>>>>>>>>> > >> >>>>>>>>>> 1. Why do we have org.apache.spark.sql.ignite package in our > >> >>>>>>>>>> codebase? > >> >>>>>>>>>> Can > >> >>>>>>>>>> these classes reside under org.apache.ignite.spark instead? > >> >>>>>>>>>> 2. IgniteRelationProvider contains multiple constants which I > >> >> guess > >> >>>>>>>>>> are > >> >>>>>>>>>> some king of config options. Can you describe the purpose of > >> each > >> >> of > >> >>>>>>>>>> them? > >> >>>>>>>>>> 3. IgniteCatalog vs. IgniteExternalCatalog. Why do we have > two > >> >>>>>>>>>> Catalog > >> >>>>>>>>>> implementations and what is the difference? > >> >>>>>>>>>> 4. IgniteStrategy and IgniteOptimization are currently no-op. > >> What > >> >>>>>>>>>> are > >> >>>>>>>>>> our > >> >>>>>>>>>> plans on implementing them? Also, what exactly is planned in > >> >>>>>>>>>> IgniteOptimization and what is its purpose? > >> >>>>>>>>>> 5. I don't like that IgniteStrategy and IgniteOptimization > have > >> >> to be > >> >>>>>>>>>> set > >> >>>>>>>>>> manually on SQLContext each time it's created. This seems to > be > >> >> very > >> >>>>>>>>>> error > >> >>>>>>>>>> prone. Is there any way to automate this and improve > usability? > >> >>>>>>>>>> 6. What is the purpose of IgniteSparkSession? I see it's used > >> >>>>>>>>>> in IgniteCatalogExample but not in IgniteDataFrameExample, > >> which > >> >> is > >> >>>>>>>>>> confusing. > >> >>>>>>>>>> 7. To create IgniteSparkSession we first create > IgniteContext. > >> Is > >> >> it > >> >>>>>>>>>> really > >> >>>>>>>>>> needed? It looks like we can directly provide the > configuration > >> >>>>>>>>>> file; if > >> >>>>>>>>>> IgniteSparkSession really requires IgniteContext, it can > create > >> >> it by > >> >>>>>>>>>> itself under the hood. Actually, I think it makes sense to > >> create > >> >> a > >> >>>>>>>>>> builder > >> >>>>>>>>>> similar to SparkSession.builder(), it would be good if our > APIs > >> >> here > >> >>>>>>>>>> are > >> >>>>>>>>>> consistent with Spark APIs. > >> >>>>>>>>>> 8. Can you clarify the query syntax > >> >>>>>>>>>> inIgniteDataFrameExample#nativeSparkSqlFromCacheExample2? > >> >>>>>>>>>> 9. Do I understand correctly that IgniteCacheRelation is for > >> the > >> >> case > >> >>>>>>>>>> when > >> >>>>>>>>>> we don't have SQL configured on Ignite side? I thought we > >> decided > >> >>>>>>>>>> not to > >> >>>>>>>>>> support this, no? Or this is something else? > >> >>>>>>>>>> > >> >>>>>>>>>> Thanks! > >> >>>>>>>>>> > >> >>>>>>>>>> -Val > >> >>>>>>>>>> > >> >>>>>>>>>> On Tue, Oct 17, 2017 at 4:40 AM, Anton Vinogradov < > >> >>>>>>>>>> avinogra...@gridgain.com> > >> >>>>>>>>>> wrote: > >> >>>>>>>>>> > >> >>>>>>>>>> Sounds awesome. > >> >>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>>> I'll try to review API & tests this week. > >> >>>>>>>>>>> > >> >>>>>>>>>>> Val, > >> >>>>>>>>>>> Your review still required :) > >> >>>>>>>>>>> > >> >>>>>>>>>>> On Tue, Oct 17, 2017 at 2:36 PM, Николай Ижиков < > >> >>>>>>>>>>> nizhikov....@gmail.com> > >> >>>>>>>>>>> wrote: > >> >>>>>>>>>>> > >> >>>>>>>>>>> Yes > >> >>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> 17 окт. 2017 г. 2:34 PM пользователь "Anton Vinogradov" < > >> >>>>>>>>>>>> avinogra...@gridgain.com> написал: > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> Nikolay, > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> So, it will be able to start regular spark and ignite > >> clusters > >> >>>>>>>>>>>>> and, > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> using > >> >>>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>>> peer classloading via spark-context, perform any DataFrame > >> >> request, > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>> correct? > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> On Tue, Oct 17, 2017 at 2:25 PM, Николай Ижиков < > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> nizhikov....@gmail.com> > >> >>>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>>> wrote: > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> Hello, Anton. > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> An example you provide is a path to a master *local* > file. > >> >>>>>>>>>>>>>> These libraries are added to the classpath for each > remote > >> >> node > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> running > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>> submitted job. > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> Please, see documentation: > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> http://spark.apache.org/docs/latest/api/java/org/apache/ > >> >>>>>>>>>>>>>> spark/SparkContext.html#addJar(java.lang.String) > >> >>>>>>>>>>>>>> http://spark.apache.org/docs/latest/api/java/org/apache/ > >> >>>>>>>>>>>>>> spark/SparkContext.html#addFile(java.lang.String) > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> 2017-10-17 13:10 GMT+03:00 Anton Vinogradov < > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> avinogra...@gridgain.com > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>> : > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> Nikolay, > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> With Data Frame API implementation there are no > >> requirements > >> >> to > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> have > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>> any > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> Ignite files on spark worker nodes. > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> What do you mean? I see code like: > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> spark.sparkContext.addJar(MAVEN_HOME + > >> >>>>>>>>>>>>>>> "/org/apache/ignite/ignite-core/2.3.0-SNAPSHOT/ignite- > >> >>>>>>>>>>>>>>> core-2.3.0-SNAPSHOT.jar") > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> On Mon, Oct 16, 2017 at 5:22 PM, Николай Ижиков < > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> nizhikov....@gmail.com> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> Hello, guys. > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> I have created example application to run Ignite Data > >> Frame > >> >> on > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> standalone > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> Spark cluster. > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> With Data Frame API implementation there are no > >> >> requirements to > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> have > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>> any > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> Ignite files on spark worker nodes. > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> I ran this application on the free dataset: ATP tennis > >> match > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> statistics. > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> data - https://github.com/nizhikov/atp_matches > >> >>>>>>>>>>>>>>>> app - https://github.com/nizhikov/ig > >> nite-spark-df-example > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> Valentin, do you have a chance to look at my changes? > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> 2017-10-12 6:03 GMT+03:00 Valentin Kulichenko < > >> >>>>>>>>>>>>>>>> valentin.kuliche...@gmail.com > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> : > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> Hi Nikolay, > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> Sorry for delay on this, got a little swamped lately. > I > >> >> will > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> do > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>> my > >> >>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> best > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> to > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> review the code this week. > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> -Val > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> On Mon, Oct 9, 2017 at 11:48 AM, Николай Ижиков < > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> nizhikov....@gmail.com> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> Hello, Valentin. > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> Did you have a chance to look at my changes? > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> Now I think I have done almost all required features. > >> >>>>>>>>>>>>>>>>>> I want to make some performance test to ensure my > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> implementation > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>> work > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> properly with a significant amount of data. > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> And I definitely need some feedback for my changes. > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> 2017-10-09 18:45 GMT+03:00 Николай Ижиков < > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> nizhikov....@gmail.com > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>> : > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> Hello, guys. > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> Which version of Spark do we want to use? > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> 1. Currently, Ignite depends on Spark 2.1.0. > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> * Can be run on JDK 7. > >> >>>>>>>>>>>>>>>>>>> * Still supported: 2.1.2 will be released soon. > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> 2. Latest Spark version is 2.2.0. > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> * Can be run only on JDK 8+ > >> >>>>>>>>>>>>>>>>>>> * Released Jul 11, 2017. > >> >>>>>>>>>>>>>>>>>>> * Already supported by huge vendors(Amazon for > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> example). > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>>>> Note that in IGNITE-3084 I implement some internal Spark > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> API. > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>> So It will take some effort to switch between Spark 2.1 and > >> >>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> 2.2 > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> 2017-09-27 2:20 GMT+03:00 Valentin Kulichenko < > >> >>>>>>>>>>>>>>>>>>> valentin.kuliche...@gmail.com>: > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> I will review in the next few days. > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> -Val > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:23 PM, Denis Magda < > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> dma...@apache.org > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> Hello Nikolay, > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> This is good news. Finally this capability is > >> coming to > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> Ignite. > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> Val, Vladimir, could you do a preliminary review? > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> Answering on your questions. > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> 1. Yardstick should be enough for performance > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> measurements. > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>> As a > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> Spark > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> user, I will be curious to know what’s the point of > this > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> integration. > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> Probably we need to compare Spark + Ignite and Spark + > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> Hive > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>> or > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> Spark + > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> RDBMS cases. > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> 2. If Spark community is reluctant let’s include > the > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> module > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>> in > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> ignite-spark integration. > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> — > >> >>>>>>>>>>>>>>>>>>>>> Denis > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> On Sep 25, 2017, at 11:14 AM, Николай Ижиков < > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> nizhikov....@gmail.com> > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> Hello, guys. > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> Currently, I’m working on integration between > Spark > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> and > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>> Ignite > >> >>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>> [1]. > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> For now, I implement following: > >> >>>>>>>>>>>>>>>>>>>>>> * Ignite DataSource implementation( > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> IgniteRelationProvider) > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>> * DataFrame support for Ignite SQL table. > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> * IgniteCatalog implementation for a transparent > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> resolving > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>> of > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> ignites > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> SQL tables. > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> Implementation of it can be found in PR [2] > >> >>>>>>>>>>>>>>>>>>>>>> It would be great if someone provides feedback > for > >> a > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> prototype. > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> I made some examples in PR so you can see how API > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> suppose > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>> to > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> be > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> used [3]. > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> [4]. > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> I need some advice. Can you help me? > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> 1. How should this PR be tested? > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> Of course, I need to provide some unit tests. But > >> what > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> about > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>> scalability > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> tests, etc. > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> Maybe we need some Yardstick benchmark or > similar? > >> >>>>>>>>>>>>>>>>>>>>>> What are your thoughts? > >> >>>>>>>>>>>>>>>>>>>>>> Which scenarios should I consider in the first > >> place? > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> 2. Should we provide Spark Catalog implementation > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> inside > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>> Ignite > >> >>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>>> codebase? > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> A current implementation of Spark Catalog based on > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> *internal > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>> Spark > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> API*. > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> Spark community seems not interested in making > >> Catalog > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> API > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>> public > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> or > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> including Ignite Catalog in Spark code base [5], [6]. > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> *Should we include Spark internal API > >> implementation > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> inside > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>> Ignite > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> code > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> base?* > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> Or should we consider to include Catalog > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> implementation > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>> in > >> >>>>>>>>>>> > >> >>>>>>>>>>> some > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> external > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> module? > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> That will be created and released outside > >> Ignite?(we > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> still > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>> can > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> support > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> and > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> develop it inside Ignite community). > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira > >> /browse/IGNITE-3084 > >> >>>>>>>>>>>>>>>>>>>>>> [2] https://github.com/apache/ignite/pull/2742 > >> >>>>>>>>>>>>>>>>>>>>>> [3] https://github.com/apache/ > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> ignite/pull/2742/files#diff- > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>> f4ff509cef3018e221394474775e0905 > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> [4] https://github.com/apache/ > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> ignite/pull/2742/files#diff- > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>> f2b670497d81e780dfd5098c5dd8a89c > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> [5] http://apache-spark-developers-list.1001551.n3. > >> >>>>>>>>>>>>>>>>>>>>>> nabble.com/Spark-Core-Custom- > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> Catalog-Integration-between- > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>> Apache-Ignite-and-Apache-Spark-td22452.html > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>> [6] https://issues.apache.org/jira/browse/SPARK-17767 > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> -- > >> >>>>>>>>>>>>>>>>>>>>>> Nikolay Izhikov > >> >>>>>>>>>>>>>>>>>>>>>> nizhikov....@gmail.com > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> -- > >> >>>>>>>>>>>>>>>>>>> Nikolay Izhikov > >> >>>>>>>>>>>>>>>>>>> nizhikov....@gmail.com > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> -- > >> >>>>>>>>>>>>>>>>>> Nikolay Izhikov > >> >>>>>>>>>>>>>>>>>> nizhikov....@gmail.com > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> -- > >> >>>>>>>>>>>>>>>> Nikolay Izhikov > >> >>>>>>>>>>>>>>>> nizhikov....@gmail.com > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> -- > >> >>>>>>>>>>>>>> Nikolay Izhikov > >> >>>>>>>>>>>>>> nizhikov....@gmail.com > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>> > >> >>>>>>> -- > >> >>>>>>> Nikolay Izhikov > >> >>>>>>> nizhikov....@gmail.com > >> >>>>>>> > >> >>>>>>> > >> >>>>> > >> >> > >> >> > >> > >> > > > -- Nikolay Izhikov nizhikov....@gmail.com
