[
https://issues.apache.org/jira/browse/IGNITE-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16304114#comment-16304114
]
Valentin Kulichenko commented on IGNITE-3084:
---------------------------------------------
[~NIzhikov], looks much better now, here are some points I want to reiterate on
though.
* {{onApplicationEnd}} method - makes sense. But it sounds like it should be on
{{IgniteContext}} level, what do you think?
* {{IgniteSQLRelation#calcPartitions}} - got it, but what will happen if
topology changes? Will partitions be recalculated?
* {{IgniteCacheRelation}} - let's remove it for now and discuss on dev@ as a
separate task. If we come up with good API for this talk, then create a Jira
ticket and implement. Although I feel that there are more important tasks at
the moment, like implementing custom strategy for SQL execution.
* {{org.apache.spark.sql.ignite}} package. We currently have three classes
there, and it looks like only {{IgniteSparkSession}} is supposed to be used in
application code. Can we move it to {{org.apache.ignite.spark}} package and put
it with all other public classes? {{IgniteExternalCatalog}} and
{{IgniteSharedState}} can then remain in this weird package, as they are
implementation only and not public.
> Spark Data Frames Support in Apache Ignite
> ------------------------------------------
>
> Key: IGNITE-3084
> URL: https://issues.apache.org/jira/browse/IGNITE-3084
> Project: Ignite
> Issue Type: Task
> Components: spark
> Affects Versions: 1.5.0.final
> Reporter: Vladimir Ozerov
> Assignee: Nikolay Izhikov
> Priority: Critical
> Labels: bigdata, important
> Fix For: 2.4
>
>
> Apache Spark already benefits from integration with Apache Ignite. The latter
> provides shared RDDs, an implementation of Spark RDD, that help Spark to
> share a state between Spark workers and execute SQL queries much faster. The
> next logical step is to enable support for modern Spark Data Frames API in a
> similar way.
> As a contributor, you will be fully in charge of the integration of Spark
> Data Frame API and Apache Ignite.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
