Val, Thank you.
I fixed issues and answered questions from comments.
Please, take a look.
2017-12-13 3:28 GMT+03:00 Valentin Kulichenko :
> Hi Nikolay,
>
> I reviewed the code and left several comments in the ticket [1]. Please
> take a look.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-
Hi Nikolay,
I reviewed the code and left several comments in the ticket [1]. Please
take a look.
[1] https://issues.apache.org/jira/browse/IGNITE-3084
-Val
On Mon, Dec 4, 2017 at 3:03 PM, Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:
> Denis,
>
> Nikolay was doing final changes a
Denis,
Nikolay was doing final changes and TC stabilization. I'm planning to do
final review this week, so hopefully we will merge the code soon.
-Val
On Mon, Dec 4, 2017 at 1:31 PM, Denis Magda wrote:
> Nikolay, Val,
>
> Since we agreed to release the feature without the strategy support, can
Nikolay, Val,
Since we agreed to release the feature without the strategy support, can the
current integration meet the world in 2.4 release? Please chime in this
conversation:
http://apache-ignite-developers.2346864.n4.nabble.com/Time-and-scope-for-Apache-Ignite-2-4-td24987.html
—
Denis
> On
Denis,
Agree. I will do the final review in next few days and merge the code.
-Val
On Tue, Nov 28, 2017 at 5:28 PM, Denis Magda wrote:
> Guys,
>
> Looking into the parallel discussion about the strategy support I would
> change my initial stance and support the idea of releasing the integratio
Guys,
Looking into the parallel discussion about the strategy support I would change
my initial stance and support the idea of releasing the integration in its
current state. Is the code ready to be merged into the master? Let’s
concentrate on this first and handle the strategy support as a sep
Nikolay,
Let's estimate the strategy implementation work, and then decide weather to
merge the code in current state or not. If anything is unclear, please
start a separate discussion.
-Val
On Fri, Nov 24, 2017 at 5:42 AM, Николай Ижиков
wrote:
> Hello, Val, Denis.
>
> > Personally, I think th
Hello, Val, Denis.
> Personally, I think that we should release the integration only after the
strategy is fully supported.
I see two major reason to propose merge of DataFrame API implementation without
custom strategy:
1. My PR is relatively huge, already. From my experience of interaction
Val, Nikolay,
Personally, I think that we should release the integration only after the
strategy is fully supported. Without the strategy we don’t really leverage from
Ignite’s SQL engine and introduce redundant data movement between Ignite and
Spark nodes.
How big is the effort to support the
Nikolay,
Custom strategy implementation is actually very important for this
integration. Basically, it will allow to create a SQL query for Ignite and
execute it directly on the cluster. Your current implementation only adds a
new DataSource which means that Spark will fetch data in its own memory
Hello.
> 3. IgniteCatalog vs. IgniteExternalCatalog. Why do we have two Catalog
implementations and what is the difference?
IgniteCatalog removed.
> 5. I don't like that IgniteStrategy and IgniteOptimization have to be set
manually on SQLContext each time it's createdIs there any way to
auto
Hello, Valentin.
My answers is below.
Dmitry, do we need to move discussion to Jira?
> 1. Why do we have org.apache.spark.sql.ignite package in our codebase?
As I mentioned earlier, to implement and override Spark Catalog one have to use
internal(private) Spark API.
So I have to use package `o
Val, thanks for the review. Can I ask you to add the same comments to the
ticket?
On Tue, Oct 17, 2017 at 3:20 PM, Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:
> Nikolay, Anton,
>
> I did a high level review of the code. First of all, impressive results!
> However, I have some ques
Nikolay, Anton,
I did a high level review of the code. First of all, impressive results!
However, I have some questions/comments.
1. Why do we have org.apache.spark.sql.ignite package in our codebase? Can
these classes reside under org.apache.ignite.spark instead?
2. IgniteRelationProvider contai
Sounds awesome.
I'll try to review API & tests this week.
Val,
Your review still required :)
On Tue, Oct 17, 2017 at 2:36 PM, Николай Ижиков
wrote:
> Yes
>
> 17 окт. 2017 г. 2:34 PM пользователь "Anton Vinogradov" <
> avinogra...@gridgain.com> написал:
>
> > Nikolay,
> >
> > So, it will be abl
Yes
17 окт. 2017 г. 2:34 PM пользователь "Anton Vinogradov" <
avinogra...@gridgain.com> написал:
> Nikolay,
>
> So, it will be able to start regular spark and ignite clusters and, using
> peer classloading via spark-context, perform any DataFrame request,
> correct?
>
> On Tue, Oct 17, 2017 at 2:
Nikolay,
So, it will be able to start regular spark and ignite clusters and, using
peer classloading via spark-context, perform any DataFrame request, correct?
On Tue, Oct 17, 2017 at 2:25 PM, Николай Ижиков
wrote:
> Hello, Anton.
>
> An example you provide is a path to a master *local* file.
>
Hello, Anton.
An example you provide is a path to a master *local* file.
These libraries are added to the classpath for each remote node running
submitted job.
Please, see documentation:
http://spark.apache.org/docs/latest/api/java/org/apache/
spark/SparkContext.html#addJar(java.lang.String)
htt
Nikolay,
> With Data Frame API implementation there are no requirements to have any
> Ignite files on spark worker nodes.
What do you mean? I see code like:
spark.sparkContext.addJar(MAVEN_HOME +
"/org/apache/ignite/ignite-core/2.3.0-SNAPSHOT/ignite-core-2.3.0-SNAPSHOT.jar")
On Mon, Oct 16, 201
Hello, guys.
I have created example application to run Ignite Data Frame on standalone
Spark cluster.
With Data Frame API implementation there are no requirements to have any
Ignite files on spark worker nodes.
I ran this application on the free dataset: ATP tennis match statistics.
data - https
Hi Nikolay,
Sorry for delay on this, got a little swamped lately. I will do my best to
review the code this week.
-Val
On Mon, Oct 9, 2017 at 11:48 AM, Николай Ижиков
wrote:
> Hello, Valentin.
>
> Did you have a chance to look at my changes?
>
> Now I think I have done almost all required feat
Hello, Valentin.
Did you have a chance to look at my changes?
Now I think I have done almost all required features.
I want to make some performance test to ensure my implementation work
properly with a significant amount of data.
And I definitely need some feedback for my changes.
2017-10-09 18
Hello, guys.
Which version of Spark do we want to use?
1. Currently, Ignite depends on Spark 2.1.0.
* Can be run on JDK 7.
* Still supported: 2.1.2 will be released soon.
2. Latest Spark version is 2.2.0.
* Can be run only on JDK 8+
* Released Jul 11, 2017.
* Already suppor
I will review in the next few days.
-Val
On Tue, Sep 26, 2017 at 2:23 PM, Denis Magda wrote:
> Hello Nikolay,
>
> This is good news. Finally this capability is coming to Ignite.
>
> Val, Vladimir, could you do a preliminary review?
>
> Answering on your questions.
>
> 1. Yardstick should be eno
Hello Nikolay,
This is good news. Finally this capability is coming to Ignite.
Val, Vladimir, could you do a preliminary review?
Answering on your questions.
1. Yardstick should be enough for performance measurements. As a Spark user, I
will be curious to know what’s the point of this integra
25 matches
Mail list logo
