Re: [SparkDataFrame] Query Optimization. Prototype

Hello, liyuj > I want to know why I suppose we are talking about Ignite SQL engine, not about Spark integration module(as it written in the message topic) The simple answer - because, it is not implemented. You can file a ticket and we wil see what we can do. You can contribute the support of

Re: [SparkDataFrame] Query Optimization. Prototype

Hi Nikolay, Because in the discussion below, there is a list describing that ROWNUM is not applicable to Ignite, and I want to know why. 在 2019/9/18 下午11:14, Nikolay Izhikov 写道: Hello, liyuj. Please, clarify. Do you want to contribute this to Ignite? What explanation do you expect? В Ср, 1

Re: [SparkDataFrame] Query Optimization. Prototype

Hello, liyuj. Please, clarify. Do you want to contribute this to Ignite? What explanation do you expect? В Ср, 18/09/2019 в 22:46 +0800, liyuj пишет: > Hi, > > Can anyone explain how difficult it is to implement ROWNUM? > > This is a very common requirement. > > 在 2018/1/23 下午4:05, Serge Puch

Re: [SparkDataFrame] Query Optimization. Prototype

Hi, Can anyone explain how difficult it is to implement ROWNUM? This is a very common requirement. 在 2018/1/23 下午4:05, Serge Puchnin 写道: yes, the Cust function is supporting both Ignite and H2. I've updated the documentation for next system functions: CASEWHEN Function, CAST, CONVERT, TABLE

Re: [SparkDataFrame] Query Optimization. Prototype

a lot of unit tests to cover various scenarios. > > > I think we will have this information in a matter of 1-2 months. > > > > So the answer is no, we haven't documentation for a join limitations. > > > > That's why I propose to exclude join optimization

Re: [SparkDataFrame] Query Optimization. Prototype

on from my PR until: > > 1. We create documentation for all join limitations. > 2. Create the way to check is certain join satisfy current limitations. > > [1] http://apache-ignite-developers.2346864.n4.nabble.com/ > SparkDataFrame-Query-Optimization-Prototype-tp26249p26361.h

Re: [SparkDataFrame] Query Optimization. Prototype

ons. 2. Create the way to check is certain join satisfy current limitations. [1] http://apache-ignite-developers.2346864.n4.nabble.com/SparkDataFrame-Query-Optimization-Prototype-tp26249p26361.html В Вт, 13/02/2018 в 09:55 -0800, Valentin Kulichenko пишет: > Nikolay, > > Looks like thi

Re: [SparkDataFrame] Query Optimization. Prototype

Nikolay, Looks like this is because you enabled non-collocated joins. I was not aware of this limitation though, do we have this documented somewhere? -Val On Tue, Feb 13, 2018 at 8:21 AM, Nikolay Izhikov wrote: > Val, > > Source code check: https://github.com/apache/ignite/blob/master/modules

Re: [SparkDataFrame] Query Optimization. Prototype

Val, Source code check: https://github.com/apache/ignite/blob/master/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/opt/GridH2CollocationModel.java#L382 Stack trace: javax.cache.CacheException: Failed to prepare distributed join query: join condition does not us

Re: [SparkDataFrame] Query Optimization. Prototype

Nikolay, This doesn't make sense to me. Not having an index should not cause query to fail. What is the exception? -Val On Tue, Feb 13, 2018 at 8:07 AM, Nikolay Izhikov wrote: > Hello, Valentin. > > > When you're talking about join optimization, what exactly are you > referring to? > > I'm ref

Re: [SparkDataFrame] Query Optimization. Prototype

Hello, Valentin. > When you're talking about join optimization, what exactly are you referring > to? I'm referring to my PR [1] Currently, it contains transformation from Spark joins to Ignite joins [2] But, if I understand Vladimir answer right, for now, we don't *fully* support SQL join quer

Re: [SparkDataFrame] Query Optimization. Prototype

Sounds good. Let me know when you feel it's ready and I'll take a look. -Val On Tue, Feb 13, 2018 at 7:56 AM, Nikolay Izhikov wrote: > Hello, Valentin. > > > Since other parts of data frames integration are already merged, > > I think it's a good time to resurrect this thread? > > Does it make

Re: [SparkDataFrame] Query Optimization. Prototype

Hello, Valentin. > Since other parts of data frames integration are already merged, > I think it's a good time to resurrect this thread? > Does it make sense to review it right now? > Or you want to make some more changes? I've already merged PR [1] with current master. So you can review it, i

Re: [SparkDataFrame] Query Optimization. Prototype

Hello, Vladimir. I've created ticket https://issues.apache.org/jira/browse/IGNITE-7691 В Пн, 12/02/2018 в 11:20 +0300, Vladimir Ozerov пишет: > Hi Nikolay, > > I am not sure if ticket for DECIMAL column metadata exists. If you haven't > find one under "sql" component, please feel free to creat

Re: [SparkDataFrame] Query Optimization. Prototype

The testing of the SQL 99 *Core* specification is done and this page will be released once 2.4 goes live: https://apacheignite-sql.readme.io/v2.3/docs/sql-conformance However, Wiki page is already live: https://en.wikipedia.org/wiki/SQL_compliance — Denis > On Feb 12, 2018, at 12:20 AM, Vladimi

Re: [SparkDataFrame] Query Optimization. Prototype

Nikolay, When you're talking about join optimization, what exactly are you referring to? Since other parts of data frames integration are already merged, I think it's a good time to resurrect this thread? Does it make sense to review it right now? Or you want to make some more changes? -Val On

Re: [SparkDataFrame] Query Optimization. Prototype

Hi Nikolay, I am not sure if ticket for DECIMAL column metadata exists. If you haven't find one under "sql" component, please feel free to create it on your own. As far as testing of joins, I think it makes sense to start working on it when we finish ANSI compliance testing which is already in pro

Re: [SparkDataFrame] Query Optimization. Prototype

Hello, Vladimir. Thank you for an answer. > Do you mean whether it is possible to read it from table metadata? Yes, you are right. I want to read scale and precision of DECIMAL column from table metadata. > This will be fixed at some point in future, but I do not have any dates at > the mome

Re: [SparkDataFrame] Query Optimization. Prototype

Hi Nikolay, Could you please clarify your question about scale and precision? Do you mean whether it is possible to read it from table metadata? If yes, it is not possible at the moment unfortunately - we do not store information about lengths, scales and precision, only actual data types are pass

Re: [SparkDataFrame] Query Optimization. Prototype

Agree. The unsupported functions should be mentioned on the page that will cover Ignite ANSI-99 compliance. We have first results available for CORE features of the specification: https://ggsystems.atlassian.net/wiki/spaces/GG/pages/45093646/ANSI+SQL+99

Re: [SparkDataFrame] Query Optimization. Prototype

I think we need a page listing the unsupported functions with explanation why, which is either it does not make sense in Ignite or is planned in future release. Sergey, do you think you will be able to do it? D. On Tue, Jan 23, 2018 at 12:05 AM, Serge Puchnin wrote: > yes, the Cust function is

Re: [SparkDataFrame] Query Optimization. Prototype

Serge, thank you! Vladimir, guys, can you take a look at another questions: * Can I know scale and precision of DECIMAL column? Example - [3] * Ignite have some limitation for a *distributed* join. For example, we can execute join only for indexed columns. Example - [4]. *

Re: [SparkDataFrame] Query Optimization. Prototype

yes, the Cust function is supporting both Ignite and H2. I've updated the documentation for next system functions: CASEWHEN Function, CAST, CONVERT, TABLE https://apacheignite-sql.readme.io/docs/system-functions And for my mind, next functions aren't applicable for Ignite: ARRAY_GET, ARRAY_LENGT

Re: [SparkDataFrame] Query Optimization. Prototype

I see that CAST is supported by H2 but not presented among Ignite system functions: https://apacheignite-sql.readme.io/docs/system-functions Serge, is this an oversight on our doc side? H2 system functions scope is much broader in general. Could you check this up? Denis On Sunday, January 21, 20

Re: [SparkDataFrame] Query Optimization. Prototype

Hello, Val. Thank you. Looking forward for your review. One more issues with Ignite SQL: Do we have description of CAST function? I want to find out which types that can be converted from one to other. Something like following table for a MsSQL [1] Should I wrote my questions about Ignite SQL

Re: [SparkDataFrame] Query Optimization. Prototype

Hi Nikolay, I will review this sometime next week. -Val On Fri, Jan 19, 2018 at 2:44 AM, Nikolay Izhikov wrote: > Hello, guys. > > I have done prototype of implementation of optimization query from Spark > to Ignite [1]. > > Please, take a look at PR [2]. > > But still there are some issues I

[SparkDataFrame] Query Optimization. Prototype

Hello, guys. I have done prototype of implementation of optimization query from Spark to Ignite [1]. Please, take a look at PR [2]. But still there are some issues I want to clarify with community: * Can I know scale and precision of DECIMAL column? Example - [3] * Ignite have