Igor. > There is no decision, here we should decide.
Great. > At now Calcite based engine is placed in different module What project hosted Calcite based engine? > It’s possible to develop it as an experimental extension at first (not a > replacement) For me, Ignite 3 are the place where the new engine has to be placed. Personally, I'm against the support of two independent implementation of SQL engine for several releases. Ignite has too many partially implemented features to include on more :) Let's start with the IEP clarification and replace the SQL engine with the best one for Ignite good. В Пт, 27/09/2019 в 18:08 +0300, Seliverstov Igor пишет: > Nikolay, > > At last we have better questions. > > There is no decision, here we should decide. > > Doing nothing isn’t a decision, it’s just doing nothing > > Spark Catalyst is a good example, but under the hood it has absolutely the > same idea, but adopted to Spark. Calcite is the same, but general. That’s why > it’s better start point. > > Implementing an engine from scratch is really cool, but looks like inventing > a bicycle, don’t think it makes sense. At least I against this option. > > I added requirements to IEP (as you asked), you may see it’s in DRAFT state > and will be complemented by details. > > We have some thoughts on how to make smooth replacement, but at first we > should decide what to replace and what with. > > At now Calcite based engine is placed in different module, we checked it can > build execution graph for both local and distributed cases, it has good > expandability. > We talked to Calcite community to identify possible future issues and > everything points to the fact it’s the best option. > It’s possible to develop it as an experimental extension at first (not a > replacement) until we make sure that it works as expected. This way there are > no risks for anybody who uses Ignite on production environment. > > Regards, > Igor > > > > 27 сент. 2019 г., в 17:25, Nikolay Izhikov <nizhi...@apache.org> написал(а): > > > > Igor. > > > > > The main issue - there is no *selection*. > > > > 1. I don't remember community decision about this. > > > > 2. We should avoid to make such long-term decision so quickly. > > We done this kind of decision with H2 and come to the point when we should > > review it. > > > > > 1) Implementing white papers from scratch > > > 2) Adopting Calcite to our needs. > > > > The third option don't fix issues we have with H2. > > The fourth option I know is using spark-catalyst. > > > > What is wrong with writing engine from scratch? > > > > I ask you to start with engine requirements. > > Can we, please, discuss it? > > > > > If you have an alternative - you're welcome, I'll gratefully listen to > > > you. > > > > We have alternative for now - H2 based engine. > > > > > The main question isn't "WHAT" but "HOW" - that's the discussion topic > > > from my point of view. > > > > When we make a decision about engine we can discuss roadmap for replacement. > > One more time - replacement of SQL engine to some more customizable make > > sense for me. > > But, this kind of decisions need carefull discussion. > > > > В Пт, 27/09/2019 в 17:08 +0300, Seliverstov Igor пишет: > > > Nikolay, > > > > > > The main issue - there is no *selection*. > > > > > > There is a field of knowledge - relational algebra, which describes how > > > to transform relational expressions saving their semantics, and a couple > > > of implementations (Calcite is only one written in Java). > > > > > > There are only two alternatives: > > > > > > 1) Implementing white papers from scratch > > > 2) Adopting Calcite to our needs. > > > > > > The second way was chosen by several other projects, there is experience, > > > there is a list of known issues (like using indexes) so, almost > > > everything is already done for us. > > > > > > Implementing a planner is a big deal, I think anybody understands it > > > there. That's why our proposal to reuse others experience is obvious. > > > > > > If you have an alternative - you're welcome, I'll gratefully listen to > > > you. > > > > > > The main question isn't "WHAT" but "HOW" - that's the discussion topic > > > from my point of view. > > > > > > Regards, > > > Igor > > > > > > > 27 сент. 2019 г., в 16:37, Nikolay Izhikov <nizhi...@apache.org> > > > > написал(а): > > > > > > > > Roman. > > > > > > > > > Nikolay, Maxim, I understand that our arguments may not be as obvious > > > > > for you as it obvious for SQL team. So, please arrange your questions > > > > > in > > > > > a more constructive way. > > > > > > > > What is SQL team? > > > > I only know Ignite community :) > > > > > > > > Please, share you knowledge in IEP. > > > > I want to join to the process of engine *selection*. > > > > It should start with the requirements to such engine. > > > > Can you write it in IEP, please? > > > > > > > > My point is very simple: > > > > > > > > 1. We made the wrong decision with H2 > > > > 2. We should make a well-thought decision about the new engine. > > > > > > > > > How many tickets would satisfy you? > > > > > > > > You write about "issueS" with the H2. > > > > All I see is one open ticket. > > > > IEP doesn't provide enough information. > > > > So it's not about the number of tickets, it's about > > > > > > > > > These two points (single map-reduce execution and inflexible > > > > > optimizer) > > > > > are the main problems with the current engine. > > > > > > > > We may come to the point when Calcite(or any other engine) brings us > > > > third and other "main problems". > > > > This is how it happens with H2. > > > > > > > > Let's start from what we want to get with the engine and move forward > > > > from this base. > > > > What do you think? > > > > > > > > > > > > > > > > В Пт, 27/09/2019 в 16:15 +0300, Roman Kondakov пишет: > > > > > Maxim, Nikolay, > > > > > > > > > > I've listed two issues which show the ideological flaws of the > > > > > current > > > > > engine. > > > > > > > > > > 1. IGNITE-11448 - Open. This ticket describes the impossibility of > > > > > executing queries which can not be fit in the hardcoded one pass > > > > > map-reduce paradigm. > > > > > > > > > > 2. IGNITE-6085 - Closed (won't fix) - This ticket describes the > > > > > second > > > > > major problem with the current engine: H2 query optimizer is very > > > > > primitive and can not perform many useful optimizations. > > > > > > > > > > These two points (single map-reduce execution and inflexible > > > > > optimizer) > > > > > are the main problems with the current engine. It means that our > > > > > engine > > > > > is currently suitable for execution only a very limited subset of > > > > > the > > > > > typical SQL queries. For example it can not even run most of the > > > > > TPC-H > > > > > benchmark queries because they don't fit to the simple map-reduce > > > > > paradigm. > > > > > > > > > > > All I see is links to two tickets: > > > > > > > > > > How many tickets would satisfy you? I named two. And it looks like it > > > > > is > > > > > not enough from your point of view. Ok, so how many is enough? The > > > > > set > > > > > of problems caused by listed above tickets is infinite, therefore I > > > > > can > > > > > not create a ticket for each of them. > > > > > > Tech details also should be added. > > > > > > > > > > Tech details are in the tickets. > > > > > > > > > > > We can't discuss such a huge change as an execution engine > > > > > > replacement with descrition like: > > > > > > "No data co-location control, i.e. arbitrary data can be returned > > > > > > silently" or > > > > > > "Low control on how query executes internally, as a result we have > > > > > > limited possibility to implement improvements/fixes." > > > > > > > > > > Why not? Don't you understand these problems? Or you don't think this > > > > > is > > > > > a problem? > > > > > > > > > > > Let's make these descriptions more specific. > > > > > > > > > > What do you mean by "more specific"? What is the criteria of the > > > > > specific description? > > > > > > > > > > > > > > > > > > > > Nikolay, Maxim, I understand that our arguments may not be as obvious > > > > > for you as it obvious for SQL team. So, please arrange your questions > > > > > in > > > > > a more constructive way. > > > > > > > > > > Thank you! > > > > > > > >
signature.asc
Description: This is a digitally signed message part
