Hi, Overrall, I'd like to give +1 to this proposal, for Kylin5 has implemented such a lot significant breakthroughs. Below are some of my questions:
1. Would it possible for users of Kylin 2-4 to upgrade their metadata to Kylin5 easily? 2. Would the structures(URL, request and response) of the Restiful Apis in Kylin 2-4 be kept? 3. Any benchmark test has been done? 4. Are features like Realtime Cubing, Cube Planner in Kylin3 are included in Kylin5? Li Yang <liy...@apache.org> 于2023年2月15日周三 17:23写道： > As Xiaoxiang mentioned, the code donation has a lot of improvements > compared to current Kylin 4. Many are long wanted, like > > - The flexible model can greatly improve the smoothness of adding new > dimensions in a production environment. > - The computed column can mind the gap of last-mile data transformation. > - The new model metadata design that is more friendly to dynamic > indexing. > - Support of 63+ dimensions. > > Accepting this code base a good thing for the whole Kylin community. > > Cheers > Yang > > > On Tue, Feb 14, 2023 at 10:46 PM ShaoFeng Shi <shaofeng...@apache.org> > wrote: > > > The current limitations are very difficult to solve in normal ways. For > > example, the Cuboid ID is represented by a Long number, which is 64 bit, > > and the sequence of each dimension is fixed. The Cuboid ID appears in > every > > part of Kylin's source code. This design couldn't be refactored easily. > So > > I agree that a whole new design is necessary, in long term it can help a > > lot. > > > > Best regards, > > > > Shaofeng Shi 史少锋 > > Apache Kylin PMC, > > Apache Incubator PMC, > > Email: shaofeng...@apache.org > > > > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html > > Join Kylin user mail group: user-subscr...@kylin.apache.org > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org > > > > > > > > > > Xiaoxiang Yu <x...@apache.org> 于2023年2月14日周二 14:22写道： > > > > > A formatted version of the discussion with the same content: > > > > > > ## Background ## > > > > > > As we discussed in the mailing list[2] last year, Kylin 4.0 has > achieved > > > its goal in new storage (columnar file) and new query engine (Spark > > based), > > > and gained some adoptions from the community. But due to the old design > > > from the early versions, Kylin 4.0 still keep some limitations from > > > previous versions, such as max. 63 dimension cap, cube structure > couldn't > > > be modified once built, etc. We think the only way to solve those > > > limitations is to do a whole redesign, especially in the metadata. > > > > > > The good news is, Kyligence has started to do that from years ago, and > > its > > > comercial version has been verified by many customers in terms of its > > > functionality, performance and stability. Last year, Kyligence open > > sourced > > > its core under Apache License v2.0, and signed CCLA to Apache Software > > > Foundataion. We staged it in a separate branch of the github repository > > for > > > review[1]. Engineers from other teams such as eBay also reviewed the > > > codebase, and put forward many new ideas. We think based on the > codebase, > > > Kylin will not only gain a flexible metadata design, a faster computing > > > engine, but also will gain richer user scenarios. > > > > > > The new codebase has the following features compared with the latest > > > release (Kylin 4.0.3): > > > > > > - More flexible and enhanced data model > > > * Allow adding new dimensions and measures to the existing data > model > > > * The model adapts to table schema changes while retaining the > > > existing index at the best effort > > > * Support last-mile data transformation using Computed Column > > > * Support raw query (non-aggregation query) using Table Index > > > * Support changing dimension table (SCD2) > > > - Simplified metadata design > > > * Merge DataModel and CubeDesc into new DataModel > > > * Add DataFlow for more generic data sequence, e.g. streaming like > > > data flow > > > * New metadata AuditLog for better cache synchronization > > > - More flexible index management > > > * Add IndexPlan to support flexible index management > > > * Add IndexEntity to support different index type > > > * Add LayoutEntity to support different storage layouts of the same > > > Index > > > - Toward a native and vectorized query engine > > > * Experiment: Integrate with a native execution engine, leveraging > > > Gluten > > > * Support async query > > > * Enhance cost-based index optimizer > > > - More > > > * Build engine refactoring and performance optimization > > > * New WEB UI based on Vue.js, a brand new front-end framework, to > > > replace AngularJS > > > * Smooth modeling process on one canvas > > > > > > > > > > > > > > > ## Proposal ## > > > So, I'd like to propose adopting the new codebase from Kyligence as > Kylin > > > 's future code base, e.g, Kylin 5. If accepted, we will request an IP > > > clearance in Apache Incubator for it as the next step. > > > > > > > > > > > > > > > > > > ## Reference ## > > > https://github.com/apache/kylin/tree/kylin5 > > > https://lists.apache.org/thread/4fkhyw1fyf0jg5cb18v7vxyqbn6vm3zv > > > > > > > > > -- > > > > > > Best wishes to you ! > > > From ：Xiaoxiang Yu > > > > > > > > > > > > > > > > > > At 2023-02-14 14:09:31, "Xiaoxiang Yu" <x...@apache.org> wrote: > > > >Background > > > > > > > > > > > >As we discussed in the mailing list[2] last year, Kylin 4.0 has > achieved > > > its goal in new storage (columnar file) and new query engine (Spark > > based), > > > and gained some adoptions from the community. But due to the old design > > > from the early versions, Kylin 4.0 still keep some limitations from > > > previous versions, such as max. 63 dimension cap, cube structure > couldn't > > > be modified once built, etc. We think the only way to solve those > > > limitations is to do a whole redesign, especially in the metadata. > > > > > > > > > > > >The good news is, Kyligence has started to do that from years ago, and > > > its comercial version has been verified by many customers in terms of > its > > > functionality, performance and stability. Last year, Kyligence open > > sourced > > > its core under Apache License v2.0, and signed CCLA to Apache Software > > > Foundataion. We staged it in a separate branch of the github repository > > for > > > review[1]. Engineers from other teams such as eBay also reviewed the > > > codebase, and put forward many new ideas. We think based on the > codebase, > > > Kylin will not only gain a flexible metadata design, a faster computing > > > engine, but also will gain richer user scenarios. > > > > > > > > > > > >The new codebase has the following features compared with the latest > > > release (Kylin 4.0.3): > > > >More flexible and enhanced data model > > > >Allow adding new dimensions and measures to the existing data model > > > >The model adapts to table schema changes while retaining the existing > > > index at the best effort > > > >Support last-mile data transformation using Computed Column > > > >Support raw query (non-aggregation query) using Table Index > > > >Support changing dimension table (SCD2) > > > >Simplified metadata design > > > >Merge DataModel and CubeDesc into new DataModel > > > >Add DataFlow for more generic data sequence, e.g. streaming like data > > flow > > > >New metadata AuditLog for better cache synchronization > > > >More flexible index management > > > >Add IndexPlan to support flexible index management > > > >Add IndexEntity to support different index type > > > >Add LayoutEntity to support different storage layouts of the same > Index > > > >Toward a native and vectorized query engine > > > >Experiment: Integrate with a native execution engine, leveraging > Gluten > > > >Support async query > > > >Enhance cost-based index optimizer > > > >More > > > >Build engine refactoring and performance optimization > > > >New WEB UI based on Vue.js, a brand new front-end framework, to > replace > > > AngularJS > > > >Smooth modeling process on one canvas > > > >Proposal > > > >So, I'd like to propose adopting the new codebase from Kyligence as > > Kylin > > > 's future code base, e.g, Kylin 5. If accepted, we will request an IP > > > clearance in Apache Incubator for it as the next step. > > > >Reference > > > >https://github.com/apache/kylin/tree/kylin5 > > > >https://lists.apache.org/thread/4fkhyw1fyf0jg5cb18v7vxyqbn6vm3zv > > > >https://kylin.apache.org/5.0/blog/introduction_of_metastore_cn > > > > > > > >-- > > > > > > > >Best wishes to you ! > > > >From ：Xiaoxiang Yu > > > -- --------------------- Best regards, Ni Chunen / George
