1. Would it possible for users of Kylin 2-4 to upgrade their metadata to Kylin5 easily?
I was talking with some early users of new codebase, they told me that they have a plan to upgrade to new codebase(kylin 5) from kylin 3, and they plan to developed and contributed the metadata upgradation tools. So I think this issue will be solved soon. 2. Would the structures(URL, request and response) of the Restiful Apis in Kylin 2-4 be kept? It is a good question, but I have to say most REST APIs have been rewritten so they are call in new way. I think new REST doc will help to solve this partially. 3. Any benchmark test has been done? I think I will do a benchmark in next month. 4. Are features like Realtime Cubing, Cube Planner in Kylin3 are included in Kylin5? Kafka streaming/Realtime cubing/JDBC source are implemented in new way so previous code are not exists. For cube planner, Liu Kun are trying hard to implement it in new codebase (see https://github.com/apache/kylin/pull/2089 ). -- Best wishes to you ! From ：Xiaoxiang Yu At 2023-02-15 21:03:40, "George Ni" <n...@apache.org> wrote: >Hi, > >Overrall, I'd like to give +1 to this proposal, for Kylin5 has implemented >such a lot significant breakthroughs. Below are some of my questions: > >1. Would it possible for users of Kylin 2-4 to upgrade their metadata to >Kylin5 easily? >2. Would the structures(URL, request and response) of the Restiful Apis in >Kylin 2-4 be kept? >3. Any benchmark test has been done? >4. Are features like Realtime Cubing, Cube Planner in Kylin3 are included >in Kylin5? > >Li Yang <liy...@apache.org> 于2023年2月15日周三 17:23写道： > >> As Xiaoxiang mentioned, the code donation has a lot of improvements >> compared to current Kylin 4. Many are long wanted, like >> >> - The flexible model can greatly improve the smoothness of adding new >> dimensions in a production environment. >> - The computed column can mind the gap of last-mile data transformation. >> - The new model metadata design that is more friendly to dynamic >> indexing. >> - Support of 63+ dimensions. >> >> Accepting this code base a good thing for the whole Kylin community. >> >> Cheers >> Yang >> >> >> On Tue, Feb 14, 2023 at 10:46 PM ShaoFeng Shi <shaofeng...@apache.org> >> wrote: >> >> > The current limitations are very difficult to solve in normal ways. For >> > example, the Cuboid ID is represented by a Long number, which is 64 bit, >> > and the sequence of each dimension is fixed. The Cuboid ID appears in >> every >> > part of Kylin's source code. This design couldn't be refactored easily. >> So >> > I agree that a whole new design is necessary, in long term it can help a >> > lot. >> > >> > Best regards, >> > >> > Shaofeng Shi 史少锋 >> > Apache Kylin PMC, >> > Apache Incubator PMC, >> > Email: shaofeng...@apache.org >> > >> > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html >> > Join Kylin user mail group: user-subscr...@kylin.apache.org >> > Join Kylin dev mail group: dev-subscr...@kylin.apache.org >> > >> > >> > >> > >> > Xiaoxiang Yu <x...@apache.org> 于2023年2月14日周二 14:22写道： >> > >> > > A formatted version of the discussion with the same content: >> > > >> > > ## Background ## >> > > >> > > As we discussed in the mailing list[2] last year, Kylin 4.0 has >> achieved >> > > its goal in new storage (columnar file) and new query engine (Spark >> > based), >> > > and gained some adoptions from the community. But due to the old design >> > > from the early versions, Kylin 4.0 still keep some limitations from >> > > previous versions, such as max. 63 dimension cap, cube structure >> couldn't >> > > be modified once built, etc. We think the only way to solve those >> > > limitations is to do a whole redesign, especially in the metadata. >> > > >> > > The good news is, Kyligence has started to do that from years ago, and >> > its >> > > comercial version has been verified by many customers in terms of its >> > > functionality, performance and stability. Last year, Kyligence open >> > sourced >> > > its core under Apache License v2.0, and signed CCLA to Apache Software >> > > Foundataion. We staged it in a separate branch of the github repository >> > for >> > > review[1]. Engineers from other teams such as eBay also reviewed the >> > > codebase, and put forward many new ideas. We think based on the >> codebase, >> > > Kylin will not only gain a flexible metadata design, a faster computing >> > > engine, but also will gain richer user scenarios. >> > > >> > > The new codebase has the following features compared with the latest >> > > release (Kylin 4.0.3): >> > > >> > > - More flexible and enhanced data model >> > > * Allow adding new dimensions and measures to the existing data >> model >> > > * The model adapts to table schema changes while retaining the >> > > existing index at the best effort >> > > * Support last-mile data transformation using Computed Column >> > > * Support raw query (non-aggregation query) using Table Index >> > > * Support changing dimension table (SCD2) >> > > - Simplified metadata design >> > > * Merge DataModel and CubeDesc into new DataModel >> > > * Add DataFlow for more generic data sequence, e.g. streaming like >> > > data flow >> > > * New metadata AuditLog for better cache synchronization >> > > - More flexible index management >> > > * Add IndexPlan to support flexible index management >> > > * Add IndexEntity to support different index type >> > > * Add LayoutEntity to support different storage layouts of the same >> > > Index >> > > - Toward a native and vectorized query engine >> > > * Experiment: Integrate with a native execution engine, leveraging >> > > Gluten >> > > * Support async query >> > > * Enhance cost-based index optimizer >> > > - More >> > > * Build engine refactoring and performance optimization >> > > * New WEB UI based on Vue.js, a brand new front-end framework, to >> > > replace AngularJS >> > > * Smooth modeling process on one canvas >> > > >> > > >> > > >> > > >> > > ## Proposal ## >> > > So, I'd like to propose adopting the new codebase from Kyligence as >> Kylin >> > > 's future code base, e.g, Kylin 5. If accepted, we will request an IP >> > > clearance in Apache Incubator for it as the next step. >> > > >> > > >> > > >> > > >> > > >> > > ## Reference ## >> > > https://github.com/apache/kylin/tree/kylin5 >> > > https://lists.apache.org/thread/4fkhyw1fyf0jg5cb18v7vxyqbn6vm3zv >> > > >> > > >> > > -- >> > > >> > > Best wishes to you ! >> > > From ：Xiaoxiang Yu >> > > >> > > >> > > >> > > >> > > >> > > At 2023-02-14 14:09:31, "Xiaoxiang Yu" <x...@apache.org> wrote: >> > > >Background >> > > > >> > > > >> > > >As we discussed in the mailing list[2] last year, Kylin 4.0 has >> achieved >> > > its goal in new storage (columnar file) and new query engine (Spark >> > based), >> > > and gained some adoptions from the community. But due to the old design >> > > from the early versions, Kylin 4.0 still keep some limitations from >> > > previous versions, such as max. 63 dimension cap, cube structure >> couldn't >> > > be modified once built, etc. We think the only way to solve those >> > > limitations is to do a whole redesign, especially in the metadata. >> > > > >> > > > >> > > >The good news is, Kyligence has started to do that from years ago, and >> > > its comercial version has been verified by many customers in terms of >> its >> > > functionality, performance and stability. Last year, Kyligence open >> > sourced >> > > its core under Apache License v2.0, and signed CCLA to Apache Software >> > > Foundataion. We staged it in a separate branch of the github repository >> > for >> > > review[1]. Engineers from other teams such as eBay also reviewed the >> > > codebase, and put forward many new ideas. We think based on the >> codebase, >> > > Kylin will not only gain a flexible metadata design, a faster computing >> > > engine, but also will gain richer user scenarios. >> > > > >> > > > >> > > >The new codebase has the following features compared with the latest >> > > release (Kylin 4.0.3): >> > > >More flexible and enhanced data model >> > > >Allow adding new dimensions and measures to the existing data model >> > > >The model adapts to table schema changes while retaining the existing >> > > index at the best effort >> > > >Support last-mile data transformation using Computed Column >> > > >Support raw query (non-aggregation query) using Table Index >> > > >Support changing dimension table (SCD2) >> > > >Simplified metadata design >> > > >Merge DataModel and CubeDesc into new DataModel >> > > >Add DataFlow for more generic data sequence, e.g. streaming like data >> > flow >> > > >New metadata AuditLog for better cache synchronization >> > > >More flexible index management >> > > >Add IndexPlan to support flexible index management >> > > >Add IndexEntity to support different index type >> > > >Add LayoutEntity to support different storage layouts of the same >> Index >> > > >Toward a native and vectorized query engine >> > > >Experiment: Integrate with a native execution engine, leveraging >> Gluten >> > > >Support async query >> > > >Enhance cost-based index optimizer >> > > >More >> > > >Build engine refactoring and performance optimization >> > > >New WEB UI based on Vue.js, a brand new front-end framework, to >> replace >> > > AngularJS >> > > >Smooth modeling process on one canvas >> > > >Proposal >> > > >So, I'd like to propose adopting the new codebase from Kyligence as >> > Kylin >> > > 's future code base, e.g, Kylin 5. If accepted, we will request an IP >> > > clearance in Apache Incubator for it as the next step. >> > > >Reference >> > > >https://github.com/apache/kylin/tree/kylin5 >> > > >https://lists.apache.org/thread/4fkhyw1fyf0jg5cb18v7vxyqbn6vm3zv >> > > >https://kylin.apache.org/5.0/blog/introduction_of_metastore_cn >> > > > >> > > >-- >> > > > >> > > >Best wishes to you ! >> > > >From ：Xiaoxiang Yu >> > >> > > >-- > >--------------------- > >Best regards, > > > >Ni Chunen / George
