This is an automated email from the ASF dual-hosted git repository.
xxyu pushed a commit to branch doc5.0
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/doc5.0 by this push:
new 006dd3f22f Add toc of dev design
006dd3f22f is described below
commit 006dd3f22f52fcc9980db1b128701df562b2f7e5
Author: XiaoxiangYu <[email protected]>
AuthorDate: Wed Sep 21 15:32:03 2022 +0800
Add toc of dev design
---
.../development/dev_design/metastore_design.md | 42 +++++++++++++-------
website/docs/development/how_to_release.md | 2 +-
.../development/how_to_subscribe_mailing_list.md | 19 +++++----
.../development/how_to_understand_kylin_design.md | 46 ++++++++++++++++++++++
website/sidebars.js | 4 ++
5 files changed, 89 insertions(+), 24 deletions(-)
diff --git a/website/docs/development/dev_design/metastore_design.md
b/website/docs/development/dev_design/metastore_design.md
index 2022f3b9ab..36660d1732 100644
--- a/website/docs/development/dev_design/metastore_design.md
+++ b/website/docs/development/dev_design/metastore_design.md
@@ -5,7 +5,7 @@ sidebar_label: Metastore Design of Kylin 5
pagination_label: Metastore Design of Kylin 5
toc_min_heading_level: 2
toc_max_heading_level: 6
-pagination_prev: null
+pagination_prev: development/how_to_understand_kylin_design
pagination_next: null
showLastUpdateAuthor: true
showLastUpdateTime: true
@@ -28,15 +28,15 @@ last_update:
#### Core Class and Interface
-| Class | Comment
|
-|-----------------------------|----------------------------------------------------|
-| ResourceStore | 用于管理内存中对元数据的操作
|
-| InMemResourceStore | ResourceStore 的实现类, 用于绝大部分情况
|
-| ThreadViewResourceStore | ResourceStore 的实现类, 作为一个沙盒式的 ResourceStore,
在事务中使用 |
-| MetadataStore | 用于管理元数据持久化的操作
|
-| AuditLogStore | 用于节点间元数据同步, 以及诊断元数据异常情况
|
-| Epoch | 用于保证同时只有一个进程对指定项目下的元数据进行修改操作, 或者提交作业
|
-| EpochStore | 用于持久化 Epoch
|
+| Class | Comment
|
+|-------------------------|----------------------------------------------------|
+| ResourceStore | 用于管理内存中对元数据的操作
|
+| InMemResourceStore | ResourceStore 的实现类, 用于绝大部分情况
|
+| ThreadViewResourceStore | ResourceStore 的实现类, 作为一个沙盒式的 ResourceStore, 在事务中使用
|
+| MetadataStore | 用于管理元数据持久化的操作
|
+| AuditLogStore | 用于节点间元数据同步, 以及诊断元数据异常情况
|
+| Epoch | 用于保证同时只有一个进程对指定项目下的元数据进行修改操作, 或者提交作业
|
+| EpochStore | 用于持久化 Epoch
|
### Question and Answer
@@ -58,7 +58,7 @@ Follower 同步元数据变更，通过两个方式，代码在 AuditLogReplayWo
按照设计，Follower 元数据的 delay 在 1-2s 左右(被动广播同步)，最多 5s(主动定期同步).
-#### 4. How to read meta from metastore？
+#### 4. How ResourceStore was inited when Kylin started?
todo
#### 5. As a kylin developer, how should I write my code to update metadata?
@@ -113,13 +113,25 @@ class SomeService {
3. 两个事务写的时候别复用同一个对象, 以避免元数据更新时, MVCC 检查失败
:::
-#### 6. AuditLog replay and clean
+#### 6. What is Epoch and how do Epoch works?
-todo
+1. 什么是 Epoch?
+
+ `Epoch` 是全局级别的, 项目粒度的元数据写锁, 用于确保同一时刻只有一个进程会修改指定项目下的元数据.
-#### 7. Metadata dumped logic
+2. 什么样的进程可以获取 Epoch?
-todo
+ Job 和 All, Query 不可以获得 Epoch, Query 节点被设计为不需要提交任务或者是修改元数据.
+
+3. Epoch 如何管理?
+
+
+相关类
+- EpochOrchestrator
+- EpochOrchestrator.EpochChecker
+- EpochOrchestrator.EpochRenewer
+- EpochManager
+- EpochChangedListener
### <span id="metadata_write">Diagram of write a piece of meta </span>
diff --git a/website/docs/development/how_to_release.md
b/website/docs/development/how_to_release.md
index dfa8976855..d0d8fba09d 100644
--- a/website/docs/development/how_to_release.md
+++ b/website/docs/development/how_to_release.md
@@ -6,7 +6,7 @@ pagination_label: How to release
toc_min_heading_level: 2
toc_max_heading_level: 6
pagination_prev: development/how_to_package
-pagination_next: null
+pagination_next: development/how_to_understand_kylin_design
showLastUpdateAuthor: true
showLastUpdateTime: true
keywords:
diff --git a/website/docs/development/how_to_subscribe_mailing_list.md
b/website/docs/development/how_to_subscribe_mailing_list.md
index f48653c6c2..6e37297afa 100644
--- a/website/docs/development/how_to_subscribe_mailing_list.md
+++ b/website/docs/development/how_to_subscribe_mailing_list.md
@@ -1,8 +1,8 @@
---
-title: How to subscribe mailing list
+title: How to subscribe and unsubscribe mailing list
language: en
-sidebar_label: How to subscribe mailing list
-pagination_label: How to subscribe mailing list
+sidebar_label: How to subscribe and unsubscribe mailing list
+pagination_label: How to subscribe and unsubscribe mailing list
toc_min_heading_level: 2
toc_max_heading_level: 6
keywords:
@@ -16,11 +16,11 @@ last_update:
### Mailing List Table
These are the mailing lists that have been established for kylin project. For
each list, there is a subscribe, unsubscribe, and an archive link.
-| Mailing List | Subscribe Link | Unsubscribe Link | Archive Link
|
-|:-----------------:|:-----------------:|:-----------------:|:-----------------:|
-| user | [subscribe](mailto:[email protected]) |
[unsubscribe](mailto:[email protected]) | [mail
archive](https://lists.apache.org/[email protected]) |
-| dev | [subscribe](mailto:[email protected]) |
[unsubscribe](mailto:[email protected]) | [mail
archive](https://lists.apache.org/[email protected]) |
-| issue | [subscribe](mailto:[email protected]) |
[unsubscribe](mailto:[email protected]) | [mail
archive](https://lists.apache.org/[email protected]) |
+| Mailing List | Subscribe Link |
Unsubscribe Link |
Archive Link |
+|:------------:|:----------------------------------------------------:|:--------------------------------------------------------:|:-------------------------------------------------------------------------:|
+| user | [subscribe](mailto:[email protected]) |
[unsubscribe](mailto:[email protected]) | [mail
archive](https://lists.apache.org/[email protected]) |
+| dev | [subscribe](mailto:[email protected]) |
[unsubscribe](mailto:[email protected]) | [mail
archive](https://lists.apache.org/[email protected]) |
+| issue | [subscribe](mailto:[email protected]) |
[unsubscribe](mailto:[email protected]) | [mail
archive](https://lists.apache.org/[email protected]) |
### <span id="mailing_list">Subscribe mailing list</span>
@@ -48,6 +48,9 @@ Following is a successful case.
+### <span id="unsubscribe_mailing_list">Unsubscribe mailing list</span>
+todo
+
----
This doc is rewrite based on [apache mailing list
doc](https://www.apache.org/foundation/mailinglists.html#subscribing).
\ No newline at end of file
diff --git a/website/docs/development/how_to_understand_kylin_design.md
b/website/docs/development/how_to_understand_kylin_design.md
new file mode 100644
index 0000000000..735c550ee5
--- /dev/null
+++ b/website/docs/development/how_to_understand_kylin_design.md
@@ -0,0 +1,46 @@
+---
+title: Overall Design of Kylin 5
+language: en
+sidebar_label: Overall Design of Kylin 5
+pagination_label: Overall Design of Kylin 5
+toc_min_heading_level: 2
+toc_max_heading_level: 6
+pagination_prev: development/how_to_release
+pagination_next: development/dev_design/metastore_design
+showLastUpdateAuthor: true
+showLastUpdateTime: true
+keywords:
+ - dev-design
+draft: false
+last_update:
+ date: 09/21/2022
+ author: Xiaoxiang Yu
+---
+
+:::info
+Unless more comments, all source code analysis are based on [this code
snapshot](https://github.com/apache/kylin/tree/edab8698b6a9770ddc4cd00d9788d718d032b5e8)
.
+:::
+
+### About Design of Kylin 5.0
+1. Metadata Store
+ - [x] Metadata Store
+ - [ ] Metadata Cache
+ - [x] Transaction(CRUD of Metadata)
+ - [ ] Epoch, AuditLog etc.
+2. Metadata Format/Schema
+ - [ ] DataModel, IndexPlan and Dataflow
+ - [ ] Index and Layout
+ - [ ] Computed Column
+3. Query Engine
+ - [ ] How a SQL query was executed in Kylin?
+ - [ ] Query Cache
+ - [ ] Query Profile
+4. Model Engine
+ - [ ] Schema Change(Model Change, Table Change)
+5. Build Engine
+ - [ ] Build AggIndex
+ - [ ] Build TableIndex
+ - [ ] Build Snapshot
+6. Job Engine
+ - [ ] JobScheduler
+ - [ ] HA(and epoch)
\ No newline at end of file
diff --git a/website/sidebars.js b/website/sidebars.js
index a1043e0181..59ec7d333d 100644
--- a/website/sidebars.js
+++ b/website/sidebars.js
@@ -974,6 +974,10 @@ const sidebars = {
type: 'doc',
id: 'development/how_to_subscribe_mailing_list'
},
+ {
+ type: 'doc',
+ id: 'development/how_to_understand_kylin_design'
+ },
{
type: 'doc',
id: 'development/dev_design/metastore_design'
