This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new 6a911a04ca [INLONG-949][Audit] Remove audit documents related to
ClickHouse and ElasticSearch (#950)
6a911a04ca is described below
commit 6a911a04ca1ec6f7e50e0957bcc13ca1e083b2eb
Author: doleyzi <43397300+dole...@users.noreply.github.com>
AuthorDate: Wed May 8 21:28:32 2024 +0800
[INLONG-949][Audit] Remove audit documents related to ClickHouse and
ElasticSearch (#950)
Co-authored-by: Charles Zhang <dockerzh...@apache.org>
---
docs/modules/audit/configure.md | 64 +++-------
docs/modules/audit/img/audit_api.png | Bin 31395 -> 0 bytes
docs/modules/audit/img/audit_architecture.png | Bin 83639 -> 51469 bytes
docs/modules/audit/img/audit_mysql.png | Bin 19356 -> 0 bytes
docs/modules/audit/img/audit_openapi.png | Bin 56423 -> 0 bytes
docs/modules/audit/img/audit_proxy.png | Bin 29358 -> 0 bytes
docs/modules/audit/img/audit_sdk.png | Bin 81979 -> 0 bytes
.../audit/img/audit_sdk_disaster_recovery.png | Bin 35275 -> 0 bytes
docs/modules/audit/img/audit_ui.png | Bin 76860 -> 0 bytes
docs/modules/audit/img/elasticsearch_index.png | Bin 22117 -> 0 bytes
docs/modules/audit/img/elasticsearch_overview.png | Bin 47297 -> 0 bytes
docs/modules/audit/img/elasticsearch_write.png | Bin 26657 -> 0 bytes
docs/modules/audit/overview.md | 53 ++------
docs/modules/audit/quick_start.md | 52 ++++----
.../current/modules/audit/configure.md | 135 ++++++++-------------
.../current/modules/audit/img/audit_api.png | Bin 33174 -> 0 bytes
.../modules/audit/img/audit_architecture.png | Bin 83639 -> 51469 bytes
.../current/modules/audit/img/audit_mysql.png | Bin 19165 -> 0 bytes
.../current/modules/audit/img/audit_openapi.png | Bin 56423 -> 0 bytes
.../current/modules/audit/img/audit_proxy.png | Bin 31329 -> 0 bytes
.../current/modules/audit/img/audit_sdk.png | Bin 77771 -> 0 bytes
.../audit/img/audit_sdk_disaster_recovery.png | Bin 29207 -> 0 bytes
.../current/modules/audit/img/audit_ui.png | Bin 79285 -> 0 bytes
.../modules/audit/img/elasticsearch_index.png | Bin 21801 -> 0 bytes
.../modules/audit/img/elasticsearch_overview.png | Bin 46579 -> 0 bytes
.../modules/audit/img/elasticsearch_write.png | Bin 27577 -> 0 bytes
.../current/modules/audit/overview.md | 129 ++++++++------------
.../current/modules/audit/quick_start.md | 55 ++++-----
28 files changed, 175 insertions(+), 313 deletions(-)
diff --git a/docs/modules/audit/configure.md b/docs/modules/audit/configure.md
index 7f7b234db0..b6e9ba7ba9 100644
--- a/docs/modules/audit/configure.md
+++ b/docs/modules/audit/configure.md
@@ -9,7 +9,7 @@ sidebar_position: 3
* Audit-store storage service configuration (application.properties).
* OpenAPI audit-service configuration audit-service.properties.
-## Audit-proxy source-channel-sink pipeline configuration
(`audit-proxy-{tube|pulsar|kafka}.conf`)
+## Audit Proxy Configuration (`audit-proxy-{tube|pulsar|kafka}.conf`)
### Basic Configuration
@@ -18,9 +18,8 @@ sidebar_position: 3
| agent1.sources | source typpe | tcp-source
| |
| agent1.channels | used channel | ch-msg1
|
|
| agent1.sinks | used sink | pulsar-sink-msg1
| |
-|
-### sources Configuration
+### Sources Configuration
| Parameter | Description | Default value
| Notes |
|---------------------------|--------------------|-------|---------------------------------------------|
@@ -28,14 +27,14 @@ sidebar_position: 3
| agent1.sources.tcp-source.host | tcp ip binding and listening | 0.0.0.0 |
|
| agent1.sources.tcp-source.port | tcp port binding | 10081 | |
-### channel Configuration
+### Channel Configuration
| Parameter | Description | Default
value | Notes |
|--------------------------|----------------------------|------|-----|
| agent1.channels.ch-msg1.type | memory channel type | memory | |
| agent1.channels.ch-msg2.type | file channel type | file | |
-### sink Configuration
+### Sink Configuration
| Parameter | Description |
Default value | Notes |
|------------------------|----------------------------------|-------|-----|
@@ -46,59 +45,28 @@ sidebar_position: 3
| agent1.sinks.pulsar-sink-msg1.enable_token_auth | Whether security
certification is required | false | |
| agent1.sinks.pulsar-sink-msg1.auth_token | pulsar authentication token |
| |
-## Audit-store storage service configuration (`application.properties`)
+## Audit Store Configuration (`application.properties`)
-### MQ configuration
+### MQ Configuration
-| Parameter | Description
| Default value |
Notes |
-|----------------------------|---------------------------------------------|-------------------------------------------------------|--------------------------------------------------------|
-| audit.config.proxy.type | MQ type | pulsar
| |
-| audit.pulsar.server.url | pulsar broker |
pulsar://127.0.0.1:6650 |
|
-| audit.pulsar.topic | pulsar topic | persistent://public/Default
value/inlong-audit |
|
-| audit.pulsar.consumer.sub.name | consumer | inlong-audit-subscription
|
|
-| audit.pulsar.enable.auth | Whether security certification is required |
false |
|
-| audit.pulsar.token | pulsar authentication token |
|
|
-
-### DB configuration
-
-| Parameter | Description | Default value
| Notes |
-|---------------------------|--------------------|-------|---------------------------------------------|
-| spring.datasource.druid.driver-class-name | Set the driver type |
com.mysql.cj.jdbc.Driver | |
-| spring.datasource.druid.url| jdbc URL |
jdbc:mysql://127.0.0.1:3306/apache_inlong_audit?characterEncoding=utf8&useSSL=false&serverTimezone=GMT%2b8&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=CONVERT_TO_NULL&allowPublicKeyRetrieval=true
| |
-| spring.datasource.druid.username | account name | root | |
-| spring.datasource.druid.password | password | inlong | |
-
-### ElasticSearch configuration
-
-| Parameter | Description | Default
value | Notes |
-|--------------------------|----------------------------|------|-----|
-| elasticsearch.host | broker | 127.0.0.1 | |
-| elasticsearch.port | port | 9200 | |
-| elasticsearch.authEnable | Whether security certification is required |
false | |
-| elasticsearch.username | account name | elastic | |
-| elasticsearch.password | password | inlong | |
-| elasticsearch.indexDeleteDay | Index retention time, days | 5 | |
-| elasticsearch.auditIdSet | List of audit IDs that are allowed to write |
1,2 | |
-
-### clickhouse configuration
-
-| Parameter | Description | Default value
| Notes |
-|---------------------------|--------------------|-------|---------------------------------------------|
-| clickhouse.driver | Set the driver type |
ru.yandex.clickhouse.ClickHouseDriver |
|
-| clickhouse.url| clickhouse URL | jdbc:clickhouse://127.0.0.1:8123/Default
value | |
-| clickhouse.username | account name | Default value | |
-| clickhouse.password | password | Default value | |
+| Parameter | Description
| Default value |
Notes |
+|----------------------------|---------------------------------------------|-------------------------------------------------------|---------------|
+| audit.config.proxy.type | MQ type | pulsar
| |
+| audit.pulsar.server.url | pulsar broker |
pulsar://127.0.0.1:6650 | |
+| audit.pulsar.topic | pulsar topic | persistent://public/Default
value/inlong-audit | |
+| audit.pulsar.consumer.sub.name | consumer | inlong-audit-subscription
| |
+| audit.pulsar.enable.auth | Whether security certification is required |
false | |
+| audit.pulsar.token | pulsar authentication token |
| |
-### StarRocks configuration
+### MySQL / StarRocks Configuration
| Parameter | Description | Default value | Notes
|
|---------------------|---------------|-----------------|---------------------------------------------|
-| jdbc.driver | Driver type | com.mysql.cj.jdbc.Driver |
|
| jdbc.url | StarRocks URL | jdbc:mysql://127.0.0.1:8123/default |
|
| jdbc.username | account name | default | |
| jdbc.password | password | default | |
-## OpenAPI audit-service
+## OpenAPI Audit Service
configuration `audit-service.properties`
| Parameter | Description | Default value | Notes
|
diff --git a/docs/modules/audit/img/audit_api.png
b/docs/modules/audit/img/audit_api.png
deleted file mode 100644
index 49367978e0..0000000000
Binary files a/docs/modules/audit/img/audit_api.png and /dev/null differ
diff --git a/docs/modules/audit/img/audit_architecture.png
b/docs/modules/audit/img/audit_architecture.png
index 3539c489f9..d9b003f442 100644
Binary files a/docs/modules/audit/img/audit_architecture.png and
b/docs/modules/audit/img/audit_architecture.png differ
diff --git a/docs/modules/audit/img/audit_mysql.png
b/docs/modules/audit/img/audit_mysql.png
deleted file mode 100644
index cb21c5200f..0000000000
Binary files a/docs/modules/audit/img/audit_mysql.png and /dev/null differ
diff --git a/docs/modules/audit/img/audit_openapi.png
b/docs/modules/audit/img/audit_openapi.png
deleted file mode 100644
index c7bc0f9774..0000000000
Binary files a/docs/modules/audit/img/audit_openapi.png and /dev/null differ
diff --git a/docs/modules/audit/img/audit_proxy.png
b/docs/modules/audit/img/audit_proxy.png
deleted file mode 100644
index 25342daf79..0000000000
Binary files a/docs/modules/audit/img/audit_proxy.png and /dev/null differ
diff --git a/docs/modules/audit/img/audit_sdk.png
b/docs/modules/audit/img/audit_sdk.png
deleted file mode 100644
index a63e89c3dd..0000000000
Binary files a/docs/modules/audit/img/audit_sdk.png and /dev/null differ
diff --git a/docs/modules/audit/img/audit_sdk_disaster_recovery.png
b/docs/modules/audit/img/audit_sdk_disaster_recovery.png
deleted file mode 100644
index 0f43f2c3e8..0000000000
Binary files a/docs/modules/audit/img/audit_sdk_disaster_recovery.png and
/dev/null differ
diff --git a/docs/modules/audit/img/audit_ui.png
b/docs/modules/audit/img/audit_ui.png
deleted file mode 100644
index 9b63bcf398..0000000000
Binary files a/docs/modules/audit/img/audit_ui.png and /dev/null differ
diff --git a/docs/modules/audit/img/elasticsearch_index.png
b/docs/modules/audit/img/elasticsearch_index.png
deleted file mode 100644
index 41b654a618..0000000000
Binary files a/docs/modules/audit/img/elasticsearch_index.png and /dev/null
differ
diff --git a/docs/modules/audit/img/elasticsearch_overview.png
b/docs/modules/audit/img/elasticsearch_overview.png
deleted file mode 100644
index 82f1f6e7cb..0000000000
Binary files a/docs/modules/audit/img/elasticsearch_overview.png and /dev/null
differ
diff --git a/docs/modules/audit/img/elasticsearch_write.png
b/docs/modules/audit/img/elasticsearch_write.png
deleted file mode 100644
index 2f5a6ad0df..0000000000
Binary files a/docs/modules/audit/img/elasticsearch_write.png and /dev/null
differ
diff --git a/docs/modules/audit/overview.md b/docs/modules/audit/overview.md
index 3f1c5bfc52..834779bc55 100644
--- a/docs/modules/audit/overview.md
+++ b/docs/modules/audit/overview.md
@@ -13,12 +13,21 @@ The transmission status of each module, and whether the
data stream is lost or r
1. The audit SDK is nested in the service that needs to be audited, audits the
service, and sends the audit result to the audit access layer
2. The audit proxy writes audit data to MQ (Pulsar, Kafka or TubeMQ)
-3. The distribution service consumes the audit data of MQ, and writes the
audit data to MySQL, Elasticsearch and ClickHouse.
-4. The interface layer encapsulates the data of MySQL, Elasticsearch and
ClickHouse.
+3. The distribution service consumes the audit data of MQ, and writes the
audit data to MySQL or StarRocks.
+4. The interface layer encapsulates the data of MySQL or StarRocks.
5. Application scenarios mainly include report display, audit reconciliation,
etc.
6. Support audit and reconciliation of data supplementary recording scenarios.
7. Support audit reconciliation in Flink checkpoint scenarios.
+## Module
+
+| Modules | Description
|
+|:----------------------------|:---------------------------------------------------------------------------------------------|
+| audit-sdk | Audit hidden points are reported. Each module
uses the SDK to report audit data |
+| audit-proxy | Audit proxy layer, receives data reported by
SDK and forwards it to MQ (pulsar/kafka/tubeMQ) |
+| audit-store | Audit storage layer, supporting common JDBC
protocol |
+| audit-service | Audit service layer, providing aggregation,
cache, OpenAPI and other capabilities |
+
## Audit Dimension
| | | || | | | | | |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
@@ -34,11 +43,9 @@ The receiving and sending of each module are respectively an
independent audit i
|Inlong Agent Send Successfully |4|
|Inlong DataProxy Received Successfully |5|
|Inlong DataProxy Send Successfully |6|
-|Inlong Sort Received Successfully |7|
-|Inlong Sort Send Successfully |8|
## Data Transfer Protocol
-The transmission protocol between sdk, access layer, and distribution layer is
Protocol Buffers
+The transmission protocol between SDK, Audit Proxy, and Audit Store is
Protocol Buffers
```markdown
syntax = "proto3";
@@ -96,38 +103,4 @@ message AuditReply {
RSP_CODE rsp_code = 1;
optional string message = 2;
}
-```
-## Audit SDK Implementation Details
-### Target
-***1. Support local disaster recovery***
-***2. Data Uniqueness***
-***3. Reduce data loss caused by abnormal restart***
-
-### Service Discovery
-Audit name discovery between sdk and access layer, support plug-in, including
domain name, vip, etc.
-
-### Disaster Recovery
-* When the SDK fails to send the access layer, it will be placed in the
failure queue.
-* When the failure queue reaches the threshold, it will be written to the
local disaster recovery file.
-* When the local disaster recovery file reaches the threshold, the old data
will be eliminated (eliminated by time).
-
-## Access layer Implementation
-### Target
-***1.High reliability***
-***2.at least once***
-
-## Distribution Implementation
-### Target
-***1. High real-time performance (minute level)***
-***2. Can operate tens of billions of audit data per day***
-***3. Can be deduplicated***
-
-## OpenAPI Implementation
-### Architecture
-
-* The audit interface layer provides OpenAPI capabilities to the outside world
through real-time aggregation and local caching of multiple audit data sources.
-
-### UI Interface display
-### Architecture
-
-* The front-end page pulls the audit data of each module through the interface
layer and displays it.
\ No newline at end of file
+```
\ No newline at end of file
diff --git a/docs/modules/audit/quick_start.md
b/docs/modules/audit/quick_start.md
index a3004639b9..0010457bec 100644
--- a/docs/modules/audit/quick_start.md
+++ b/docs/modules/audit/quick_start.md
@@ -54,46 +54,36 @@ The configuration file is `conf/application.properties`.
# proxy.type: pulsar / tube / kafka
audit.config.proxy.type=pulsar
-# store.server: mysql / elasticsearch / clickhouse
-audit.config.store.mode=mysql
+# Supports common JDBC protocol
+audit.config.store.mode=jdbc
# manger config
manager.hosts=127.0.0.1:8083
proxy.cluster.tag=default_cluster
-# audit pulsar config (optional)
+# pulsar config
audit.pulsar.topic=persistent://public/default/inlong-audit
-audit.pulsar.consumer.sub.name=sub-audit
+audit.pulsar.consumer.sub.name=inlong-audit-subscription
+audit.pulsar.token=
+audit.pulsar.enable.auth=false
-# audit tube config (optional)
+# tube config
audit.tube.topic=inlong-audit
audit.tube.consumer.group.name=inlong-audit-consumer
-# kafka config (optional)
+# kafka config
audit.kafka.topic=inlong-audit
+# create a topic if the topic does not exist.
+audit.kafka.topic.numPartitions=3
+audit.kafka.topic.replicationFactor=2
audit.kafka.consumer.name=inlong-audit-consumer
audit.kafka.group.id=audit-consumer-group
-# mysql config
-spring.datasource.druid.url=jdbc:mysql://127.0.0.1:3306/apache_inlong_audit?characterEncoding=utf8&useSSL=false&serverTimezone=GMT%2b8&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=CONVERT_TO_NULL
-spring.datasource.druid.username=root
-spring.datasource.druid.password=inlong
-
-# es config (optional)
-elasticsearch.host=127.0.0.1
-elasticsearch.port=9200
-
-# clickhouse config (optional)
-clickhouse.driver=ru.yandex.clickhouse.ClickHouseDriver
-clickhouse.url=jdbc:clickhouse://127.0.0.1:8123/default
-clickhouse.username=default
-clickhouse.password=default
-
-# starrocks config (optional)
+# Generic jdbc storage
jdbc.driver=com.mysql.cj.jdbc.Driver
-jdbc.url=jdbc:mysql://127.0.0.1:9020/apache_inlong_audit?characterEncoding=utf8&useSSL=false&serverTimezone=GMT%2b8&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=CONVERT_TO_NULL
-jdbc.username=*******
-jdbc.password=********
+jdbc.url=jdbc:mysql://127.0.0.1:3306/apache_inlong_audit?characterEncoding=utf8&useSSL=false&serverTimezone=GMT%2b8&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=CONVERT_TO_NULL
+jdbc.username=root
+jdbc.password=inlong
```
### Dependencies
@@ -111,14 +101,14 @@ bash +x ./bin/store-start.sh
The configuration file is `conf/audit-service.properties`
```Shell
mysql.jdbc.url=jdbc:mysql://127.0.0.1:3306/apache_inlong_audit?characterEncoding=utf8&useUnicode=true&rewriteBatchedStatements=true
-mysql.username=*****
-mysql.password=*****
+mysql.username=root
+mysql.password=inlong
```
-### Configure audit data sources
-In the audit_source_config table used by the Audit Service, configure the data
source for audit storage.
+### (optional) Configure Audit Data Sources
+In the audit_source_config table used by the Audit Service, configure the data
source for audit storage. By default, the same MySQL configuration is used as
the Audit Service
-### Configure audit audit items
-In the audit_id_config table used by the Audit Service, configure the audit
items that need to be cached.
+### (optional) Configure Audit Items
+In the audit_id_config table used by the Audit Service, configure the audit
items that need to be cached. By default, Agent is used to receive
successfully, Agent is sent successfully, DataProxy is received successfully,
and DataProxy is sent successfully.
### Dependencies
- If the backend database is MySQL, please download
[mysql-connector-java-8.0.28.jar](https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.28/mysql-connector-java-8.0.28.jar)
and put it into `lib/` directory.
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/configure.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/configure.md
index a15d9ab44b..a8dfb21222 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/configure.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/configure.md
@@ -5,106 +5,75 @@ sidebar_position: 3
## 概览
-* 审计代理服务audit-proxy在`audit-proxy-{tube|pulsar|kafka}.conf`中设置。
-* 审计存储服务audit-store在`application.properties`中设置。
-* 审计OpenAPI服务audit-service`在audit-service.properties`中配置。
+* 审计代理服务 audit-proxy 在 `audit-proxy-{tube|pulsar|kafka}.conf` 中设置。
+* 审计存储服务 audit-store 在 `application.properties` 中设置。
+* 审计 OpenAPI 服务 audit-service `在audit-service.properties` 中配置。
-## 审计代理层 audit-proxy
+## 审计代理层 Audit Proxy
配置（audit-proxy-{tube|pulsar|kafka}.conf）
### 通用设置
-| 参数 | 描述 | 默认值
| 备注
|
-|----------------------------|---------------------------------------------|-------------------------------------------------------|--------------------------------------------------------|
-| agent1.sources | source类型 | tcp-source
| |
-| agent1.channels | 使用的channel | ch-msg1
|
|
-| agent1.sinks | 使用的sink | pulsar-sink-msg1
| |
+| 参数 | 描述 | 默认值
| 备注 |
+|----------------------------|-------------|-------------------------------------------------------|--------------------------------------------------------|
+| agent1.sources | source 类型 | tcp-source
| |
+| agent1.channels | 使用的 channel | ch-msg1
| |
+| agent1.sinks | 使用的 sink | pulsar-sink-msg1
| |
-### sources 相关设置
+### Sources 相关设置
-| 参数 | 描述 | 默认值 | 备注
|
-|---------------------------|--------------------|-------|---------------------------------------------|
-| agent1.sources.tcp-source.channels | 定义source中使用到的channel | ch-msg1 |
|
-| agent1.sources.tcp-source.host | 服务ip | 0.0.0.0 |
|
-| agent1.sources.tcp-source.port | 监听端口 | 10081 | |
+| 参数 | 描述 | 默认值 | 备注
|
+|---------------------------|-------------------------|-------|---------------------------------------------|
+| agent1.sources.tcp-source.channels | 定义 source 中使用到的 channel | ch-msg1 |
|
+| agent1.sources.tcp-source.host | 服务 ip | 0.0.0.0 |
|
+| agent1.sources.tcp-source.port | 监听端口 | 10081 |
|
-### channel 相关配置
+### Channel 相关配置
-| 参数 | 描述 | 默认值 | 备注 |
-|--------------------------|----------------------------|------|-----|
-| agent1.channels.ch-msg1.type | memory channel类型 | memory | |
-| agent1.channels.ch-msg2.type | file channel类型 | file | |
+| 参数 | 描述 | 默认值 | 备注 |
+|--------------------------|-------------------|------|-----|
+| agent1.channels.ch-msg1.type | memory channel 类型 | memory | |
+| agent1.channels.ch-msg2.type | file channel 类型 | file | |
-### sink 相关配置
+### Sink 相关配置
-| 参数 | 描述 | 默认值 | 备注 |
-|------------------------|----------------------------------|-------|-----|
-| agent1.sinks.pulsar-sink-msg1.channel | sink的上游channel名称 |
ch-msg1 | |
-| agent1.sinks.pulsar-sink-msg1.type | sink类实现 |
org.apache.inlong.audit.sink.PulsarSink | |
-| agent1.sinks.pulsar-sink-msg1.pulsar_server_url | pulsar集群broker节点
| pulsar://localhost:6650 | |
-| agent1.sinks.pulsar-sink-msg1.topic | pulsar集群topic |
persistent://public/default/inlong-audit | |
-| agent1.sinks.pulsar-sink-msg1.enable_token_auth | 是否需要安全认证 | false |
|
-| agent1.sinks.pulsar-sink-msg1.auth_token | pulsar认证token | 空 | |
+| 参数 | 描述 | 默认值 | 备注 |
+|------------------------|---------------------|-------|-----|
+| agent1.sinks.pulsar-sink-msg1.channel | sink 的上游 channel 名称 | ch-msg1 | |
+| agent1.sinks.pulsar-sink-msg1.type | sink 类实现 |
org.apache.inlong.audit.sink.PulsarSink | |
+| agent1.sinks.pulsar-sink-msg1.pulsar_server_url | pulsar 集群 broker 节点 |
pulsar://localhost:6650 | |
+| agent1.sinks.pulsar-sink-msg1.topic | pulsar 集群 topic |
persistent://public/default/inlong-audit | |
+| agent1.sinks.pulsar-sink-msg1.enable_token_auth | 是否需要安全认证 |
false | |
+| agent1.sinks.pulsar-sink-msg1.auth_token | pulsar 认证 token | 空 |
|
-## 审计存储服务 audit-store
+## 审计存储服务 Audit Store
配置 `application.properties`
### MQ配置
-| 参数 | 描述 | 默认值
| 备注
|
-|----------------------------|---------------------------------------------|-------------------------------------------------------|--------------------------------------------------------|
-| audit.config.proxy.type | MQ类型 | pulsar
| |
-| audit.pulsar.server.url | pulsar的集群地址 |
pulsar://127.0.0.1:6650 |
|
+| 参数 | 描述 | 默认值
| 备注 |
+|----------------------------|--------------|-------------------------------------------------------|--------------------------------------------------------|
+| audit.config.proxy.type | MQ 类型 | pulsar
| |
+| audit.pulsar.server.url | pulsar 的集群地址 | pulsar://127.0.0.1:6650
|
|
| audit.pulsar.topic | pulsar topic |
persistent://public/default/inlong-audit
| |
-| audit.pulsar.consumer.sub.name | 订阅组 | inlong-audit-subscription
|
|
-| audit.pulsar.enable.auth | 是否需要安全认证 | false
| |
-| audit.pulsar.token | 认证token |
| |
-
-### DB 相关设置
-
-| 参数 | 描述 | 默认值 | 备注
|
-|---------------------------|--------------------|-------|---------------------------------------------|
-| spring.datasource.druid.driver-class-name | 设置Driver类型 |
com.mysql.cj.jdbc.Driver | |
-| spring.datasource.druid.url| jdbc的URL |
jdbc:mysql://127.0.0.1:3306/apache_inlong_audit?characterEncoding=utf8&useSSL=false&serverTimezone=GMT%2b8&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=CONVERT_TO_NULL&allowPublicKeyRetrieval=true
| |
-| spring.datasource.druid.username | 账号名 | root | |
-| spring.datasource.druid.password | 密码 | inlong | |
-
-### ElasticSearch 配置
-
-| 参数 | 描述 | 默认值 | 备注 |
-|--------------------------|----------------------------|------|-----|
-| elasticsearch.host | 集群地址 | 127.0.0.1 | |
-| elasticsearch.port | 集群端口 | 9200 | |
-| elasticsearch.authEnable | 是否认证 | false | |
-| elasticsearch.username | 账号名 | elastic | |
-| elasticsearch.password | 密码 | inlong | |
-| elasticsearch.indexDeleteDay | 索引保存时间，天 | 5 | |
-| elasticsearch.auditIdSet | 允许写入的审计ID列表 | 1,2 | |
-
-### ClickHouse 相关配置
-
-| 参数 | 描述 | 默认值 | 备注
|
-|---------------------------|--------------------|-------|---------------------------------------------|
-| clickhouse.driver | 设置Driver类型 | ru.yandex.clickhouse.ClickHouseDriver |
|
-| clickhouse.url| clickhouse的URL | jdbc:clickhouse://127.0.0.1:8123/default |
|
-| clickhouse.username | 账号名 | default | |
-| clickhouse.password | 密码 | default | |
-
-### StarRocks 相关配置
-
-| 参数 | 描述 | 默认值
| 备注 |
-|---------------------|--------------------|---------------------------------------|---------------------------------------------|
-| jdbc.driver | 设置Driver类型 | com.mysql.cj.jdbc.Driver |
|
-| jdbc.url | StarRocks的URL | jdbc:mysql://127.0.0.1:8123/default |
|
-| jdbc.username | 账号名 | default | |
-| jdbc.password | 密码 | default | |
-
-## OpenAPI服务 audit-service
+| audit.pulsar.consumer.sub.name | 订阅组 |
inlong-audit-subscription |
|
+| audit.pulsar.enable.auth | 是否需要安全认证 | false
| |
+| audit.pulsar.token | 认证 token |
| |
+
+### MySQL / StarRocks 相关配置
+
+| 参数 | 描述 | 默认值
| 备注 |
+|---------------------|-----------------|---------------------------------------|---------------------------------------------|
+| jdbc.url | StarRocks 的 URL | jdbc:mysql://127.0.0.1:8123/default |
|
+| jdbc.username | 账号名 | default | |
+| jdbc.password | 密码 | default | |
+
+## OpenAPI服务 Audit Service
配置 `audit-service.properties`
-| 参数 | 描述 | 默认值 | 备注
|
-|-------------------|-----------|-------|---------------------------------------------|
-| mysql.jdbc.url | mysql的URL | jdbc:mysql://127.0.0.1:8123/default |
|
-| mysql.username | 账号名 | default | |
-| mysql.password | 密码 | default | |
\ No newline at end of file
+| 参数 | 描述 | 默认值 | 备注
|
+|-------------------|-------------|-------|---------------------------------------------|
+| mysql.jdbc.url | mysql 的 URL | jdbc:mysql://127.0.0.1:8123/default |
|
+| mysql.username | 账号名 | default | |
+| mysql.password | 密码 | default | |
\ No newline at end of file
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_api.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_api.png
deleted file mode 100644
index 240f1d3daf..0000000000
Binary files
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_api.png
and /dev/null differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_architecture.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_architecture.png
index 3539c489f9..d9b003f442 100644
Binary files
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_architecture.png
and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_architecture.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_mysql.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_mysql.png
deleted file mode 100644
index 5ccd355236..0000000000
Binary files
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_mysql.png
and /dev/null differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_openapi.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_openapi.png
deleted file mode 100644
index c7bc0f9774..0000000000
Binary files
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_openapi.png
and /dev/null differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_proxy.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_proxy.png
deleted file mode 100644
index faf160ce64..0000000000
Binary files
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_proxy.png
and /dev/null differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_sdk.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_sdk.png
deleted file mode 100644
index 243403daa8..0000000000
Binary files
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_sdk.png
and /dev/null differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_sdk_disaster_recovery.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_sdk_disaster_recovery.png
deleted file mode 100644
index e07f7ca3e6..0000000000
Binary files
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_sdk_disaster_recovery.png
and /dev/null differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_ui.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_ui.png
deleted file mode 100644
index 4de5f2e889..0000000000
Binary files
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/audit_ui.png
and /dev/null differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/elasticsearch_index.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/elasticsearch_index.png
deleted file mode 100644
index a93f2a17bb..0000000000
Binary files
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/elasticsearch_index.png
and /dev/null differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/elasticsearch_overview.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/elasticsearch_overview.png
deleted file mode 100644
index e472566ba8..0000000000
Binary files
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/elasticsearch_overview.png
and /dev/null differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/elasticsearch_write.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/elasticsearch_write.png
deleted file mode 100644
index 0fefe47677..0000000000
Binary files
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/img/elasticsearch_write.png
and /dev/null differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/overview.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/overview.md
index df85b260a3..e0b925d893 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/overview.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/overview.md
@@ -3,45 +3,50 @@ title: 总览
sidebar_position: 1
---
-* InLong审计是独立于InLong的一个子系统，对InLong系统的Agent、DataProxy、Sort模块的入流量、出流量进行实时审计对账。
+* InLong 审计是独立于 InLong 的一个子系统，对 InLong 系统的 Agent、DataProxy、Sort
模块的入流量、出流量进行实时审计对账。
* 对账的粒度有分钟、10分钟、30分钟、小时、天等等。
-审计对账以日志上报时间为统一的口径，参与审计的各个服务将按照相同的日志时间进行实时对账。通过审计对账，我们可以清晰的了解InLong
+审计对账以日志上报时间为统一的口径，参与审计的各个服务将按照相同的日志时间进行实时对账。通过审计对账，我们可以清晰的了解 InLong
各个模块的传输情况，以及数据流是否有丢失或者重复
## 架构
1. 审计SDK嵌套在需要审计的服务，对服务进行审计，将审计结果发送到审计接入层。
-2. 审计接入层将审计数据写到 MQ(Pulsar、Kafka 或者 TubeMQ)。
-3. 分发服务消费 MQ 的审计数据，将审计数据写到 MySQL、Elasticsearch、ClickHouse、StarRocks。
-4. 接口层将 MySQL、Elasticsearch、ClickHouse、StarRocks的数据进行实时聚合并且cache,对外提供OpenAPI。
+2. 审计接入层将审计数据写到 MQ (Pulsar、Kafka 或者 TubeMQ)。
+3. 分发服务消费 MQ 的审计数据，将审计数据写到 MySQL、StarRocks。
+4. 接口层将 MySQL、StarRocks 的数据进行实时聚合并且 cache,对外提供 OpenAPI。
5. 应用场景主要包括报表展示、审计对账等等。
6. 支持数据补录场景的审计对账。
-7. 支持Flink checkpoint场景的审计对账。
+7. 支持 Flink CheckPoint 场景的审计对账。
+
+## 模块
+
+| 模块 | 描述 |
+|:---------------|:---------------------------------------------------|
+| audit-sdk | 审计埋点上报，各个模块使用该 SDK 上报审计数据 |
+| audit-proxy | 审计代理层，接收 SDK 上报数据，转发到 MQ (pulsar / kafka / tubeMQ) |
+| audit-store | 审计存储层，支持通用的 JDBC 协议 |
+| audit-service | 审计服务层，提供聚合、cache、OpenAPI 等能力 |
## 审计维度
-| | | || | | | | | |
-| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
-| 机器ip | 容器ID | 线程ID | 日志时间(分钟) | 审计ID | inlong_group_id | inlong_stream_id |
条数 | 大小 | 传输时延(ms) |
-
-## 审计项ID
-每个模块的接收与发送分别为一个独立的审计项ID
-
-|Inlong服务模块 |审计ID |
-|----|----|
-|Inlong api接收成功 |1 |
-|Inlong api发送成功 |2|
-|Inlong agent接收成功 |3|
-|Inlong agent发送成功 |4|
-|Inlong DataProxy接收成功 |5|
-|Inlong DataProxy发送成功 |6|
-|Inlong分发服务1接收成功 |7|
-|Inlong分发服务1发送成功 |8|
-|Inlong分发服务2接收成功 |9|
-|Inlong分发服务2发送成功 |10|
+| | | || | | | | | |
+|-------|-------|-------| ---- |-------| ---- | ---- | ---- | ---- | ---- |
+| 机器 ip | 容器 ID | 线程 ID | 日志时间(分钟) | 审计 ID | inlong_group_id |
inlong_stream_id | 条数 | 大小 | 传输时延(ms) |
+
+## 审计项 ID
+每个模块的接收与发送分别为一个独立的审计项 ID
+
+| InLong 服务模块 | 审计 ID |
+|------------------------|-------|
+| InLong API 接收成功 | 1 |
+| InLong API 发送成功 | 2 |
+| InLong Agent 接收成功 | 3 |
+| InLong Agent 发送成功 | 4 |
+| InLong DataProxy 接收成功 | 5 |
+| InLong DataProxy 发送成功 | 6 |
## 数据传输协议
-sdk、接入层、分发层之间的传输协议为Protocol Buffers
+SDK、接入层、分发层之间的传输协议为 Protocol Buffers
```markdown
syntax = "proto3";
@@ -68,69 +73,35 @@ message Pong {
}
message AuditRequest {
- AuditMessageHeader msg_header = 1; //包头
- repeated AuditMessageBody msg_body = 2; //包体
+ AuditMessageHeader msg_header = 1; // 包头
+ repeated AuditMessageBody msg_body = 2; // 包体
}
message AuditMessageHeader {
- string ip = 1; //sdk客户端ip
- string docker_id = 2; //sdk所在容器ID
- string thread_id = 3; //sdk所在的线程ID
- uint64 sdk_ts = 4; //sdk上报时间
- uint64 packet_id = 5; //sdk上报的包ID
+ string ip = 1; // SDK 客户端 ip
+ string docker_id = 2; // SDK 所在容器 ID
+ string thread_id = 3; // SDK 所在的线程 ID
+ uint64 sdk_ts = 4; // SDK 上报时间
+ uint64 packet_id = 5; // SDK 上报的包 ID
}
message AuditMessageBody {
- uint64 log_ts = 1; //日志时间
- string inlong_group_id= 2; //inlong_group_id
- string inlong_stream_id= 3; //inlong_stream_id
- string audit_id = 4; //审计ID
- uint64 count = 5; //条数
- uint64 size = 6; //大小
- int64 delay = 7; //总传输延时
+ uint64 log_ts = 1; // 日志时间
+ string inlong_group_id= 2; // InLong Group ID
+ string inlong_stream_id= 3; // InLong Stream ID
+ string audit_id = 4; // 审计 ID
+ uint64 count = 5; // 条数
+ uint64 size = 6; // 大小
+ int64 delay = 7; // 总传输延时
}
message AuditReply {
enum RSP_CODE {
- SUCCESS = 0; //成功
- FAILED = 1; //失败
- DISASTER = 2; //容灾
+ SUCCESS = 0; // 成功
+ FAILED = 1; // 失败
+ DISASTER = 2; // 容灾
}
- RSP_CODE rsp_code = 1; //服务端返回码
+ RSP_CODE rsp_code = 1; // 服务端返回码
optional string message = 2;
}
-```
-## 审计SDK实现
-### 目标
-***1.支持本地容灾***
-***2.数据唯一性***
-***3.减少异常重启导致的数据丢失***
-
-### 服务发现
-* 审计sdk与接入层之间的名字发现，支持插件化，包括域名、vip等
-
-### 容灾逻辑
-* sdk发送接入层失败时，会放入失败队列
-* 失败队列达到阈值时，将写到本地容灾文件
-* 本地容灾文件达到阈值时，将淘汰旧数据(按时间淘汰)
-
-## 接入层实现
-### 目标
-***1.高可靠***
-***2.at least once***
-
-## 分发层实现
-### 目标
-***1.高实时性(分钟级)***
-***2.部署简单***
-***3.可去重***
-
-## OpenAPI实现
-### 主要逻辑图
-
-* 审计接口层通过对多个审计数据源进行实时聚合、本地cache，对外提供OpenAPI能力。
-
-### UI 界面展示
-### 主要逻辑图
-
-* 前端页面通过接口层，拉取各个模块的审计数据，进行展示
\ No newline at end of file
+```
\ No newline at end of file
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/quick_start.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/quick_start.md
index d10280fa07..db9f2dcd48 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/quick_start.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/quick_start.md
@@ -2,13 +2,13 @@
title: 安装部署
---
-所有的安装文件都在 `inlong-audit` 目录下，如果使用 MySQL
存储审计数据，需要先通过`sql/apache_inlong_audit_mysql.sql`初始化数据库。
+所有的安装文件都在 `inlong-audit` 目录下，如果使用 MySQL 存储审计数据，需要先通过
`sql/apache_inlong_audit_mysql.sql` 初始化数据库。
```shell
# 初始化 database
mysql -uDB_USER -pDB_PASSWD < sql/apache_inlong_audit_mysql.sql
```
-如果使用 StarRocks 存储审计数据，需要先通过`sql/apache_inlong_audit_starrocks.sql`初始化数据库。
+如果使用 StarRocks 存储审计数据，需要先通过 `sql/apache_inlong_audit_starrocks.sql` 初始化数据库。
```shell
# 初始化 StarRocks database
mysql -uDB_USER -pDB_PASSWD < sql/apache_inlong_audit_starrocks.sql
@@ -57,45 +57,36 @@ Audit Proxy 默认监听端口为 `10081`。
# proxy.type: pulsar / tube / kafka
audit.config.proxy.type=pulsar
-# store.server: mysql / elasticsearch
-audit.config.store.mode=mysql
+# Supports common JDBC protocol
+audit.config.store.mode=jdbc
# manger config
manager.hosts=127.0.0.1:8083
proxy.cluster.tag=default_cluster
-# audit pulsar config (optional)
+# pulsar config
audit.pulsar.topic=persistent://public/default/inlong-audit
-audit.pulsar.consumer.sub.name=sub-audit
+audit.pulsar.consumer.sub.name=inlong-audit-subscription
+audit.pulsar.token=
+audit.pulsar.enable.auth=false
-# audit tube config (optional)
+# tube config
audit.tube.topic=inlong-audit
audit.tube.consumer.group.name=inlong-audit-consumer
-# kafka config (optional)
+# kafka config
audit.kafka.topic=inlong-audit
+# create a topic if the topic does not exist.
+audit.kafka.topic.numPartitions=3
+audit.kafka.topic.replicationFactor=2
audit.kafka.consumer.name=inlong-audit-consumer
audit.kafka.group.id=audit-consumer-group
-# mysql config
-spring.datasource.druid.url=jdbc:mysql://127.0.0.1:3306/apache_inlong_audit?characterEncoding=utf8&useSSL=false&serverTimezone=GMT%2b8&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=CONVERT_TO_NULL
-spring.datasource.druid.username=root
-spring.datasource.druid.password=inlong
-
-# es config (optional)
-elasticsearch.host=127.0.0.1
-elasticsearch.port=9200
-
-# clickhouse config (optional)
-clickhouse.url=jdbc:clickhouse://127.0.0.1:8123/default
-clickhouse.username=default
-clickhouse.password=default
-
-# starrocks config (optional)
+# Generic jdbc storage
jdbc.driver=com.mysql.cj.jdbc.Driver
-jdbc.url=jdbc:mysql://127.0.0.1:9020/apache_inlong_audit?characterEncoding=utf8&useSSL=false&serverTimezone=GMT%2b8&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=CONVERT_TO_NULL
-jdbc.username=*******
-jdbc.password=********
+jdbc.url=jdbc:mysql://127.0.0.1:3306/apache_inlong_audit?characterEncoding=utf8&useSSL=false&serverTimezone=GMT%2b8&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=CONVERT_TO_NULL
+jdbc.username=root
+jdbc.password=inlong
```
### 依赖
@@ -112,14 +103,14 @@ bash +x ./bin/store-start.sh
配置文件 `conf/audit-service.properties`
```Shell
mysql.jdbc.url=jdbc:mysql://127.0.0.1:3306/apache_inlong_audit?characterEncoding=utf8&useUnicode=true&rewriteBatchedStatements=true
-mysql.username=*****
-mysql.password=*****
+mysql.username=root
+mysql.password=inlong
```
-#### 配置审计数据源
-在Audit Service服务使用的audit_source_config表中，配置审计存储的数据源。
+#### (可选)配置审计数据源
+在 Audit Service 服务使用的 audit_source_config 表中，配置审计存储的数据源。默认和 Audit Service
使用相同的 MySQL 配置
-#### 配置审计审计项
-在Audit Service服务使用的audit_id_config表中，配置需要cache的审计项。
+#### (可选)配置审计审计项
+在 Audit Service 服务使用的 audit_id_config 表中，配置需要 cache 的审计项。默认使用 Agent 接收成功、Agent
发送成功、DataProxy 接收成功、DataProxy 发送成功
### 依赖
- 如果后端连接 MySQL 数据库，请下载
[mysql-connector-java-8.0.28.jar](https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.26/mysql-connector-java-8.0.28.jar),
并将其放入 `lib/` 目录。
