Re: [PR] [INLONG-984][Doc] Add data node management document [inlong-website]
fuweng11 commented on code in PR #987: URL: https://github.com/apache/inlong-website/pull/987#discussion_r1774303971 ## i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/node_management.md: ## @@ -0,0 +1,47 @@ +--- +title: 节点管理 +sidebar_position: 4 +--- + +节点管理主要提供数据节点信息复用的能力，用户可以在新建、修改、删除数据节点，并且在进行数据接入时，复用配置好的数据节点信息： + +### 新建数据节点 + +用户可以通过新建数据节点来生成一个常用的节点信息： + + + +- 节点名：用户自定义名，用来标识此节点信息 +- 类型：节点类型 +- 责任人：该节点负责人，只有责任人能够修改该节点的配置信息 +- 描述：节点描述信息 + +以上为节点的公用配置，另外，根据不同的节点类型，还会有不同的配置信息，如 ClickHouse 节点需要填写用户名、密码、ClickHouse 地址信息。 + +### 删除数据节点 + +用户可以对已创建节点进行删除，删除后此节点将停止使用： + + + +### 修改数据节点 + +用户可以修改节点信息，点击【编辑】，输入新的节点信息，确认后此节点配置将生效： + + + +### 测试连接 + +用户通过测试连接校验节点状态是否能够正常： + + + +### 使用节点 + +用户配置好节点后，可以在数据订阅中使用已经配置好的节点信息： + +- 选择`数据接入`，点击`详情` Review Comment: Fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-984][Doc] Add data node management document [inlong-website]
fuweng11 commented on code in PR #987: URL: https://github.com/apache/inlong-website/pull/987#discussion_r1774303805 ## i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/node_management.md: ## @@ -0,0 +1,47 @@ +--- +title: 节点管理 +sidebar_position: 4 +--- + +节点管理主要提供数据节点信息复用的能力，用户可以在新建、修改、删除数据节点，并且在进行数据接入时，复用配置好的数据节点信息： + +### 新建数据节点 + +用户可以通过新建数据节点来生成一个常用的节点信息： + + + +- 节点名：用户自定义名，用来标识此节点信息 +- 类型：节点类型 +- 责任人：该节点负责人，只有责任人能够修改该节点的配置信息 +- 描述：节点描述信息 + +以上为节点的公用配置，另外，根据不同的节点类型，还会有不同的配置信息，如 ClickHouse 节点需要填写用户名、密码、ClickHouse 地址信息。 + +### 删除数据节点 + +用户可以对已创建节点进行删除，删除后此节点将停止使用： + + + +### 修改数据节点 + +用户可以修改节点信息，点击【编辑】，输入新的节点信息，确认后此节点配置将生效： + + + +### 测试连接 + +用户通过测试连接校验节点状态是否能够正常： Review Comment: Fixed. ## i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/node_management.md: ## @@ -0,0 +1,47 @@ +--- +title: 节点管理 +sidebar_position: 4 +--- + +节点管理主要提供数据节点信息复用的能力，用户可以在新建、修改、删除数据节点，并且在进行数据接入时，复用配置好的数据节点信息： + +### 新建数据节点 + +用户可以通过新建数据节点来生成一个常用的节点信息： + + + +- 节点名：用户自定义名，用来标识此节点信息 +- 类型：节点类型 +- 责任人：该节点负责人，只有责任人能够修改该节点的配置信息 +- 描述：节点描述信息 + +以上为节点的公用配置，另外，根据不同的节点类型，还会有不同的配置信息，如 ClickHouse 节点需要填写用户名、密码、ClickHouse 地址信息。 + +### 删除数据节点 + +用户可以对已创建节点进行删除，删除后此节点将停止使用： + + + +### 修改数据节点 + +用户可以修改节点信息，点击【编辑】，输入新的节点信息，确认后此节点配置将生效： + + + +### 测试连接 + +用户通过测试连接校验节点状态是否能够正常： + + + +### 使用节点 + +用户配置好节点后，可以在数据订阅中使用已经配置好的节点信息： + +- 选择`数据接入`，点击`详情` +  +- 选择数据目标`新建`，选择已经配置好的数据节点，并完善其他配置后点击保存。 Review Comment: Fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong-website) branch master updated (bd5c3d9eda -> bdc7e44221)
This is an automated email from the ASF dual-hosted git repository. luchunliang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong-website.git from bd5c3d9eda [INLONG-731][BOT] Bump send and express (#978) add bdc7e44221 [INLONG-982][Doc] Add DataProxy SDK document of Python (#983) No new revisions were added by this update. Summary of changes: docs/sdk/dataproxy-sdk/python.md | 102 + .../current/sdk/dataproxy-sdk/python.md| 102 + 2 files changed, 204 insertions(+) create mode 100644 docs/sdk/dataproxy-sdk/python.md create mode 100644 i18n/zh-CN/docusaurus-plugin-content-docs/current/sdk/dataproxy-sdk/python.md
Re: [I] [Improve][Doc] Add DataProxy SDK document of Python [inlong-website]
luchunliang closed issue #982: [Improve][Doc] Add DataProxy SDK document of Python URL: https://github.com/apache/inlong-website/issues/982 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-982][Doc] Add DataProxy SDK document of Python [inlong-website]
luchunliang merged PR #983: URL: https://github.com/apache/inlong-website/pull/983 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11178][Dashboard] Added http sink and http node [inlong]
wohainilaodou commented on code in PR #11181:
URL: https://github.com/apache/inlong/pull/11181#discussion_r1774327305
##
inlong-dashboard/src/plugins/nodes/defaults/Http.ts:
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { DataWithBackend } from '@/plugins/DataWithBackend';
+import { RenderRow } from '@/plugins/RenderRow';
+import { RenderList } from '@/plugins/RenderList';
+import { NodeInfo } from '../common/NodeInfo';
+import i18n from 'i18next';
+import { Input } from 'antd';
+
+const { I18n } = DataWithBackend;
+const { FieldDecorator } = RenderRow;
+
+export default class HttpNodeInfo
+ extends NodeInfo
+ implements DataWithBackend, RenderRow, RenderList
+{
+ @FieldDecorator({
+type: 'input',
+rules: [{ required: true }],
+ })
+ @I18n('meta.Nodes.Http.BaseUrl')
+ baseUrl: string;
+
+ @FieldDecorator({
+type: 'radio',
+rules: [{ required: true }],
+initialValue: false,
+props: {
+ options: [
+{
+ label: i18n.t('是'),
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[I] [Improve][Doc] Add cluster management document [inlong-website]
fuweng11 opened a new issue, #988: URL: https://github.com/apache/inlong-website/issues/988 ### Description Add cluster management document. ### Are you willing to submit PR? - [X] Yes, I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-731][BOT] Bump body-parser and express [inlong-website]
gosonzhang merged PR #979: URL: https://github.com/apache/inlong-website/pull/979 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong-website) branch dependabot/npm_and_yarn/multi-9423f4c335 deleted (was f8f9355d22)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/multi-9423f4c335 in repository https://gitbox.apache.org/repos/asf/inlong-website.git was f8f9355d22 Bump body-parser and express The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(inlong-website) branch master updated: Bump body-parser and express (#979)
This is an automated email from the ASF dual-hosted git repository.
gosonzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new ba8e771fba Bump body-parser and express (#979)
ba8e771fba is described below
commit ba8e771fbaface5ea14b86fe404c55a14acf711f
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
AuthorDate: Tue Sep 24 19:44:37 2024 +0800
Bump body-parser and express (#979)
Bumps [body-parser](https://github.com/expressjs/body-parser) and
[express](https://github.com/expressjs/express). These dependencies needed to
be updated together.
Updates `body-parser` from 1.20.2 to 1.20.3
- [Release notes](https://github.com/expressjs/body-parser/releases)
-
[Changelog](https://github.com/expressjs/body-parser/blob/master/HISTORY.md)
-
[Commits](https://github.com/expressjs/body-parser/compare/1.20.2...1.20.3)
Updates `express` from 4.19.2 to 4.21.0
- [Release notes](https://github.com/expressjs/express/releases)
- [Changelog](https://github.com/expressjs/express/blob/4.21.0/History.md)
- [Commits](https://github.com/expressjs/express/compare/4.19.2...4.21.0)
---
updated-dependencies:
- dependency-name: body-parser
dependency-type: indirect
- dependency-name: express
dependency-type: indirect
...
Signed-off-by: dependabot[bot]
Co-authored-by: dependabot[bot]
<49699333+dependabot[bot]@users.noreply.github.com>
---
package-lock.json | 449 --
1 file changed, 335 insertions(+), 114 deletions(-)
diff --git a/package-lock.json b/package-lock.json
index 0d8ec9ad5e..a8ea8ea4d1 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -4571,9 +4571,9 @@
}
},
"node_modules/body-parser": {
- "version": "1.20.2",
- "resolved":
"https://registry.npmjs.org/body-parser/-/body-parser-1.20.2.tgz";,
- "integrity":
"sha512-ml9pReCu3M61kGlqoTm2umSXTlRTuGTx0bfYj+uIUKKYycG5NtSbeetV3faSU6R7ajOPw0g/J1PvK4qNy7s5bA==",
+ "version": "1.20.3",
+ "resolved":
"https://registry.npmjs.org/body-parser/-/body-parser-1.20.3.tgz";,
+ "integrity":
"sha512-7rAxByjUMqQ3/bHJy7D6OGXvx/MMc4IqBn/X0fcM1QUcAItpZrBEYhWGem+tzXH90c+G01ypMcYJBO9Y30203g==",
"dependencies": {
"bytes": "3.1.2",
"content-type": "~1.0.5",
@@ -4583,7 +4583,7 @@
"http-errors": "2.0.0",
"iconv-lite": "0.4.24",
"on-finished": "2.4.1",
-"qs": "6.11.0",
+"qs": "6.13.0",
"raw-body": "2.5.2",
"type-is": "~1.6.18",
"unpipe": "1.0.0"
@@ -4778,11 +4778,18 @@
}
},
"node_modules/call-bind": {
- "version": "1.0.2",
- "license": "MIT",
+ "version": "1.0.7",
+ "resolved": "https://registry.npmjs.org/call-bind/-/call-bind-1.0.7.tgz";,
+ "integrity":
"sha512-GHTSNSYICQ7scH7sZ+M2rFopRoLh8t2bLSW6BbgrtLsahOIB5iyAVJf9GjWK3cYTDaMj4XdBpM1cA6pIS0Kv2w==",
"dependencies": {
-"function-bind": "^1.1.1",
-"get-intrinsic": "^1.0.2"
+"es-define-property": "^1.0.0",
+"es-errors": "^1.3.0",
+"function-bind": "^1.1.2",
+"get-intrinsic": "^1.2.4",
+"set-function-length": "^1.2.1"
+ },
+ "engines": {
+"node": ">= 0.4"
},
"funding": {
"url": "https://github.com/sponsors/ljharb";
@@ -5985,6 +5992,22 @@
"version": "1.1.3",
"license": "MIT"
},
+"node_modules/define-data-property": {
+ "version": "1.1.4",
+ "resolved":
"https://registry.npmjs.org/define-data-property/-/define-data-property-1.1.4.tgz";,
+ "integrity":
"sha512-rBMvIzlpA8v6E+SJZoo++HAYqsLrkg7MSfIinMPFhmkorw7X+dOXVJQs+QT69zGkzMyfDnIMN2Wid1+NbL3T+A==",
+ "dependencies": {
+"es-define-property": "^1.0.0",
+"es-errors": "^1.3.0",
+"gopd": "^1.0.1"
+ },
+ "engines": {
+"node": ">= 0.4"
+ },
+ "funding": {
+"url": "https://github.com/sponsors/ljharb";
+ }
+},
"node_modules/define-lazy-prop": {
"version": "2.0.0",
"resolved":
"https://registry.npmjs.org/define-lazy-prop/-/define-lazy-prop-2.0.0.tgz";,
@@ -6298,9 +6321,9 @@
}
},
"node_modules/encodeurl": {
- "version": "1.0.2",
- "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-1.0.2.tgz";,
- "integrity":
"sha512-TPJXq8JqFaVYm2CWmPvnP2Iyo4ZSM7/QKcSmuMLDObfpH5fi7RUGmd/rTDf+rut/saiDiQEeVTNgAmJEdAOx0w==",
+ "version": "2.0.0",
+ "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz";,
+ "integrity":
"sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==",
"engines": {
"node": ">= 0.8"
}
@@ -6350,6 +6373,25 @@
"is-arrayish": "^
(inlong-website) branch dependabot/npm_and_yarn/micromatch-4.0.8 deleted (was d0be50715c)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/micromatch-4.0.8 in repository https://gitbox.apache.org/repos/asf/inlong-website.git was d0be50715c Bump micromatch from 4.0.5 to 4.0.8 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
Re: [PR] [INLONG-731][BOT] Bump serve-static and express [inlong-website]
dependabot[bot] commented on PR #977: URL: https://github.com/apache/inlong-website/pull/977#issuecomment-2371032122 Looks like these dependencies are up-to-date now, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong-website) branch dependabot/npm_and_yarn/multi-d66d039ac5 deleted (was e6e094461f)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/multi-d66d039ac5 in repository https://gitbox.apache.org/repos/asf/inlong-website.git was e6e094461f Bump serve-static and express The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
Re: [PR] [INLONG-731][BOT] Bump send and express [inlong-website]
dockerzhang merged PR #978: URL: https://github.com/apache/inlong-website/pull/978 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong-website) branch master updated (b65299a59b -> bd5c3d9eda)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong-website.git from b65299a59b [INLONG-731][BOT] Bump micromatch from 4.0.5 to 4.0.8 (#973) add bd5c3d9eda [INLONG-731][BOT] Bump send and express (#978) No new revisions were added by this update. Summary of changes:
Re: [PR] [INLONG-731][BOT] Bump micromatch from 4.0.5 to 4.0.8 [inlong-website]
dockerzhang merged PR #973: URL: https://github.com/apache/inlong-website/pull/973 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong-website) branch dependabot/npm_and_yarn/multi-cf87d80143 deleted (was 500faa9ca8)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/multi-cf87d80143 in repository https://gitbox.apache.org/repos/asf/inlong-website.git was 500faa9ca8 Bump send and express The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
Re: [PR] [INLONG-731][BOT] Bump serve-static and express [inlong-website]
dependabot[bot] closed pull request #977: [INLONG-731][BOT] Bump serve-static and express URL: https://github.com/apache/inlong-website/pull/977 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong-website) branch master updated (ba8e771fba -> b65299a59b)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong-website.git from ba8e771fba Bump body-parser and express (#979) add b65299a59b [INLONG-731][BOT] Bump micromatch from 4.0.5 to 4.0.8 (#973) No new revisions were added by this update. Summary of changes: package-lock.json | 16 1 file changed, 8 insertions(+), 8 deletions(-)
(inlong-website) branch dependabot/npm_and_yarn/multi-23dea901ca updated (bbb34601ed -> b80ab180ff)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/multi-23dea901ca in repository https://gitbox.apache.org/repos/asf/inlong-website.git discard bbb34601ed Bump axios, @docusaurus/core, @docusaurus/plugin-content-docs and @docusaurus/preset-classic add aec0e5cb5e [INLONG-980][Agent] Add Agent data format doc (#981) add ba8e771fba Bump body-parser and express (#979) add b65299a59b [INLONG-731][BOT] Bump micromatch from 4.0.5 to 4.0.8 (#973) add b80ab180ff Bump axios, @docusaurus/core, @docusaurus/plugin-content-docs and @docusaurus/preset-classic This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (bbb34601ed) \ N -- N -- N refs/heads/dependabot/npm_and_yarn/multi-23dea901ca (b80ab180ff) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../current/development/binary_protocol/agent.md | 154 .../development/binary_protocol/img/agent_1.png| Bin 0 -> 68576 bytes .../development/binary_protocol/img/cache_1.png| Bin 0 -> 16452 bytes .../development/binary_protocol/img/sink_1.png | Bin 0 -> 13001 bytes .../development/binary_protocol/img/source_1.png | Bin 0 -> 20274 bytes .../development/binary_protocol/img/total.png | Bin 0 -> 25575 bytes package-lock.json | 876 ++--- 7 files changed, 730 insertions(+), 300 deletions(-) create mode 100644 i18n/zh-CN/docusaurus-plugin-content-docs/current/development/binary_protocol/agent.md create mode 100644 i18n/zh-CN/docusaurus-plugin-content-docs/current/development/binary_protocol/img/agent_1.png create mode 100644 i18n/zh-CN/docusaurus-plugin-content-docs/current/development/binary_protocol/img/cache_1.png create mode 100644 i18n/zh-CN/docusaurus-plugin-content-docs/current/development/binary_protocol/img/sink_1.png create mode 100644 i18n/zh-CN/docusaurus-plugin-content-docs/current/development/binary_protocol/img/source_1.png create mode 100644 i18n/zh-CN/docusaurus-plugin-content-docs/current/development/binary_protocol/img/total.png
Re: [PR] [INLONG-982][Doc] Add DataProxy SDK document of Python [inlong-website]
luchunliang commented on code in PR #983: URL: https://github.com/apache/inlong-website/pull/983#discussion_r1773191791 ## i18n/zh-CN/docusaurus-plugin-content-docs/current/sdk/dataproxy-sdk/python.md: ## @@ -0,0 +1,138 @@ +--- +title: Python SDK +sidebar_position: 5 +--- + +## 新建数据接入 +在 Dashboard 或者通过命令行工具创建数据接入，数据源类型使用 `Auto Push` (自主推送)。 + +## 引入 Python SDK +Python SDK 通过封装 C++ SDK 构建得到，对外暴露相关接口。运行时调用底层 C++ SDK 实现相关操作(如发送、关闭等)。 +因为底层是基于 C++ SDK 运行的，与系统 C++ 环境高度绑定，如果提供的软件包使用的 C++ 环境与用户系统环境不完全一致，可能会导致运行报错。 +首先需要从源码自行编译，见 [C++ SDK 编译使用](https://github.com/apache/inlong/tree/master/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp)。 +若成功编译，会在 dataproxy-sdk-cpp/release/lib 目录下发现静态库文件 dataproxy_sdk.a 。 +进入 Python SDK 目录[dataproxy-sdk-python](https://github.com/apache/inlong/tree/master/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-python)，依次执行以下命令: +```shell +chmod +x ./build.sh +./build.sh +``` +当.so文件生成时，您会看到以下消息，您可以选择进入.so文件的目标目录。默认情况下，.so文件将被复制到系统python site-packages目录。 +```shell +Your system's Python site-packages directory is: xxx/xxx +Enter the target directory for the .so files (Press Enter to use the default site-packages directory): +``` +构建过程完成后，您可以在 Python 项目中导入该软件包，使用InLong DataProxy SDK 上报数据。 +```shell +import inlong_dataproxy +``` +> **Note**: 当 C++ SDK 或您正在使用的 Python 版本更新时，您需要按照上述步骤重新构建它。 + +## 数据上报流程 +引入 SDK 后，可以通过调用 SDK 的`send`相关接口进行单条（批量）数据的上报，发送 demo 可参考 [send_demo.py](https://git.woa.com/InLong/SyncWithGitHub/tree/internal-dev/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-python/demo) 。整体流程包括以下三个步骤： + +### 创建 SDK实例 + +SDK 支持进程创建1个SDK实例，多线程安全，也支持进程创建多个SDK实例，各个SDK实例相互独立，各个SDK实例也线程安全： Review Comment: Fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-982][Doc] Add DataProxy SDK document of Python [inlong-website]
luchunliang commented on code in PR #983: URL: https://github.com/apache/inlong-website/pull/983#discussion_r1773191299 ## i18n/zh-CN/docusaurus-plugin-content-docs/current/sdk/dataproxy-sdk/python.md: ## @@ -0,0 +1,138 @@ +--- +title: Python SDK +sidebar_position: 5 +--- + +## 新建数据接入 +在 Dashboard 或者通过命令行工具创建数据接入，数据源类型使用 `Auto Push` (自主推送)。 + +## 引入 Python SDK +Python SDK 通过封装 C++ SDK 构建得到，对外暴露相关接口。运行时调用底层 C++ SDK 实现相关操作(如发送、关闭等)。 +因为底层是基于 C++ SDK 运行的，与系统 C++ 环境高度绑定，如果提供的软件包使用的 C++ 环境与用户系统环境不完全一致，可能会导致运行报错。 +首先需要从源码自行编译，见 [C++ SDK 编译使用](https://github.com/apache/inlong/tree/master/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp)。 +若成功编译，会在 dataproxy-sdk-cpp/release/lib 目录下发现静态库文件 dataproxy_sdk.a 。 +进入 Python SDK 目录[dataproxy-sdk-python](https://github.com/apache/inlong/tree/master/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-python)，依次执行以下命令: Review Comment: Fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-985][Agent] Add English Agent binary protocol doc [inlong-website]
justinwwhuang opened a new pull request, #986: URL: https://github.com/apache/inlong-website/pull/986 Fixes #985 ### Motivation In order to deepen everyone's understanding of the binary protocol of agent ### Modifications Add English Agent binary protocol doc ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation No doc needed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-11195][Manager] It is not allowed to modify group information when ordinary users are not responsible [inlong]
fuweng11 opened a new pull request, #11196: URL: https://github.com/apache/inlong/pull/11196 Fixes #11195 ### Motivation It is not allowed to modify group information when ordinary users are not responsible. ### Modifications Add verification of operator permissions when modifying group information. ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-980][Agent] Add Agent data format doc [inlong-website]
aloyszhang commented on code in PR #981:
URL: https://github.com/apache/inlong-website/pull/981#discussion_r1772906743
##
i18n/zh-CN/docusaurus-plugin-content-docs/current/development/Internal_protocol_format/agent.md:
##
@@ -0,0 +1,154 @@
+---
+title: Agent 数据组织
+sidebar_position: 2
+---
+## 概述
+本文主要介绍 Agent 数据在子模块间流转以及对应的数据结构。
+## 数据流转
+
+
+在 Agent 模块介绍时我们知道 Agent 中有 Source 和 Sink 模块，其中 Source 负责从数据源采集数据；Sink
负责将数据发送到下游，当
+前我们认为下游只有 DataProxy。Instance 则是负责将数据从 Source 搬运到 Sink。
+## 数据格式
+### Source
+
+
+Source 主要有 3 个功能：
+
+1 从数据源采集数据，每条数据填充到一个新的 SourceData 对象。
+
+2 将填充完成的 SourceData 对象放入 Source 模块的缓存队列中。
+
+3 外部调用 Source 模块 Read 方法时从缓存队列中取出一个 SourceData 组装成 Message 并返回。
+ SourceData
+```
+public class SourceData {
+private byte[] data;
+private String offset;
+}
+```
+从数据源采集的数据可能会有各种各样的格式，因此我们用 byte[] 来存放（原始数据在后续各个缓解流转时均保持
byte[]）。同时每一条数据都会有对应的位点信息，
Review Comment:
```suggestion
从数据源采集的数据可能会有各种各样的格式，因此我们用 byte[] 来存放（原始数据在后续各个环节流转时均保持
byte[]）。同时每一条数据都会有对应的位点信息，
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(inlong-website) branch master updated (67ace16b44 -> aec0e5cb5e)
This is an automated email from the ASF dual-hosted git repository. wenweihuang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong-website.git from 67ace16b44 [INLONG-731][BOT] Bump webpack from 5.76.2 to 5.94.0 (#972) add aec0e5cb5e [INLONG-980][Agent] Add Agent data format doc (#981) No new revisions were added by this update. Summary of changes: .../current/development/binary_protocol/agent.md | 154 + .../development/binary_protocol/img/agent_1.png| Bin 0 -> 68576 bytes .../development/binary_protocol/img/cache_1.png| Bin 0 -> 16452 bytes .../development/binary_protocol/img/sink_1.png | Bin 0 -> 13001 bytes .../development/binary_protocol/img/source_1.png | Bin 0 -> 20274 bytes .../development/binary_protocol/img/total.png | Bin 0 -> 25575 bytes 6 files changed, 154 insertions(+) create mode 100644 i18n/zh-CN/docusaurus-plugin-content-docs/current/development/binary_protocol/agent.md create mode 100644 i18n/zh-CN/docusaurus-plugin-content-docs/current/development/binary_protocol/img/agent_1.png create mode 100644 i18n/zh-CN/docusaurus-plugin-content-docs/current/development/binary_protocol/img/cache_1.png create mode 100644 i18n/zh-CN/docusaurus-plugin-content-docs/current/development/binary_protocol/img/sink_1.png create mode 100644 i18n/zh-CN/docusaurus-plugin-content-docs/current/development/binary_protocol/img/source_1.png create mode 100644 i18n/zh-CN/docusaurus-plugin-content-docs/current/development/binary_protocol/img/total.png
(inlong-website) branch dependabot/npm_and_yarn/multi-23dea901ca updated (b80ab180ff -> 064767894b)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/multi-23dea901ca in repository https://gitbox.apache.org/repos/asf/inlong-website.git omit b80ab180ff Bump axios, @docusaurus/core, @docusaurus/plugin-content-docs and @docusaurus/preset-classic add bd5c3d9eda [INLONG-731][BOT] Bump send and express (#978) add 064767894b Bump axios, @docusaurus/core, @docusaurus/plugin-content-docs and @docusaurus/preset-classic This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (b80ab180ff) \ N -- N -- N refs/heads/dependabot/npm_and_yarn/multi-23dea901ca (064767894b) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes:
[I] [Improve][Agent] Add Agent binary protocol doc [inlong-website]
justinwwhuang opened a new issue, #985: URL: https://github.com/apache/inlong-website/issues/985 ### Description Add Agent binary protocol doc ### Are you willing to submit PR? - [X] Yes, I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong-website) branch INLONG-984 created (now e9ee826538)
This is an automated email from the ASF dual-hosted git repository. wakefu pushed a change to branch INLONG-984 in repository https://gitbox.apache.org/repos/asf/inlong-website.git at e9ee826538 [INLONG-984][Doc] Add data node management document This branch includes the following new commits: new e9ee826538 [INLONG-984][Doc] Add data node management document The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(inlong-website) 01/01: [INLONG-984][Doc] Add data node management document
This is an automated email from the ASF dual-hosted git repository. wakefu pushed a commit to branch INLONG-984 in repository https://gitbox.apache.org/repos/asf/inlong-website.git commit e9ee8265386d7c324d93278abaaeac5391ee88a1 Author: fuwenkai <834260...@qq.com> AuthorDate: Tue Sep 24 20:28:43 2024 +0800 [INLONG-984][Doc] Add data node management document --- .../current/administration/img/create_node.png | Bin 0 -> 368792 bytes .../current/administration/img/delete_node.png | Bin 0 -> 294356 bytes .../current/administration/img/test_connection.png | Bin 0 -> 358822 bytes .../current/administration/img/update_node.png | Bin 0 -> 297353 bytes .../current/administration/img/use_node_1.png | Bin 0 -> 358238 bytes .../current/administration/img/use_node_2.png | Bin 0 -> 498801 bytes .../current/administration/node_management.md | 47 + 7 files changed, 47 insertions(+) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/img/create_node.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/img/create_node.png new file mode 100644 index 00..d888fbe3d3 Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/img/create_node.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/img/delete_node.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/img/delete_node.png new file mode 100644 index 00..ac223046bc Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/img/delete_node.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/img/test_connection.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/img/test_connection.png new file mode 100644 index 00..b9bb9f538d Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/img/test_connection.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/img/update_node.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/img/update_node.png new file mode 100644 index 00..7432974d87 Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/img/update_node.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/img/use_node_1.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/img/use_node_1.png new file mode 100644 index 00..2d96101293 Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/img/use_node_1.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/img/use_node_2.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/img/use_node_2.png new file mode 100644 index 00..24e2cf631b Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/img/use_node_2.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/node_management.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/node_management.md new file mode 100644 index 00..3b30d516e7 --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/node_management.md @@ -0,0 +1,47 @@ +--- +title: 节点管理 +sidebar_position: 4 +--- + +节点管理主要提供数据节点信息复用的能力，用户可以在新建、修改、删除数据节点，并且在进行数据接入时，复用配置好的数据节点信息： + +### 新建数据节点 + +用户可以通过新建数据节点来生成一个常用的节点信息： + + + +- 节点名：用户自定义名，用来标识此节点信息 +- 类型：节点类型 +- 责任人：该节点负责人，只有责任人能够修改该节点的配置信息 +- 描述：节点描述信息 + +以上为节点的公用配置，另外，根据不同的节点类型，还会有不同的配置信息，如 ClickHouse 节点需要填写用户名、密码、ClickHouse 地址信息。 + +### 删除数据节点 + +用户可以对已创建节点进行删除，删除后此节点将停止使用： + + + +### 修改数据节点 + +用户可以修改节点信息，点击【编辑】，输入新的节点信息，确认后此节点配置将生效： + + + +### 测试连接 + +用户通过测试连接校验节点状态是否能够正常： + + + +### 使用节点 + +用户配置好节点后，可以在数据订阅中使用已经配置好的节点信息： + +- 选择`数据接入`，点击`详情` +  +- 选择数据目标`新建`，选择已经配置好的数据节点，并完善其他配置后点击保存。 +  + 至此完成了节点的使用 \ No newline at end of file
[PR] [INLONG-984][Doc] Add data node management document [inlong-website]
fuweng11 opened a new pull request, #987: URL: https://github.com/apache/inlong-website/pull/987 Fixes #984 ### Motivation Add data node management document. ### Modifications Add data node management document. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-984][Doc] Add data node management document [inlong-website]
aloyszhang commented on code in PR #987: URL: https://github.com/apache/inlong-website/pull/987#discussion_r1773266059 ## i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/node_management.md: ## @@ -0,0 +1,47 @@ +--- +title: 节点管理 +sidebar_position: 4 +--- + +节点管理主要提供数据节点信息复用的能力，用户可以在新建、修改、删除数据节点，并且在进行数据接入时，复用配置好的数据节点信息： + +### 新建数据节点 + +用户可以通过新建数据节点来生成一个常用的节点信息： + + + +- 节点名：用户自定义名，用来标识此节点信息 +- 类型：节点类型 +- 责任人：该节点负责人，只有责任人能够修改该节点的配置信息 +- 描述：节点描述信息 + +以上为节点的公用配置，另外，根据不同的节点类型，还会有不同的配置信息，如 ClickHouse 节点需要填写用户名、密码、ClickHouse 地址信息。 + +### 删除数据节点 + +用户可以对已创建节点进行删除，删除后此节点将停止使用： + + + +### 修改数据节点 + +用户可以修改节点信息，点击【编辑】，输入新的节点信息，确认后此节点配置将生效： + + + +### 测试连接 + +用户通过测试连接校验节点状态是否能够正常： + + + +### 使用节点 + +用户配置好节点后，可以在数据订阅中使用已经配置好的节点信息： + +- 选择`数据接入`，点击`详情` +  +- 选择数据目标`新建`，选择已经配置好的数据节点，并完善其他配置后点击保存。 Review Comment: ```suggestion - 选择数据目标【新建】，选择已经配置好的数据节点，并完善其他配置后点击保存。 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-984][Doc] Add data node management document [inlong-website]
aloyszhang commented on code in PR #987: URL: https://github.com/apache/inlong-website/pull/987#discussion_r1773264498 ## i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/node_management.md: ## @@ -0,0 +1,47 @@ +--- +title: 节点管理 +sidebar_position: 4 +--- + +节点管理主要提供数据节点信息复用的能力，用户可以在新建、修改、删除数据节点，并且在进行数据接入时，复用配置好的数据节点信息： + +### 新建数据节点 + +用户可以通过新建数据节点来生成一个常用的节点信息： + + + +- 节点名：用户自定义名，用来标识此节点信息 +- 类型：节点类型 +- 责任人：该节点负责人，只有责任人能够修改该节点的配置信息 +- 描述：节点描述信息 + +以上为节点的公用配置，另外，根据不同的节点类型，还会有不同的配置信息，如 ClickHouse 节点需要填写用户名、密码、ClickHouse 地址信息。 + +### 删除数据节点 + +用户可以对已创建节点进行删除，删除后此节点将停止使用： + + + +### 修改数据节点 + +用户可以修改节点信息，点击【编辑】，输入新的节点信息，确认后此节点配置将生效： + + + +### 测试连接 + +用户通过测试连接校验节点状态是否能够正常： Review Comment: ```suggestion 用户通过测试连接校验节点状态是否正常： ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-984][Doc] Add data node management document [inlong-website]
aloyszhang commented on code in PR #987: URL: https://github.com/apache/inlong-website/pull/987#discussion_r1773265695 ## i18n/zh-CN/docusaurus-plugin-content-docs/current/administration/node_management.md: ## @@ -0,0 +1,47 @@ +--- +title: 节点管理 +sidebar_position: 4 +--- + +节点管理主要提供数据节点信息复用的能力，用户可以在新建、修改、删除数据节点，并且在进行数据接入时，复用配置好的数据节点信息： + +### 新建数据节点 + +用户可以通过新建数据节点来生成一个常用的节点信息： + + + +- 节点名：用户自定义名，用来标识此节点信息 +- 类型：节点类型 +- 责任人：该节点负责人，只有责任人能够修改该节点的配置信息 +- 描述：节点描述信息 + +以上为节点的公用配置，另外，根据不同的节点类型，还会有不同的配置信息，如 ClickHouse 节点需要填写用户名、密码、ClickHouse 地址信息。 + +### 删除数据节点 + +用户可以对已创建节点进行删除，删除后此节点将停止使用： + + + +### 修改数据节点 + +用户可以修改节点信息，点击【编辑】，输入新的节点信息，确认后此节点配置将生效： + + + +### 测试连接 + +用户通过测试连接校验节点状态是否能够正常： + + + +### 使用节点 + +用户配置好节点后，可以在数据订阅中使用已经配置好的节点信息： + +- 选择`数据接入`，点击`详情` Review Comment: ```suggestion - 选择【数据接入】，点击【详情】 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-980][Agent] Add Agent data format doc [inlong-website]
doleyzi commented on code in PR #981: URL: https://github.com/apache/inlong-website/pull/981#discussion_r1772804838 ## i18n/zh-CN/docusaurus-plugin-content-docs/current/development/Internal_protocol_format/agent.md: ## @@ -0,0 +1,154 @@ +--- +title: Agent 数据组织 +sidebar_position: 2 +--- +## 概述 +本文主要介绍 Agent 数据在子模块间流转以及对应的数据结构。 +## 数据流转 + + +在 Agent 模块介绍时我们知道 Agent 中有 Source 和 Sink 模块，其中 Source 负责从数据源采集数据；Sink 负责将数据发送到下游，当 +前我们认为下游只有 DataProxy。Instance 则是负责将数据从 Source 搬运到 Sink。 Review Comment: It is recommended not to wrap here -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11183][Dashboard] Module audit page indicator items are merged with other items [inlong]
dockerzhang merged PR #11185: URL: https://github.com/apache/inlong/pull/11185 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-11183][Dashboard] Module audit page indicator items are merged with other items (#11185)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 48bb6c3e66 [INLONG-11183][Dashboard] Module audit page indicator items
are merged with other items (#11185)
48bb6c3e66 is described below
commit 48bb6c3e66f358726f97f7fd722101d46b5ca2b1
Author: kamianlaida <165994047+wohainilao...@users.noreply.github.com>
AuthorDate: Tue Sep 24 15:48:16 2024 +0800
[INLONG-11183][Dashboard] Module audit page indicator items are merged with
other items (#11185)
---
.../ui/pages/ModuleAudit/AuditModule/config.tsx| 38
.../src/ui/pages/ModuleAudit/AuditModule/index.tsx | 5 +-
.../src/ui/pages/ModuleAudit/IdModule/config.tsx | 68 +++---
.../src/ui/pages/ModuleAudit/IdModule/index.tsx| 6 +-
.../src/ui/pages/ModuleAudit/IpModule/config.tsx | 68 +++---
.../src/ui/pages/ModuleAudit/IpModule/index.tsx| 6 +-
.../src/ui/pages/ModuleAudit/index.tsx | 24 ++--
7 files changed, 81 insertions(+), 134 deletions(-)
diff --git a/inlong-dashboard/src/ui/pages/ModuleAudit/AuditModule/config.tsx
b/inlong-dashboard/src/ui/pages/ModuleAudit/AuditModule/config.tsx
index 2ef04c5bd5..e3821a15ec 100644
--- a/inlong-dashboard/src/ui/pages/ModuleAudit/AuditModule/config.tsx
+++ b/inlong-dashboard/src/ui/pages/ModuleAudit/AuditModule/config.tsx
@@ -158,7 +158,7 @@ export const getSourceDataWithCommas = sourceData => {
let endTimeVisible = true;
-export const getFormContent = (initialValues, onSearch, onDataStreamSuccess)
=> [
+export const getFormContent = (initialValues, onSearch, onDataStreamSuccess,
auditData) => [
{
type: 'select',
label: i18n.t('pages.ModuleAudit.config.InlongGroupId'),
@@ -303,34 +303,16 @@ export const getFormContent = (initialValues, onSearch,
onDataStreamSuccess) =>
allowClear: true,
showSearch: true,
dropdownMatchSelectWidth: false,
- options: {
-requestAuto: true,
-requestTrigger: ['onOpen'],
-requestService: () => {
- return request({
-url: '/audit/getAuditBases',
-params: {
- isMetric: true,
-},
+ options: auditData?.reduce((accumulator, item) => {
+const existingItem = accumulator.find((i: { value: any }) => i.value
=== item.auditId);
+if (!existingItem) {
+ accumulator.push({
+label: i18n?.language === 'cn' ? item.nameInChinese :
item.nameInEnglish,
+value: item.auditId,
});
-},
-requestParams: {
- formatResult: result => {
-return result?.reduce((accumulator, item) => {
- const existingItem = accumulator.find(
-(i: { value: any }) => i.value === item.auditId,
- );
- if (!existingItem) {
-accumulator.push({
- label: i18n?.language === 'cn' ? item.nameInChinese :
item.nameInEnglish,
- value: item.auditId,
-});
- }
- return accumulator;
-}, []);
- },
-},
- },
+}
+return accumulator;
+ }, []),
filterOption: (keyword: string, option: { label: any }) => {
return (option?.label ??
'').toLowerCase().includes(keyword.toLowerCase());
},
diff --git a/inlong-dashboard/src/ui/pages/ModuleAudit/AuditModule/index.tsx
b/inlong-dashboard/src/ui/pages/ModuleAudit/AuditModule/index.tsx
index 5dbed68dd7..8a4491b5e2 100644
--- a/inlong-dashboard/src/ui/pages/ModuleAudit/AuditModule/index.tsx
+++ b/inlong-dashboard/src/ui/pages/ModuleAudit/AuditModule/index.tsx
@@ -32,9 +32,10 @@ import {
} from './config';
import { Table } from 'antd';
import i18n from '@/i18n';
+import { AuditProps } from '@/ui/pages/ModuleAudit';
export const auditModule = 'audit';
-const Comp: React.FC = () => {
+const Comp: React.FC = ({ auditData }) => {
const [form] = useForm();
const [query, setQuery] = useState({
@@ -114,7 +115,7 @@ const Comp: React.FC = () => {
setQuery({
diff --git a/inlong-dashboard/src/ui/pages/ModuleAudit/IdModule/config.tsx
b/inlong-dashboard/src/ui/pages/ModuleAudit/IdModule/config.tsx
index c65e0ecdff..cd868b6797 100644
--- a/inlong-dashboard/src/ui/pages/ModuleAudit/IdModule/config.tsx
+++ b/inlong-dashboard/src/ui/pages/ModuleAudit/IdModule/config.tsx
@@ -47,7 +47,7 @@ export const toTableData = (source, sourceDataMap) => {
}));
};
-export const getFormContent = (initialValues, onSearch) => [
+export const getFormContent = (initialValues, onSearch, auditData) => [
{
type: 'select',
label: i18n.t('pages.ModuleAudit.config.InlongGroupId'),
@@ -138,29 +138,16 @@ export const getFormContent = (initialValues, onSearch)
=> [
allowC
(inlong) branch master updated: [INLONG-11080][SDK] Transform SQL supports DATE_SUB function (#11193)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new ad7e141ad5 [INLONG-11080][SDK] Transform SQL supports DATE_SUB
function (#11193)
ad7e141ad5 is described below
commit ad7e141ad5e0051356c6f7c81f6970e27a84f431
Author: Zkplo <87751516+zk...@users.noreply.github.com>
AuthorDate: Tue Sep 24 18:29:02 2024 +0800
[INLONG-11080][SDK] Transform SQL supports DATE_SUB function (#11193)
Co-authored-by: ZKpLo <14148880+zk...@user.noreply.gitee.com>
---
.../process/function/DateSubFunction.java | 72
.../function/temporal/TestDateSubFunction.java | 76 ++
2 files changed, 148 insertions(+)
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/DateSubFunction.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/DateSubFunction.java
new file mode 100644
index 00..dfa14efee9
--- /dev/null
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/DateSubFunction.java
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+import org.apache.inlong.sdk.transform.process.utils.DateUtil;
+
+import net.sf.jsqlparser.expression.Expression;
+import net.sf.jsqlparser.expression.Function;
+import org.apache.commons.lang3.tuple.Pair;
+
+import java.time.temporal.ChronoField;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * DateAddFunction
+ * Description:
+ * - return NULL if date is NULL.
+ * - return DATE if the date argument is a DATE value and your calculations
involve only YEAR, MONTH, and DAY parts
+ * (that is, no time parts).
+ * - return TIME if the date argument is a TIME value and the calculations
involve only HOURS, MINUTES,
+ * and SECONDS parts (that is, no date parts).
+ * - return DATETIME if the first argument is a DATETIME (or TIMESTAMP) value,
or if the first argument is a DATE
+ * and the unit value uses HOURS, MINUTES, or SECONDS, or if the
first argument is of type TIME and the
+ * unit value uses YEAR, MONTH, or DAY.
+ * - return If the first argument is a dynamic parameter (for example, of a
prepared statement), its resolved type
+ * is DATE if the second argument is an interval that contains some
combination of YEAR, MONTH, or DAY values
+ * only; otherwise, its type is DATETIME.
+ * - return String otherwise (type VARCHAR).
+ */
+@TransformFunction(names = {"date_sub", "datesub"})
+public class DateSubFunction implements ValueParser {
+
+private ValueParser datetimeParser;
+private ValueParser intervalParser;
+
+public DateSubFunction(Function expr) {
+List expressions = expr.getParameters().getExpressions();
+datetimeParser = OperatorTools.buildParser(expressions.get(0));
+intervalParser = OperatorTools.buildParser(expressions.get(1));
+}
+
+@Override
+public Object parse(SourceData sourceData, int rowIndex, Context context) {
+Object intervalPairObj = intervalParser.parse(sourceData, rowIndex,
context);
+Object dateObj = datetimeParser.parse(sourceData, rowIndex, context);
+if (intervalPairObj == null || dateObj == null) {
+return null;
+}
+return DateUtil.dateAdd(OperatorTools.parseString(dateObj),
+(Pair>) intervalPairObj, -1);
+}
+}
diff --git
a/inlong-sdk/transform-sdk/src/test/java/org/apache/inlong/sdk/transform/process/function/temporal/TestDateSubFunction.java
b/inlong-sdk/transform-sdk/src/test/java/org/apache/inlong/sdk/transform/process/function/temporal/TestDateSubFunction.java
new file mode 100644
index 00..8b3389fc9b
Re: [PR] [INLONG-11080][SDK] Transform SQL supports DATE_SUB function [inlong]
dockerzhang merged PR #11193: URL: https://github.com/apache/inlong/pull/11193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Improve][SDK] Add DataProxy SDK document of Python [inlong-website]
luchunliang opened a new issue, #982: URL: https://github.com/apache/inlong-website/issues/982 ### Description Add DataProxy SDK document of Python ### Are you willing to submit PR? - [X] Yes, I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong-website) branch INLONG-984 updated (e9ee826538 -> 56615a931b)
This is an automated email from the ASF dual-hosted git repository. wakefu pushed a change to branch INLONG-984 in repository https://gitbox.apache.org/repos/asf/inlong-website.git from e9ee826538 [INLONG-984][Doc] Add data node management document add 56615a931b [INLONG-984][Doc] Fix comment No new revisions were added by this update. Summary of changes: docs/administration/img/create_node.png| Bin 0 -> 392047 bytes docs/administration/img/delete_node.png| Bin 0 -> 294566 bytes docs/administration/img/test_connection.png| Bin 0 -> 358367 bytes docs/administration/img/update_node.png| Bin 0 -> 290739 bytes docs/administration/img/use_node_1.png | Bin 0 -> 361320 bytes docs/administration/img/use_node_2.png | Bin 0 -> 487762 bytes docs/administration/node_management.md | 51 + .../current/administration/node_management.md | 9 ++-- 8 files changed, 57 insertions(+), 3 deletions(-) create mode 100644 docs/administration/img/create_node.png create mode 100644 docs/administration/img/delete_node.png create mode 100644 docs/administration/img/test_connection.png create mode 100644 docs/administration/img/update_node.png create mode 100644 docs/administration/img/use_node_1.png create mode 100644 docs/administration/img/use_node_2.png create mode 100644 docs/administration/node_management.md
(inlong) branch master updated: [INLONG-11132][SDK] Transform SQL support parsing SIMILAR TO (#11133)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 5c722da844 [INLONG-11132][SDK] Transform SQL support parsing SIMILAR
TO (#11133)
5c722da844 is described below
commit 5c722da8442a9b30106ffa499bf2e066df203929
Author: emptyOVO <118812562+empty...@users.noreply.github.com>
AuthorDate: Wed Sep 25 12:38:41 2024 +0800
[INLONG-11132][SDK] Transform SQL support parsing SIMILAR TO (#11133)
---
.../transform/process/parser/SimilarToParser.java | 107 +
.../process/parser/TestSimilarToParser.java| 165 +
2 files changed, 272 insertions(+)
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/parser/SimilarToParser.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/parser/SimilarToParser.java
new file mode 100644
index 00..d947f73c5f
--- /dev/null
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/parser/SimilarToParser.java
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.parser;
+
+import org.apache.inlong.common.util.StringUtil;
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+
+import lombok.extern.slf4j.Slf4j;
+import net.sf.jsqlparser.expression.operators.relational.SimilarToExpression;
+
+import java.util.regex.Pattern;
+/**
+ * SimilarToParser
+ */
+@Slf4j
+@TransformParser(values = SimilarToExpression.class)
+public class SimilarToParser implements ValueParser {
+
+private final ValueParser destParser;
+private final ValueParser patternParser;
+private final String escapeChar;
+private final boolean isNot;
+private static final String REGEX_SPECIAL_CHAR = "[]()|^-+*?{}$\\.";
+
+public SimilarToParser(SimilarToExpression expr) {
+destParser = OperatorTools.buildParser(expr.getLeftExpression());
+patternParser = OperatorTools.buildParser(expr.getRightExpression());
+escapeChar = StringUtil.isEmpty(expr.getEscape()) ? "\\" :
expr.getEscape();
+isNot = expr.isNot();
+}
+
+@Override
+public Object parse(SourceData sourceData, int rowIndex, Context context) {
+Object destObj = destParser.parse(sourceData, rowIndex, context);
+Object patternObj = patternParser.parse(sourceData, rowIndex, context);
+if (destObj == null || patternObj == null) {
+return null;
+}
+String destStr = destObj.toString();
+String pattern = patternObj.toString();
+try {
+final String regex = buildSimilarToRegex(pattern,
escapeChar.charAt(0));
+boolean isMatch = Pattern.matches(regex.toLowerCase(),
destStr.toLowerCase());
+if (isNot) {
+return !isMatch;
+}
+return isMatch;
+} catch (Exception e) {
+log.error(e.getMessage(), e);
+return null;
+}
+}
+
+private String buildSimilarToRegex(String pattern, char escapeChar) {
+int len = pattern.length();
+StringBuilder regexPattern = new StringBuilder(len + len);
+for (int i = 0; i < len; i++) {
+char c = pattern.charAt(i);
+if (REGEX_SPECIAL_CHAR.indexOf(c) >= 0) {
+regexPattern.append('\\');
+}
+if (c == escapeChar) {
+if (i == (pattern.length() - 1)) {
+regexPattern.append(c);
+continue;
+}
+char nextChar = pattern.charAt(i + 1);
+if (nextChar == '_' || nextChar == '%' || nextChar ==
escapeChar) {
+regexPattern.append(nextChar);
+i++;
+} else {
+throw new RuntimeException("Illegal pattern string");
+}
+} else if (c == '_') {
+
Re: [PR] [INLONG-11132][SDK] Transform SQL support parsing SIMILAR TO [inlong]
dockerzhang merged PR #11133: URL: https://github.com/apache/inlong/pull/11133 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-985][Agent] Add English Agent binary protocol doc [inlong-website]
aloyszhang closed pull request #986: [INLONG-985][Agent] Add English Agent binary protocol doc URL: https://github.com/apache/inlong-website/pull/986 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-985][Agent] Add English Agent binary protocol doc [inlong-website]
aloyszhang commented on PR #986: URL: https://github.com/apache/inlong-website/pull/986#issuecomment-2373137734 please add the raw file of image and add `_category_.json` file for directory `docs/deployment/binary_protocol` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-980][Agent] Add Agent data format doc [inlong-website]
aloyszhang commented on code in PR #981:
URL: https://github.com/apache/inlong-website/pull/981#discussion_r1772910002
##
i18n/zh-CN/docusaurus-plugin-content-docs/current/development/Internal_protocol_format/agent.md:
##
@@ -0,0 +1,154 @@
+---
+title: Agent 数据组织
+sidebar_position: 2
+---
+## 概述
+本文主要介绍 Agent 数据在子模块间流转以及对应的数据结构。
+## 数据流转
+
+
+在 Agent 模块介绍时我们知道 Agent 中有 Source 和 Sink 模块，其中 Source 负责从数据源采集数据；Sink
负责将数据发送到下游，当
+前我们认为下游只有 DataProxy。Instance 则是负责将数据从 Source 搬运到 Sink。
+## 数据格式
+### Source
+
+
+Source 主要有 3 个功能：
+
+1 从数据源采集数据，每条数据填充到一个新的 SourceData 对象。
+
+2 将填充完成的 SourceData 对象放入 Source 模块的缓存队列中。
+
+3 外部调用 Source 模块 Read 方法时从缓存队列中取出一个 SourceData 组装成 Message 并返回。
+ SourceData
+```
+public class SourceData {
+private byte[] data;
+private String offset;
+}
+```
+从数据源采集的数据可能会有各种各样的格式，因此我们用 byte[] 来存放（原始数据在后续各个缓解流转时均保持
byte[]）。同时每一条数据都会有对应的位点信息，
+并且由于位点信息多种多样，我们用 String 来存放位点信息。
+ 缓存队列
+顾名思义，我们将 SourceData 放到 queue 中以达到缓存的目的，这样可以解耦数据源和 Agent 内部处理速度不匹配的问题。queue 类
Review Comment:
```suggestion
顾名思义，我们将 SourceData 放到 queue 中以达到缓存的目的，这样可以解耦数据源和 Agent 内部处理速度不匹配的问题。Queue 类
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[I] [Improve][Agent] Add Agent inner data format doc [inlong-website]
justinwwhuang opened a new issue, #980: URL: https://github.com/apache/inlong-website/issues/980 ### Description Add Agent inner data format doc ### Are you willing to submit PR? - [X] Yes, I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-980][Agent] Add Agent inner data format doc [inlong-website]
justinwwhuang opened a new pull request, #981: URL: https://github.com/apache/inlong-website/pull/981 Fixes #980 ### Motivation In order to deepen everyone's understanding of the internal data flow principle of agent ### Modifications Add Agent inner data format doc ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation No doc needed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-982][Doc] Add DataProxy SDK document of Python [inlong-website]
luchunliang opened a new pull request, #983: URL: https://github.com/apache/inlong-website/pull/983 Fixes #982 ### Motivation ### Modifications ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a followup issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-982][Doc] Add DataProxy SDK document of Python [inlong-website]
fuweng11 commented on code in PR #983: URL: https://github.com/apache/inlong-website/pull/983#discussion_r1773164769 ## i18n/zh-CN/docusaurus-plugin-content-docs/current/sdk/dataproxy-sdk/python.md: ## @@ -0,0 +1,138 @@ +--- +title: Python SDK +sidebar_position: 5 +--- + +## 新建数据接入 +在 Dashboard 或者通过命令行工具创建数据接入，数据源类型使用 `Auto Push` (自主推送)。 + +## 引入 Python SDK +Python SDK 通过封装 C++ SDK 构建得到，对外暴露相关接口。运行时调用底层 C++ SDK 实现相关操作(如发送、关闭等)。 +因为底层是基于 C++ SDK 运行的，与系统 C++ 环境高度绑定，如果提供的软件包使用的 C++ 环境与用户系统环境不完全一致，可能会导致运行报错。 +首先需要从源码自行编译，见 [C++ SDK 编译使用](https://github.com/apache/inlong/tree/master/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp)。 +若成功编译，会在 dataproxy-sdk-cpp/release/lib 目录下发现静态库文件 dataproxy_sdk.a 。 +进入 Python SDK 目录[dataproxy-sdk-python](https://github.com/apache/inlong/tree/master/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-python)，依次执行以下命令: Review Comment: 进入 Python SDK 目录 [dataproxy-sdk-python] -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-982][Doc] Add DataProxy SDK document of Python [inlong-website]
fuweng11 commented on code in PR #983: URL: https://github.com/apache/inlong-website/pull/983#discussion_r1773163852 ## i18n/zh-CN/docusaurus-plugin-content-docs/current/sdk/dataproxy-sdk/python.md: ## @@ -0,0 +1,138 @@ +--- +title: Python SDK +sidebar_position: 5 +--- + +## 新建数据接入 +在 Dashboard 或者通过命令行工具创建数据接入，数据源类型使用 `Auto Push` (自主推送)。 + +## 引入 Python SDK +Python SDK 通过封装 C++ SDK 构建得到，对外暴露相关接口。运行时调用底层 C++ SDK 实现相关操作(如发送、关闭等)。 +因为底层是基于 C++ SDK 运行的，与系统 C++ 环境高度绑定，如果提供的软件包使用的 C++ 环境与用户系统环境不完全一致，可能会导致运行报错。 +首先需要从源码自行编译，见 [C++ SDK 编译使用](https://github.com/apache/inlong/tree/master/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp)。 +若成功编译，会在 dataproxy-sdk-cpp/release/lib 目录下发现静态库文件 dataproxy_sdk.a 。 +进入 Python SDK 目录[dataproxy-sdk-python](https://github.com/apache/inlong/tree/master/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-python)，依次执行以下命令: +```shell +chmod +x ./build.sh +./build.sh +``` +当.so文件生成时，您会看到以下消息，您可以选择进入.so文件的目标目录。默认情况下，.so文件将被复制到系统python site-packages目录。 +```shell +Your system's Python site-packages directory is: xxx/xxx +Enter the target directory for the .so files (Press Enter to use the default site-packages directory): +``` +构建过程完成后，您可以在 Python 项目中导入该软件包，使用InLong DataProxy SDK 上报数据。 +```shell +import inlong_dataproxy +``` +> **Note**: 当 C++ SDK 或您正在使用的 Python 版本更新时，您需要按照上述步骤重新构建它。 + +## 数据上报流程 +引入 SDK 后，可以通过调用 SDK 的`send`相关接口进行单条（批量）数据的上报，发送 demo 可参考 [send_demo.py](https://git.woa.com/InLong/SyncWithGitHub/tree/internal-dev/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-python/demo) 。整体流程包括以下三个步骤： + +### 创建 SDK实例 + +SDK 支持进程创建1个SDK实例，多线程安全，也支持进程创建多个SDK实例，各个SDK实例相互独立，各个SDK实例也线程安全： Review Comment: SDK 支持进程创建 1 个SDK实例 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Improve][Doc] Add data node management document [inlong-website]
fuweng11 opened a new issue, #984: URL: https://github.com/apache/inlong-website/issues/984 ### Description Add data node management document. ### Are you willing to submit PR? - [X] Yes, I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Improve][Agent] Add Agent inner data format doc [inlong-website]
justinwwhuang closed issue #980: [Improve][Agent] Add Agent inner data format doc URL: https://github.com/apache/inlong-website/issues/980 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-980][Agent] Add Agent data format doc [inlong-website]
justinwwhuang merged PR #981: URL: https://github.com/apache/inlong-website/pull/981 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-11154][SDK] Transform SQL supports TIMESTAMP function (#11171)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new b82f40f746 [INLONG-11154][SDK] Transform SQL supports TIMESTAMP
function (#11171)
b82f40f746 is described below
commit b82f40f74655d4bf143f726ac29148b2db037e7d
Author: Zkplo <87751516+zk...@users.noreply.github.com>
AuthorDate: Wed Sep 25 12:31:12 2024 +0800
[INLONG-11154][SDK] Transform SQL supports TIMESTAMP function (#11171)
Co-authored-by: ZKpLo <14148880+zk...@user.noreply.gitee.com>
---
.../process/function/TimestampFunction.java| 89 ++
.../sdk/transform/process/utils/DateUtil.java | 79 ++-
.../function/temporal/TestTimestampFunction.java | 86 +
3 files changed, 233 insertions(+), 21 deletions(-)
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/TimestampFunction.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/TimestampFunction.java
new file mode 100644
index 00..698942badf
--- /dev/null
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/TimestampFunction.java
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+import org.apache.inlong.sdk.transform.process.utils.DateUtil;
+
+import net.sf.jsqlparser.expression.Expression;
+import net.sf.jsqlparser.expression.Function;
+
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.util.List;
+
+/**
+ * TimestampFunction -> timestamp(expr1[, expr2])
+ * description:
+ * - return NULL if expr1 or expr2 is NULL.
+ * - return the date or datetime expression expr as a datetime value if there
is only one parameter
+ * - return the result of the date or date time expression expr1 plus the time
expression expr2 if there are two parameters
+ */
+@TransformFunction(names = {"timestamp"})
+public class TimestampFunction implements ValueParser {
+
+private ValueParser dateTimeExprParser;
+private ValueParser timeExprParser;
+
+public TimestampFunction(Function expr) {
+List expressions = expr.getParameters().getExpressions();
+dateTimeExprParser = OperatorTools.buildParser(expressions.get(0));
+if (expressions.size() == 2) {
+timeExprParser = OperatorTools.buildParser(expressions.get(1));
+}
+}
+
+@Override
+public Object parse(SourceData sourceData, int rowIndex, Context context) {
+Object dateTimeExprObj = dateTimeExprParser.parse(sourceData,
rowIndex, context);
+if (dateTimeExprObj == null) {
+return null;
+}
+String dateTimeStr = dateTimeExprObj.toString();
+LocalDateTime localDateTime =
DateUtil.parseLocalDateTime(dateTimeExprObj.toString());
+if (localDateTime == null) {
+// Not meeting the format requirements
+return null;
+}
+boolean hasMicroSecond = dateTimeStr.indexOf('.') != -1;
+String formatStr = DateUtil.YEAR_TO_SECOND;
+// Support the second parameter
+if (timeExprParser != null) {
+Object timeExprObj = timeExprParser.parse(sourceData, rowIndex,
context);
+if (timeExprObj != null) {
+String timeStr = timeExprObj.toString();
+LocalTime localTime = DateUtil.parseLocalTime(timeStr);
+if (localTime == null) {
+// Not meeting the format requirements
+return null;
+}
+hasMicroSecond |= timeStr.indexOf('.') != -1;
+localDateTime = DateUtil.dateAdd(localDateTime, localTime);
+} else {
+return nul
Re: [PR] [INLONG-11154][SDK] Transform SQL supports TIMESTAMP function [inlong]
dockerzhang merged PR #11171: URL: https://github.com/apache/inlong/pull/11171 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-11197][Dashboard] Add delete button to cluster management and template management [inlong]
wohainilaodou opened a new pull request, #11198: URL: https://github.com/apache/inlong/pull/11198 Fixes #11197 ### Motivation Add delete button to cluster management and template management ### Modifications Add delete button to cluster management and template management ### Verifying this change 1. template: before:  after:  2. tenant: before:  after:  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Improve][Doc] Adjust the HTTP reporting document of DataProxy [inlong-website]
gosonzhang opened a new issue, #989: URL: https://github.com/apache/inlong-website/issues/989 ### Description Adjust the HTTP reporting document of DataProxy 1. Adjust the document placement: The existing document is placed in "docs/next/sdk/dataproxy-sdk/", and this part of the introduction is placed in "docs/next/quick_start/data_ingestion/" to make it easier for novices to pay attention and get started quickly; 2. Adjust the document image introduction: The image is too complicated and difficult for users to understand ### Are you willing to submit PR? - [X] Yes, I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11129][Sort] Enhanced source metric instrumentation for InLong Sort Flink Connector [inlong]
aloyszhang merged PR #11130: URL: https://github.com/apache/inlong/pull/11130 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated (5c722da844 -> 0122aa4384)
This is an automated email from the ASF dual-hosted git repository. aloyszhang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong.git from 5c722da844 [INLONG-11132][SDK] Transform SQL support parsing SIMILAR TO (#11133) add 0122aa4384 [INLONG-11129][Sort] Enhanced source metric instrumentation for InLong Sort Flink Connector (#11130) No new revisions were added by this update. Summary of changes: .../org/apache/inlong/sort/base/Constants.java | 14 ++ .../sort/base/metric/SourceExactlyMetric.java | 143 + .../sort/postgre/DebeziumSourceFunction.java | 57 +++- .../inlong/sort/postgre/PostgreSQLSource.java | 11 +- .../inlong/sort/postgre/PostgreSQLTableSource.java | 2 +- .../postgre/RowDataDebeziumDeserializeSchema.java | 41 +++--- 6 files changed, 250 insertions(+), 18 deletions(-)
Re: [PR] [INLONG-11195][Manager] It is not allowed to modify group information when ordinary users are not responsible [inlong]
dockerzhang merged PR #11196: URL: https://github.com/apache/inlong/pull/11196 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-11195][Manager] It is not allowed to modify group information when ordinary users are not responsible (#11196)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new b670373562 [INLONG-11195][Manager] It is not allowed to modify group
information when ordinary users are not responsible (#11196)
b670373562 is described below
commit b670373562734e76e3c5cf7106a10a1c4f5b53d5
Author: fuweng11 <76141879+fuwen...@users.noreply.github.com>
AuthorDate: Wed Sep 25 14:13:46 2024 +0800
[INLONG-11195][Manager] It is not allowed to modify group information when
ordinary users are not responsible (#11196)
---
.../service/group/InlongGroupServiceImpl.java | 5 +
.../service/sink/StreamSinkServiceImpl.java| 15 ---
.../service/source/StreamSourceServiceImpl.java| 8 +++-
.../service/stream/InlongStreamServiceImpl.java| 22 ++
.../inlong/manager/service/user/UserService.java | 9 +
.../manager/service/user/UserServiceImpl.java | 13 +
6 files changed, 68 insertions(+), 4 deletions(-)
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/group/InlongGroupServiceImpl.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/group/InlongGroupServiceImpl.java
index 3ee94a23ce..66abc58937 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/group/InlongGroupServiceImpl.java
+++
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/group/InlongGroupServiceImpl.java
@@ -85,6 +85,7 @@ import
org.apache.inlong.manager.service.source.bounded.BoundedSourceType;
import org.apache.inlong.manager.service.stream.InlongStreamService;
import org.apache.inlong.manager.service.tenant.InlongTenantService;
import org.apache.inlong.manager.service.user.InlongRoleService;
+import org.apache.inlong.manager.service.user.UserService;
import org.apache.inlong.manager.service.workflow.WorkflowService;
import com.fasterxml.jackson.core.type.TypeReference;
@@ -166,6 +167,8 @@ public class InlongGroupServiceImpl implements
InlongGroupService {
private InlongRoleService inlongRoleService;
@Autowired
private TenantUserRoleEntityMapper tenantUserRoleEntityMapper;
+@Autowired
+private UserService userService;
@Autowired
ScheduleOperator scheduleOperator;
@@ -501,6 +504,8 @@ public class InlongGroupServiceImpl implements
InlongGroupService {
LOGGER.error("inlong group not found by groupId={}", groupId);
throw new BusinessException(ErrorCodeEnum.GROUP_NOT_FOUND);
}
+userService.checkUser(entity.getInCharges(), operator,
+"Current user does not have permission to update group info");
chkUnmodifiableParams(entity, request);
// check whether the current status can be modified
doUpdateCheck(entity, request, operator);
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/sink/StreamSinkServiceImpl.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/sink/StreamSinkServiceImpl.java
index 4ce1d1c76c..dab2cdc93a 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/sink/StreamSinkServiceImpl.java
+++
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/sink/StreamSinkServiceImpl.java
@@ -63,6 +63,7 @@ import org.apache.inlong.manager.pojo.stream.InlongStreamInfo;
import org.apache.inlong.manager.pojo.user.UserInfo;
import org.apache.inlong.manager.service.group.GroupCheckService;
import org.apache.inlong.manager.service.stream.InlongStreamProcessService;
+import org.apache.inlong.manager.service.user.UserService;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
@@ -138,6 +139,8 @@ public class StreamSinkServiceImpl implements
StreamSinkService {
private AutowireCapableBeanFactory autowireCapableBeanFactory;
@Autowired
private ObjectMapper objectMapper;
+@Autowired
+private UserService userService;
// To avoid circular dependencies, you cannot use @Autowired, it will be
injected by AutowireCapableBeanFactory
private InlongStreamProcessService streamProcessOperation;
@@ -449,7 +452,9 @@ public class StreamSinkServiceImpl implements
StreamSinkService {
throw new BusinessException(ErrorCodeEnum.SINK_INFO_NOT_FOUND);
}
chkUnmodifiableParams(curEntity, request);
-groupCheckService.checkGroupStatus(request.getInlongGroupId(),
operator);
+InlongGroupEntity groupEntity =
groupCheckService.checkGroupStatus(request.getInlongGroupId(), operator);
+userService.checkUser(groupEntity.getInCharges(), operator,
+"Cu
