[GitHub] [inlong-website] xuehuanran opened a new issue, #659: [Feature][Doc] Add docs for TubeMQ c++/python sdk.
xuehuanran opened a new issue, #659: URL: https://github.com/apache/inlong-website/issues/659 ### Description _No response_ ### Use case _No response_ ### Are you willing to submit PR? - [X] Yes, I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] xuehuanran opened a new pull request, #660: [INLONG-659][Doc] Add docs for c++/python sdk.
xuehuanran opened a new pull request, #660: URL: https://github.com/apache/inlong-website/pull/660 This PR for #659, add documents for c++/python sdk in en/zh-cn two languages. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] bluewang opened a new pull request, #7025: [INLONG-7024][Dashboard] Type readable lowercase and title optimizati…
bluewang opened a new pull request, #7025: URL: https://github.com/apache/inlong/pull/7025 …on in data integration ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes https://github.com/apache/inlong/issues/7024 ### Modifications     -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang commented on a diff in pull request #7025: [INLONG-7024][Dashboard] Type readable lowercase and title optimizati…
dockerzhang commented on code in PR #7025: URL: https://github.com/apache/inlong/pull/7025#discussion_r1055338925 ## inlong-dashboard/src/locales/en.json: ## @@ -497,8 +497,8 @@ "pages.GroupDetail.Stream.CreateDataStream": "Create", "pages.GroupDetail.Info": "Group Info", "pages.GroupDetail.Streams": "Streams", - "pages.GroupDetail.Sources": "Sources", - "pages.GroupDetail.Sinks": "Sinks", + "pages.GroupDetail.Sources": "Data Source", + "pages.GroupDetail.Sinks": "Data Sink", Review Comment: ```suggestion "pages.GroupDetail.Sinks": "Sinks", ``` ## inlong-dashboard/src/locales/en.json: ## @@ -497,8 +497,8 @@ "pages.GroupDetail.Stream.CreateDataStream": "Create", "pages.GroupDetail.Info": "Group Info", "pages.GroupDetail.Streams": "Streams", - "pages.GroupDetail.Sources": "Sources", - "pages.GroupDetail.Sinks": "Sinks", + "pages.GroupDetail.Sources": "Data Source", Review Comment: ```suggestion "pages.GroupDetail.Sources": "Sources", ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] dockerzhang commented on a diff in pull request #660: [INLONG-659][Doc] Add docs for c++/python sdk.
dockerzhang commented on code in PR #660:
URL: https://github.com/apache/inlong-website/pull/660#discussion_r1055359048
##
i18n/zh-CN/docusaurus-plugin-content-docs/current/sdk/tubemq-sdk/python.md:
##
@@ -0,0 +1,107 @@
+---
+title: Python SDK
+sidebar_position: 2
+---
+
+import {siteVariables} from '../../version';
Review Comment:
This line is not needed for this doc.
##
i18n/zh-CN/docusaurus-plugin-content-docs/current/sdk/tubemq-sdk/python.md:
##
@@ -0,0 +1,107 @@
+---
+title: Python SDK
+sidebar_position: 2
+---
+
+import {siteVariables} from '../../version';
+
+## 构建 TubeMQ Python SDK
+
+Python SDK 主要基于[pybind11](https://pybind11.readthedocs.io/en/stable/)，对C++
SDK进行封装，因此在构建Python SDK之前，需要先构建好 C++ SDK，C++ SDK
Review Comment:
Add a space before and after the English words, eg:
对C++ SDK进行
->
对 C++ SDK 进行
##
i18n/zh-CN/docusaurus-plugin-content-docs/current/sdk/tubemq-sdk/cpp.md:
##
@@ -0,0 +1,156 @@
+---
+title: C++ SDK
+sidebar_position: 1
+---
+
+import {siteVariables} from '../../version';
+
+## 构建 TubeMQ C++ SDK
+
+C++ SDK 主要基于非Boost版本的Asio进行通信，构建时使用CMake，具体操作步骤如下：
+```bash
+# 进入到 cpp sdk 的根路径
+cd `pwd`/inlong/inlong-tubemq/tubemq-client-twins/tubemq-client-cpp
+
+mkdir build && cd build
+
+cmake ..
+
+make -j8 # 可根据自己机器的核数决定编译时使用的线程数量
+
+```
+
+上述构建过程也可以在InLong提供的[镜像](https://github.com/apache/inlong/tree/master/inlong-tubemq/tubemq-docker/tubemq-cpp)中完成，具体操作如下
+
+```bash
+# 拉取镜像
+docker pull inlong/tubemq-cpp
+
+# 启动容器，将cpp sdk的源码挂载到容器中
+docker run -it --net=host -v ${REPLACE_BY_CPP_SOURCE_DIR_PATH}:/tubemq-cpp/
inlong/tubemq-cpp /bin/bash
+
+# 之后与在物理机上相同
+mkdir build && cd build
+cmake ..
+make -j8
+```
+
+
+
+## C++ SDK API
+
+与其它的消息队列系统类似，C++ SDK
分为生产者(Producer)以及消费者(Consumer)，下面分别介绍Producer和Consumer的API使用。
+
+首先，无论是生产者还是消费者，先启动TubeMQ SDK的后台全局服务 (以下例子都假定使用的默认 C++ `namespace`为`tubemq`)。
+
+```cpp
+bool StartTubeMQService(string& err_info, const TubeMQServiceConfig&
serviceConfig);
+```
+
+通过返回值以及`err_info`判断启动是否成功。
+
+### Producer
+
+对于用户来说，直接接触的类为`TubeMQProducer`，在使用时需要首先创建实例 。
+
+```cpp
+#include "tubemq/tubemq_client.h" // TubeMQProducer定义头文件
+
+TubeMQProducer producer;
+```
+
+之后设置`producer`的配置，并启动。
+
+```cpp
+ProducerConfig producer_config;
+producer_config.SetRpcReadTimeoutMs(2);
+producer_config.SetMasterAddrInfo(err_info, master_addr);
+bool result = producer.start(); // 启动producer
+```
+
+当日志显示注册master成功后，producer会以一定频率向master发送心跳，以获得最新的关于topic的信息，之后用户可以开始订阅想发送消息的topic，支持订阅多个topic，传入的topics参数类型为
`std::set`。
+
+```cpp
+std::set topics{"topic_0", "topic_1"};
+bool result = producer.Publish(err_info, topics);
+```
+
+订阅之后就可以发送消息，首先对消息进行构造
+
+```cpp
+#include "tubemq/tubemq_message.h" // tubemq::Message定义头文件
+
+std::string topic_name = "demo";
+std::string send_data = "hello_tubemq";
+Message message(topic_name, send_data.c_str(), send_data.size()); //
接受数据类型为const char*
+```
+
+构造好消息后，即可发送，SDK提供了同步和异步两种发送方式
+
+```cpp
+//同步发送
+bool TubeMQProducer::SendMessage(string& err_info, const Message& message);
+//异步发送
+void TubeMQProducer::SendMessage(const Message& message, const
std::function& callback);
+```
+
+分别通过如下方式使用
+
+```cpp
+bool result = producer.SendMessage(err_info, message);
+
+void callback(const ErrorCode&);
+producer.SendMessage(message, callback); // callback也可以为其它可调用的对象，如lambda函数
+```
+
+同步发送在得到结果前将会阻塞，异步发送则不会，调用后通过用户自定义的回调函数接收返回的结果。
+
+最后，关闭producer
+
+```cpp
+producer.ShutDown();
+```
+
+### Consumer
+
+Consumer同Producer类似，都要先初始化Consumer实例，并设置配置，订阅topic等操作。
+
+```cpp
+#include "tubemq/tubemq_client.h" // TubeMQConsumer定义头文件
+
+TubeMQConsumer consumer;
+
+// 消费相关的配置
+ConsumerConfig consumer_config;
+consumer_config.SetRpcReadTimeoutMs(2);
+consumer_config.SetMasterAddrInfo(err_info, master_addr);
+// consumer中除了订阅topic，还要设置消费组
+consumer_config.SetGroupConsumeTarget(err_info, group_name, topic_list);
+
+// 启动consumer
+consumer.start();
+```
+
+启动consumer后，调用`GetMessage`获取消息即可
+
+```cpp
+ConsumerResult get_result;
+ConsumerResult confirm_result;
+
+bool result = consumer.GetMessage(get_result);
+if (result) {
+list messages = get_result.GetMessageList();
+consumer.Confirm(get_result.GetConfirmContext(), true, confirm_result);
+}
+
+// stop the consumer
+consumer.ShutDown();
+
+```
+
+
Review Comment:
Too many empty lines.
##
docs/sdk/tubemq-sdk/cpp.md:
##
@@ -0,0 +1,153 @@
+---
+title: C++ SDK
+sidebar_position: 1
+---
+
+import {siteVariables} from '../../version';
Review Comment:
This line is not needed for this doc.
##
i18n/zh-CN/docusaurus-plugin-content-docs/current/sdk/tubemq-sdk/python.md:
##
@@ -0,0 +1,107 @@
+---
+title: Python SDK
+sidebar_position: 2
+---
+
+import {siteVariables} from '../../version';
+
+## 构建 TubeMQ Python SDK
+
Review Comment:
Remove the empty line after the titles
##
i18n/zh-CN/docusaurus-plugin-content-docs/current/sdk/tubemq-sdk/cpp.md:
##
@@ -0,0 +1,156 @@
+---
+title: C++ SDK
[GitHub] [inlong] fuweng11 opened a new pull request, #7027: [INLONG-7026][Manager] Fixed the problem of not checked before creating the kafka topic
fuweng11 opened a new pull request, #7027: URL: https://github.com/apache/inlong/pull/7027 ### Prepare a Pull Request - Fixes #7026 ### Motivation Fixed the problem of not checked before creating the kafka topic. ### Modifications Check the existence of a kafka topic before creating it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #7025: [INLONG-7024][Dashboard] Type readable lowercase and title optimization in data integration
dockerzhang merged PR #7025: URL: https://github.com/apache/inlong/pull/7025 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-7024][Dashboard] Type readable lowercase and title optimization in data integration (#7025)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 4b13fc3e6 [INLONG-7024][Dashboard] Type readable lowercase and title
optimization in data integration (#7025)
4b13fc3e6 is described below
commit 4b13fc3e6f60fa6e247f8533da4d9d00eb4be436
Author: Lizhen <88174078+bluew...@users.noreply.github.com>
AuthorDate: Thu Dec 22 19:42:18 2022 +0800
[INLONG-7024][Dashboard] Type readable lowercase and title optimization in
data integration (#7025)
Co-authored-by: Charles Zhang
---
inlong-dashboard/src/locales/cn.json | 6 +++---
inlong-dashboard/src/metas/groups/common/GroupDefaultInfo.ts | 4 +++-
inlong-dashboard/src/metas/sinks/common/SinkDefaultInfo.ts | 4 +++-
inlong-dashboard/src/metas/sources/common/SourceDefaultInfo.ts | 4 +++-
4 files changed, 12 insertions(+), 6 deletions(-)
diff --git a/inlong-dashboard/src/locales/cn.json
b/inlong-dashboard/src/locales/cn.json
index 21adc6fb5..b454a8a1f 100644
--- a/inlong-dashboard/src/locales/cn.json
+++ b/inlong-dashboard/src/locales/cn.json
@@ -500,9 +500,9 @@
"pages.GroupDetail.Stream.CreateDataStream": "新建数据流",
"pages.GroupDetail.Info": "数据流组",
"pages.GroupDetail.Streams": "数据流",
- "pages.GroupDetail.Sources": "数据源",
- "pages.GroupDetail.Sinks": "数据存储",
- "pages.GroupDetail.Audit": "审计",
+ "pages.GroupDetail.Sources": "数据来源",
+ "pages.GroupDetail.Sinks": "数据目标",
+ "pages.GroupDetail.Audit": "审计对账",
"pages.GroupDetail.Audit.DataStream": "数据流",
"pages.GroupDetail.Audit.Date": "日期",
"pages.GroupDetail.Audit.AuditIds": "展示内容",
diff --git a/inlong-dashboard/src/metas/groups/common/GroupDefaultInfo.ts
b/inlong-dashboard/src/metas/groups/common/GroupDefaultInfo.ts
index 55108cf3e..fded2e2ee 100644
--- a/inlong-dashboard/src/metas/groups/common/GroupDefaultInfo.ts
+++ b/inlong-dashboard/src/metas/groups/common/GroupDefaultInfo.ts
@@ -117,7 +117,9 @@ export class GroupDefaultInfo implements DataWithBackend,
RenderRow, RenderList
options: groups.filter(item => Boolean(item.value)),
},
})
- @ColumnDecorator()
+ @ColumnDecorator({
+render: type => groups.find(c => c.value === type)?.label || type,
+ })
@I18n('meta.Group.MQType')
mqType: string;
diff --git a/inlong-dashboard/src/metas/sinks/common/SinkDefaultInfo.ts
b/inlong-dashboard/src/metas/sinks/common/SinkDefaultInfo.ts
index 85fab396b..cd8c2892d 100644
--- a/inlong-dashboard/src/metas/sinks/common/SinkDefaultInfo.ts
+++ b/inlong-dashboard/src/metas/sinks/common/SinkDefaultInfo.ts
@@ -106,7 +106,9 @@ export class SinkDefaultInfo implements DataWithBackend,
RenderRow, RenderList {
})),
}),
})
- @ColumnDecorator()
+ @ColumnDecorator({
+render: type => sinks.find(c => c.value === type)?.label || type,
+ })
@I18n('meta.Sinks.SinkType')
sinkType: string;
diff --git a/inlong-dashboard/src/metas/sources/common/SourceDefaultInfo.ts
b/inlong-dashboard/src/metas/sources/common/SourceDefaultInfo.ts
index 6a7abf43e..fb379eb9f 100644
--- a/inlong-dashboard/src/metas/sources/common/SourceDefaultInfo.ts
+++ b/inlong-dashboard/src/metas/sources/common/SourceDefaultInfo.ts
@@ -98,7 +98,9 @@ export class SourceDefaultInfo implements DataWithBackend,
RenderRow, RenderList
})),
}),
})
- @ColumnDecorator()
+ @ColumnDecorator({
+render: type => sources.find(c => c.value === type)?.label || type,
+ })
@I18n('meta.Sources.Type')
sourceType: string;
[GitHub] [inlong] haifxu opened a new pull request, #7029: [INLONG][Manager] Fix query task status list failed
haifxu opened a new pull request, #7029: URL: https://github.com/apache/inlong/pull/7029 ### Prepare a Pull Request - Fixes #7028 ### Motivation Failed to query the task status when one of them have deleted. ### Modifications Skip this task, guaranteed to return normal information. ### Verifying this change - [ ] This change is a trivial rework/code cleanup without any test coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] gong merged pull request #7027: [INLONG-7026][Manager] Fixed the problem of not checked before creating the kafka topic
gong merged PR #7027: URL: https://github.com/apache/inlong/pull/7027 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-7026][Manager] Fixed the problem of not checked before creating the kafka topic (#7027)
This is an automated email from the ASF dual-hosted git repository.
pacinogong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 2e865155e [INLONG-7026][Manager] Fixed the problem of not checked
before creating the kafka topic (#7027)
2e865155e is described below
commit 2e865155e236fdeb59cb8a8154c063e5d05b3a99
Author: fuweng11 <76141879+fuwen...@users.noreply.github.com>
AuthorDate: Thu Dec 22 20:22:08 2022 +0800
[INLONG-7026][Manager] Fixed the problem of not checked before creating the
kafka topic (#7027)
---
.../inlong/manager/service/resource/queue/kafka/KafkaOperator.java | 5 +
1 file changed, 5 insertions(+)
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/queue/kafka/KafkaOperator.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/queue/kafka/KafkaOperator.java
index 6691b55d0..7e626ec10 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/queue/kafka/KafkaOperator.java
+++
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/queue/kafka/KafkaOperator.java
@@ -49,6 +49,11 @@ public class KafkaOperator {
NewTopic topic = new NewTopic(topicName,
inlongKafkaInfo.getNumPartitions(),
inlongKafkaInfo.getReplicationFactor());
+// Topic will be returned if it exists, and created if it does not
exist
+if (topicIsExists(kafkaClusterInfo, topicName)) {
+LOGGER.warn("kafka topic={} already exists", topicName);
+return;
+}
CreateTopicsResult result =
adminClient.createTopics(Collections.singletonList(topic));
// To prevent the client from disconnecting too quickly and causing
the Topic to not be created successfully
Thread.sleep(500);
[GitHub] [inlong] gong commented on pull request #6931: [INLONG-6925][Sort] Add read phase metric and table level metric for MongoDB-CDC
gong commented on PR #6931: URL: https://github.com/apache/inlong/pull/6931#issuecomment-1362777622 1、git pull upstream master -r 2、git rebase master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] vernedeng commented on a diff in pull request #7029: [INLONG-7028][Manager] Fix query task status list failed
vernedeng commented on code in PR #7029:
URL: https://github.com/apache/inlong/pull/7029#discussion_r1055406495
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/SortServiceImpl.java:
##
@@ -80,7 +81,15 @@ public List listSortStatus(SortStatusRequest
request) {
try {
List groupInfoList =
request.getInlongGroupIds().stream()
-.map(groupId -> groupService.get(groupId))
+.map(groupId -> {
+try {
+return groupService.get(groupId);
+} catch (Exception e) {
+log.debug("{} may have already been deleted, skip
it", groupId);
Review Comment:
Why not print the exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] haifxu commented on a diff in pull request #7029: [INLONG-7028][Manager] Fix query task status list failed
haifxu commented on code in PR #7029:
URL: https://github.com/apache/inlong/pull/7029#discussion_r1055547613
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/SortServiceImpl.java:
##
@@ -80,7 +81,15 @@ public List listSortStatus(SortStatusRequest
request) {
try {
List groupInfoList =
request.getInlongGroupIds().stream()
-.map(groupId -> groupService.get(groupId))
+.map(groupId -> {
+try {
+return groupService.get(groupId);
+} catch (Exception e) {
+log.debug("{} may have already been deleted, skip
it", groupId);
Review Comment:
The exception has been printed in `groupService.get()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] dockerzhang commented on pull request #7001: [INLONG-6998][Dashboard][Manager] Support ignore deserialization error
dockerzhang commented on PR #7001: URL: https://github.com/apache/inlong/pull/7001#issuecomment-1362960712 @healchow @fuweng11 PTAL again, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang commented on pull request #7001: [INLONG-6998][Dashboard][Manager] Support ignore deserialization error
dockerzhang commented on PR #7001: URL: https://github.com/apache/inlong/pull/7001#issuecomment-1362962867 @featzhang could we move the parse error to the advanced options? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] featzhang commented on pull request #7001: [INLONG-6998][Dashboard][Manager] Support ignore deserialization error
featzhang commented on PR #7001: URL: https://github.com/apache/inlong/pull/7001#issuecomment-1362969777 > @featzhang could we move the parse error to the advanced options? It involves the transformation of the front-end module. The larger action will be processed in another pr. I hope the front-end students will support it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] thesumery commented on a diff in pull request #6902: [INLONG-6829][Agent] Support for the task delivery of stream_source according to the tag granularity
thesumery commented on code in PR #6902: URL: https://github.com/apache/inlong/pull/6902#discussion_r1055611046 ## inlong-manager/manager-test/src/main/resources/h2/apache_inlong_manager.sql: ## @@ -329,6 +330,7 @@ CREATE TABLE IF NOT EXISTS `stream_source` `uuid`varchar(30) DEFAULT NULL COMMENT 'Mac uuid of the agent running the task', `data_node_name` varchar(128) DEFAULT NULL COMMENT 'Node name, which links to data_node table', `inlong_cluster_name` varchar(128) DEFAULT NULL COMMENT 'Cluster name of the agent running the task', +`inlong_cluster_node_tag` varchar(128) DEFAULT NULL COMMENT 'Cluster node tag', `serialization_type` varchar(20) DEFAULT NULL COMMENT 'Serialization type, support: csv, json, canal, avro, etc', Review Comment: inlong_cluster_node_tag represent this source task must match those node who has these tag. node_tag represent this node has this tag. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] featzhang opened a new pull request, #7031: [INLONG-7030] Build tool for local debugging environment
featzhang opened a new pull request, #7031: URL: https://github.com/apache/inlong/pull/7031 ### Prepare a Pull Request *[INLONG-7030] Build tool for local debugging environment* - Fixes #7030 ### Motivation The inlong-manager project relies on too many configurations and packages. When debugging locally, multiple default directories need to be created, which is complicated. Therefore, a script is urgently needed to quickly support local debugging. Furthermore, local debugging of other modules also requires similar tools. Temporarily named devkit, looking forward to the addition of more functions. ### Modifications https://user-images.githubusercontent.com/5709212/209177780-5b5a2b1d-b22d-49fe-9296-e334277aa77a.png";> https://user-images.githubusercontent.com/5709212/209179041-5b561ed4-9df7-4cbe-92bd-6b1524290e37.png";> ### Verifying this change *(Please pick either of the following options)* - [x] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang commented on a diff in pull request #7031: [INLONG-7030][Manager][Sort] Build tool for local debugging environment
dockerzhang commented on code in PR #7031: URL: https://github.com/apache/inlong/pull/7031#discussion_r1055962353 ## .gitignore: ## @@ -71,3 +71,7 @@ inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/release/bin/ # Docker mysql storage docker/docker-compose/mysql + +inlong-sort/connectors +plugins +sort-dist.jar Review Comment: It's not wise to add these files, which are used only for your local development env. All these files exist in `./inlong-distribution/target/` after the build. ## inlong-tools/dev/inlong-dev-toolkit.sh: ## @@ -0,0 +1,144 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Initialize the configuration files of inlong components + +shell_dir=$(dirname "$0") Review Comment: Please add a readme(in the same directory) to show how this script is used. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] fuweng11 commented on a diff in pull request #7001: [INLONG-6998][Dashboard][Manager] Support ignore deserialization error
fuweng11 commented on code in PR #7001:
URL: https://github.com/apache/inlong/pull/7001#discussion_r1055965721
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/stream/InlongStreamServiceImpl.java:
##
@@ -514,12 +571,35 @@ public Boolean updateWithoutCheck(InlongStreamRequest
request, String operator)
// update stream fields
updateField(groupId, streamId, request.getFieldList());
// update stream extension infos
-saveOrUpdateExt(groupId, streamId, request.getExtList());
+List extList = request.getExtList();
+extList = packExtParams(request, extList);
+saveOrUpdateExt(groupId, streamId, extList);
LOGGER.info("success to update inlong stream without check for
groupId={} streamId={}", groupId, streamId);
return true;
}
+/**
+ * Pack extended attributes into {@link InlongStreamExtInfo}
+ * @param request the request
+ * @param extList the extList from frontend
+ * @return the packed extParams
+ */
+private List packExtParams(InlongStreamRequest
request,
Review Comment:
I don't think additional parameters can be added in this way.If there are
more parameters later, the code will become very redundant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] fuweng11 commented on a diff in pull request #7001: [INLONG-6998][Dashboard][Manager] Support ignore deserialization error
fuweng11 commented on code in PR #7001:
URL: https://github.com/apache/inlong/pull/7001#discussion_r1055965721
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/stream/InlongStreamServiceImpl.java:
##
@@ -514,12 +571,35 @@ public Boolean updateWithoutCheck(InlongStreamRequest
request, String operator)
// update stream fields
updateField(groupId, streamId, request.getFieldList());
// update stream extension infos
-saveOrUpdateExt(groupId, streamId, request.getExtList());
+List extList = request.getExtList();
+extList = packExtParams(request, extList);
+saveOrUpdateExt(groupId, streamId, extList);
LOGGER.info("success to update inlong stream without check for
groupId={} streamId={}", groupId, streamId);
return true;
}
+/**
+ * Pack extended attributes into {@link InlongStreamExtInfo}
+ * @param request the request
+ * @param extList the extList from frontend
+ * @return the packed extParams
+ */
+private List packExtParams(InlongStreamRequest
request,
Review Comment:
I don't think additional parameters can be added in this way.If there are
more parameters later, the code will become very redundant.
It is recommended to use a DTO class to place additional parameters in ext_
params.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] Yizhou-Yang commented on pull request #6921: [INLONG-6780][Sort] Supports dirty data side-output for JDBC connector and related sinks
Yizhou-Yang commented on PR #6921: URL: https://github.com/apache/inlong/pull/6921#issuecomment-1363531866 close for now, I don't think this is necessary to be merged rn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] Yizhou-Yang closed pull request #6921: [INLONG-6780][Sort] Supports dirty data side-output for JDBC connector and related sinks
Yizhou-Yang closed pull request #6921: [INLONG-6780][Sort] Supports dirty data side-output for JDBC connector and related sinks URL: https://github.com/apache/inlong/pull/6921 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] GanfengTan commented on a diff in pull request #6902: [INLONG-6829][Agent] Support for the task delivery of stream_source according to the tag granularity
GanfengTan commented on code in PR #6902: URL: https://github.com/apache/inlong/pull/6902#discussion_r1055986536 ## inlong-manager/manager-test/src/main/resources/h2/apache_inlong_manager.sql: ## @@ -329,6 +330,7 @@ CREATE TABLE IF NOT EXISTS `stream_source` `uuid`varchar(30) DEFAULT NULL COMMENT 'Mac uuid of the agent running the task', `data_node_name` varchar(128) DEFAULT NULL COMMENT 'Node name, which links to data_node table', `inlong_cluster_name` varchar(128) DEFAULT NULL COMMENT 'Cluster name of the agent running the task', +`inlong_cluster_node_tag` varchar(128) DEFAULT NULL COMMENT 'Cluster node tag', `serialization_type` varchar(20) DEFAULT NULL COMMENT 'Serialization type, support: csv, json, canal, avro, etc', Review Comment: > inlong_cluster_node_tag represent this source task must match those node who has these tag. > node_tag represent this node has this tag. What to do if different cluster name have the same tag? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] leezng opened a new pull request, #7033: [INLONG-7032][Dashboard] Group creation page split basic information and MQ information
leezng opened a new pull request, #7033: URL: https://github.com/apache/inlong/pull/7033 - Fixes #7032 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] featzhang commented on a diff in pull request #7001: [INLONG-6998][Dashboard][Manager] Support ignore deserialization error
featzhang commented on code in PR #7001:
URL: https://github.com/apache/inlong/pull/7001#discussion_r1056011783
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/stream/InlongStreamServiceImpl.java:
##
@@ -514,12 +571,35 @@ public Boolean updateWithoutCheck(InlongStreamRequest
request, String operator)
// update stream fields
updateField(groupId, streamId, request.getFieldList());
// update stream extension infos
-saveOrUpdateExt(groupId, streamId, request.getExtList());
+List extList = request.getExtList();
+extList = packExtParams(request, extList);
+saveOrUpdateExt(groupId, streamId, extList);
LOGGER.info("success to update inlong stream without check for
groupId={} streamId={}", groupId, streamId);
return true;
}
+/**
+ * Pack extended attributes into {@link InlongStreamExtInfo}
+ * @param request the request
+ * @param extList the extList from frontend
+ * @return the packed extParams
+ */
+private List packExtParams(InlongStreamRequest
request,
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] fuweng11 commented on a diff in pull request #7001: [INLONG-6998][Dashboard][Manager] Support ignore deserialization error
fuweng11 commented on code in PR #7001:
URL: https://github.com/apache/inlong/pull/7001#discussion_r1056017176
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/stream/InlongStreamServiceImpl.java:
##
@@ -514,12 +569,31 @@ public Boolean updateWithoutCheck(InlongStreamRequest
request, String operator)
// update stream fields
updateField(groupId, streamId, request.getFieldList());
// update stream extension infos
-saveOrUpdateExt(groupId, streamId, request.getExtList());
+List extList = request.getExtList();
+saveOrUpdateExt(groupId, streamId, extList);
LOGGER.info("success to update inlong stream without check for
groupId={} streamId={}", groupId, streamId);
return true;
}
+/**
+ * Pack extended attributes into ExtParams
+ *
+ * @param request the request
+ * @return the packed extParams
+ */
+private String packExtParams(InlongStreamRequest request) {
+InlongStreamExtParamBuilder builder = InlongStreamExtParam.builder();
Review Comment:
It is recommended to use `CommonBeanUtils.copyProperties(request,
InlongStreamExtParam::new, true);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] featzhang commented on a diff in pull request #7001: [INLONG-6998][Dashboard][Manager] Support ignore deserialization error
featzhang commented on code in PR #7001:
URL: https://github.com/apache/inlong/pull/7001#discussion_r1056019029
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/stream/InlongStreamServiceImpl.java:
##
@@ -514,12 +569,31 @@ public Boolean updateWithoutCheck(InlongStreamRequest
request, String operator)
// update stream fields
updateField(groupId, streamId, request.getFieldList());
// update stream extension infos
-saveOrUpdateExt(groupId, streamId, request.getExtList());
+List extList = request.getExtList();
+saveOrUpdateExt(groupId, streamId, extList);
LOGGER.info("success to update inlong stream without check for
groupId={} streamId={}", groupId, streamId);
return true;
}
+/**
+ * Pack extended attributes into ExtParams
+ *
+ * @param request the request
+ * @return the packed extParams
+ */
+private String packExtParams(InlongStreamRequest request) {
+InlongStreamExtParamBuilder builder = InlongStreamExtParam.builder();
Review Comment:
The `org.apache.inlong.manager.dao.entity.InlongStreamEntity`
does not contain the same fields, json must be compressed into extParams, so
`CommonBeanUtils.copyProperties(request, InlongStreamExtParam::new, true)`
cannot be used;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] fuweng11 commented on a diff in pull request #7001: [INLONG-6998][Dashboard][Manager] Support ignore deserialization error
fuweng11 commented on code in PR #7001:
URL: https://github.com/apache/inlong/pull/7001#discussion_r1056019234
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/stream/InlongStreamServiceImpl.java:
##
@@ -385,6 +424,16 @@ public PageResult
listAll(InlongStreamPageRequest request) {
return pageResult;
}
+private void unpackExtParams(InlongStreamInfo streamInfo) {
+String extParams = streamInfo.getExtParams();
Review Comment:
It seems that there is a method with the same function in line 244. It is
recommended to unify the two methods
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] featzhang commented on a diff in pull request #7001: [INLONG-6998][Dashboard][Manager] Support ignore deserialization error
featzhang commented on code in PR #7001:
URL: https://github.com/apache/inlong/pull/7001#discussion_r1056019029
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/stream/InlongStreamServiceImpl.java:
##
@@ -514,12 +569,31 @@ public Boolean updateWithoutCheck(InlongStreamRequest
request, String operator)
// update stream fields
updateField(groupId, streamId, request.getFieldList());
// update stream extension infos
-saveOrUpdateExt(groupId, streamId, request.getExtList());
+List extList = request.getExtList();
+saveOrUpdateExt(groupId, streamId, extList);
LOGGER.info("success to update inlong stream without check for
groupId={} streamId={}", groupId, streamId);
return true;
}
+/**
+ * Pack extended attributes into ExtParams
+ *
+ * @param request the request
+ * @return the packed extParams
+ */
+private String packExtParams(InlongStreamRequest request) {
+InlongStreamExtParamBuilder builder = InlongStreamExtParam.builder();
Review Comment:
The `org.apache.inlong.manager.dao.entity.InlongStreamEntity`
does not contain the same fields, json must be compressed into extParams, so
`CommonBeanUtils.copyProperties(request, InlongStreamExtParam::new, true)`
cannot be used;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] featzhang commented on a diff in pull request #7001: [INLONG-6998][Dashboard][Manager] Support ignore deserialization error
featzhang commented on code in PR #7001:
URL: https://github.com/apache/inlong/pull/7001#discussion_r1056021411
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/stream/InlongStreamServiceImpl.java:
##
@@ -514,12 +569,31 @@ public Boolean updateWithoutCheck(InlongStreamRequest
request, String operator)
// update stream fields
updateField(groupId, streamId, request.getFieldList());
// update stream extension infos
-saveOrUpdateExt(groupId, streamId, request.getExtList());
+List extList = request.getExtList();
+saveOrUpdateExt(groupId, streamId, extList);
LOGGER.info("success to update inlong stream without check for
groupId={} streamId={}", groupId, streamId);
return true;
}
+/**
+ * Pack extended attributes into ExtParams
+ *
+ * @param request the request
+ * @return the packed extParams
+ */
+private String packExtParams(InlongStreamRequest request) {
+InlongStreamExtParamBuilder builder = InlongStreamExtParam.builder();
Review Comment:
ok, done
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/stream/InlongStreamServiceImpl.java:
##
@@ -385,6 +424,16 @@ public PageResult
listAll(InlongStreamPageRequest request) {
return pageResult;
}
+private void unpackExtParams(InlongStreamInfo streamInfo) {
+String extParams = streamInfo.getExtParams();
Review Comment:
ok, done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] fuweng11 commented on a diff in pull request #7001: [INLONG-6998][Dashboard][Manager] Support ignore deserialization error
fuweng11 commented on code in PR #7001:
URL: https://github.com/apache/inlong/pull/7001#discussion_r1056021702
##
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/stream/InlongStreamExtParam.java:
##
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.pojo.stream;
+
+import java.io.Serializable;
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import lombok.AllArgsConstructor;
+import lombok.Builder;
+import lombok.Data;
+import lombok.EqualsAndHashCode;
+import lombok.NoArgsConstructor;
+
+/**
+ * Extended params, will be saved as JSON string
+ */
+@Data
+@EqualsAndHashCode(callSuper = false)
+@NoArgsConstructor
+@AllArgsConstructor
+@Builder
+@ApiModel("Inlong stream info")
Review Comment:
Inlong stream ext param info.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] featzhang commented on a diff in pull request #7031: [INLONG-7030][Manager][Sort] Build tool for local debugging environment
featzhang commented on code in PR #7031: URL: https://github.com/apache/inlong/pull/7031#discussion_r1056022969 ## .gitignore: ## @@ -71,3 +71,7 @@ inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/release/bin/ # Docker mysql storage docker/docker-compose/mysql + +inlong-sort/connectors +plugins +sort-dist.jar Review Comment: These files are temporarily created for local debugging in the IDE, So need to be ignored. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] featzhang commented on a diff in pull request #7031: [INLONG-7030][Manager][Sort] Build tool for local debugging environment
featzhang commented on code in PR #7031: URL: https://github.com/apache/inlong/pull/7031#discussion_r1056022969 ## .gitignore: ## @@ -71,3 +71,7 @@ inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/release/bin/ # Docker mysql storage docker/docker-compose/mysql + +inlong-sort/connectors +plugins +sort-dist.jar Review Comment: These soft link files are temporarily created for local debugging in the ide, and they also need to be ignored. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] featzhang commented on a diff in pull request #7031: [INLONG-7030][Manager][Sort] Build tool for local debugging environment
featzhang commented on code in PR #7031: URL: https://github.com/apache/inlong/pull/7031#discussion_r1056022969 ## .gitignore: ## @@ -71,3 +71,7 @@ inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/release/bin/ # Docker mysql storage docker/docker-compose/mysql + +inlong-sort/connectors +plugins +sort-dist.jar Review Comment: These soft link files are temporarily created for local debugging in the ide. So they need to be ignored. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] woofyzhao opened a new pull request, #7036: [INLONG-7035][DataProxy-SDK] Fix the id generator not globally unique…
woofyzhao opened a new pull request, #7036: URL: https://github.com/apache/inlong/pull/7036 - Fixes #7035 ### Motivation The message sender generate repeated msg id when in concurrency. ### Modifications Make it globally unique by static sharing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] fuweng11 commented on a diff in pull request #7001: [INLONG-6998][Dashboard][Manager] Support ignore deserialization error
fuweng11 commented on code in PR #7001: URL: https://github.com/apache/inlong/pull/7001#discussion_r1056065451 ## inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/source/kafka/KafkaSourceOperator.java: ## @@ -111,16 +110,19 @@ public Map> getSourcesMap(InlongGroupInfo groupInfo, kafkaSource.setSourceName(streamId); kafkaSource.setBootstrapServers(bootstrapServers); kafkaSource.setTopic(streamInfo.getMqResource()); +kafkaSource.setSerializationType(streamInfo.getDataType()); +String serializationType = DataTypeEnum.forType(streamInfo.getDataType()).getType(); +kafkaSource.setSerializationType(serializationType); Review Comment: Why set the value twice? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] fuweng11 commented on a diff in pull request #7001: [INLONG-6998][Dashboard][Manager] Support ignore deserialization error
fuweng11 commented on code in PR #7001:
URL: https://github.com/apache/inlong/pull/7001#discussion_r1056068132
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/stream/InlongStreamServiceImpl.java:
##
@@ -514,12 +562,25 @@ public Boolean updateWithoutCheck(InlongStreamRequest
request, String operator)
// update stream fields
updateField(groupId, streamId, request.getFieldList());
// update stream extension infos
-saveOrUpdateExt(groupId, streamId, request.getExtList());
+List extList = request.getExtList();
+saveOrUpdateExt(groupId, streamId, extList);
LOGGER.info("success to update inlong stream without check for
groupId={} streamId={}", groupId, streamId);
return true;
}
+/**
+ * Pack extended attributes into ExtParams
+ *
+ * @param request the request
+ * @return the packed extParams
+ */
+private String packExtParams(InlongStreamRequest request) {
Review Comment:
It is recommended to put it in the class `inlongStreamExtParam`.
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/stream/InlongStreamServiceImpl.java:
##
@@ -226,6 +240,27 @@ public InlongStreamInfo get(String groupId, String
streamId) {
return streamInfo;
}
+/**
+ * Unpack extended attributes from {@link InlongStreamExtInfo}, will
remove target attributes from it.
+ *
+ * @param extParams the {@link InlongStreamEntity} load from db
+ * @param targetObject the targetObject with to fill up
+ */
+private void unpackExtParams(
Review Comment:
It is recommended to put it in the class `inlongStreamExtParam`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] vernedeng opened a new pull request, #7038: [INLONG-7073][Sort] Remove exception when topic does not exist
vernedeng opened a new pull request, #7038: URL: https://github.com/apache/inlong/pull/7038 ### Prepare a Pull Request - Fixes #7037 ### Motivation Make sure that when one stream offline from a task, the remain event will be ack and commit successfully. ### Modifications When the related topic does not exist, should return **_Null Kafka IdConfig_** instead of throw exception. ### Verifying this change - [ ] This change is a trivial rework/code cleanup without any test coverage. ### Documentation - Does this pull request introduce a new feature? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] dockerzhang commented on a diff in pull request #660: [INLONG-659][Doc] Add docs for c++/python sdk.
dockerzhang commented on code in PR #660: URL: https://github.com/apache/inlong-website/pull/660#discussion_r1056081112 ## docs/sdk/tubemq-sdk/cpp.md: ## @@ -0,0 +1,143 @@ +--- +title: C++ SDK +sidebar_position: 1 +--- + +## Build TubeMQ C++ SDK +C++ SDK is based on the non-boost asio, and the CMake is used for building, the commands are： +```bash +# enter the root directory of c++ sdk source +cd `pwd`/inlong/inlong-tubemq/tubemq-client-twins/tubemq-client-cpp Review Comment: Change to the online source URL. https://github.com/apache/inlong/tree/master/inlong-tubemq/tubemq-client-twins/tubemq-client-cpp ## i18n/zh-CN/docusaurus-plugin-content-docs/current/sdk/tubemq-sdk/cpp.md: ## @@ -0,0 +1,144 @@ +--- +title: C++ SDK +sidebar_position: 1 +--- + +## 构建 TubeMQ C++ SDK +C++ SDK 主要基于非 Boost 版本的 Asio 进行通信，构建时使用 CMake，具体操作步骤如下： +```bash +# 进入到 cpp sdk 的根路径 +cd `pwd`/inlong/inlong-tubemq/tubemq-client-twins/tubemq-client-cpp Review Comment: ditto -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] dockerzhang commented on a diff in pull request #660: [INLONG-659][Doc] Add docs for c++/python sdk.
dockerzhang commented on code in PR #660: URL: https://github.com/apache/inlong-website/pull/660#discussion_r1056083840 ## i18n/zh-CN/docusaurus-plugin-content-docs/current/sdk/tubemq-sdk/cpp.md: ## @@ -0,0 +1,144 @@ +--- +title: C++ SDK +sidebar_position: 1 +--- + +## 构建 TubeMQ C++ SDK +C++ SDK 主要基于非 Boost 版本的 Asio 进行通信，构建时使用 CMake，具体操作步骤如下： +```bash +# 进入到 cpp sdk 的根路径 +cd `pwd`/inlong/inlong-tubemq/tubemq-client-twins/tubemq-client-cpp Review Comment: ```suggestion git clone https://github.com/apache/inlong.git cd inlong/inlong-tubemq/tubemq-client-twins/tubemq-client-cpp ``` ## docs/sdk/tubemq-sdk/cpp.md: ## @@ -0,0 +1,143 @@ +--- +title: C++ SDK +sidebar_position: 1 +--- + +## Build TubeMQ C++ SDK +C++ SDK is based on the non-boost asio, and the CMake is used for building, the commands are： +```bash +# enter the root directory of c++ sdk source +cd `pwd`/inlong/inlong-tubemq/tubemq-client-twins/tubemq-client-cpp Review Comment: ```suggestion git clone https://github.com/apache/inlong.git cd inlong/inlong-tubemq/tubemq-client-twins/tubemq-client-cpp ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] dockerzhang merged pull request #660: [INLONG-659][Doc] Add docs for c++/python sdk.
dockerzhang merged PR #660: URL: https://github.com/apache/inlong-website/pull/660 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong-website] branch master updated: [INLONG-659][Doc] Add docs for c++/python sdk. (#660)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new dfcbc95ec8 [INLONG-659][Doc] Add docs for c++/python sdk. (#660)
dfcbc95ec8 is described below
commit dfcbc95ec855d15157b3f31047979844c9050a3e
Author: Xue Huanran
AuthorDate: Fri Dec 23 15:02:51 2022 +0800
[INLONG-659][Doc] Add docs for c++/python sdk. (#660)
Co-authored-by: huanranxue
Co-authored-by: Charles Zhang
---
docs/sdk/tubemq-sdk/_category_.json| 4 +
docs/sdk/tubemq-sdk/cpp.md | 144
docs/sdk/tubemq-sdk/python.md | 96 ++
.../current/sdk/tubemq-sdk/cpp.md | 145 +
.../current/sdk/tubemq-sdk/python.md | 94 +
5 files changed, 483 insertions(+)
diff --git a/docs/sdk/tubemq-sdk/_category_.json
b/docs/sdk/tubemq-sdk/_category_.json
new file mode 100644
index 00..a042e9a5db
--- /dev/null
+++ b/docs/sdk/tubemq-sdk/_category_.json
@@ -0,0 +1,4 @@
+{
+"label": "TubeMQ SDK",
+"position": 3
+ }
\ No newline at end of file
diff --git a/docs/sdk/tubemq-sdk/cpp.md b/docs/sdk/tubemq-sdk/cpp.md
new file mode 100644
index 00..dbba229f0b
--- /dev/null
+++ b/docs/sdk/tubemq-sdk/cpp.md
@@ -0,0 +1,144 @@
+---
+title: C++ SDK
+sidebar_position: 1
+---
+
+## Build TubeMQ C++ SDK
+C++ SDK is based on the non-boost asio, and the CMake is used for building,
the commands are：
+```bash
+# enter the root directory of c++ sdk source
+git clone https://github.com/apache/inlong.git
+cd inlong/inlong-tubemq/tubemq-client-twins/tubemq-client-cpp
+
+mkdir build && cd build
+
+cmake ..
+
+make -j8 # the thread num is based on the cpu cores
+
+```
+
+The building can also be completed in
[docker](https://github.com/apache/inlong/tree/master/inlong-tubemq/tubemq-docker/tubemq-cpp)
environment provided by InLong.
+
+```bash
+# pull image
+docker pull inlong/tubemq-cpp
+
+# start container and mount the source code
+docker run -it --net=host -v ${REPLACE_BY_CPP_SOURCE_DIR_PATH}:/tubemq-cpp/
inlong/tubemq-cpp /bin/bash
+
+# same as the build process in physical machines
+mkdir build && cd build
+cmake ..
+make -j8
+```
+
+## C++ SDK API
+Similar with other MQ systems，C++ SDK is diveded into **Producer** and
**Consumer**. The API of Producer and Consumer are introduced respectively
below.
+
+First of all, regardless of role, start the background global TubeMQ SDK
service (support the default C++ `namespace` is `tubemq`)。
+
+```cpp
+bool StartTubeMQService(string& err_info, const TubeMQServiceConfig&
serviceConfig);
+```
+
+Look up the return boolean value and the `err_info` to check the start process
is successful。
+
+### Producer
+The user-level api class is `TubeMQProducer`，the first thing is initializing
the class.
+
+```cpp
+#include "tubemq/tubemq_client.h" // header file of TubeMQProducer
+
+TubeMQProducer producer;
+```
+
+Set the config of producer, then start the producer
+
+```cpp
+ProducerConfig producer_config;
+producer_config.SetRpcReadTimeoutMs(2);
+producer_config.SetMasterAddrInfo(err_info, master_addr);
+bool result = producer.start(); // start producer
+```
+
+When `register2Master` is successed，producer will send heartbeat request to
master periodically to update the meta info of topics. Then users can publish
topics, multiple topics can be published at once, and the param type is
`std::set`.
+
+```cpp
+std::set topics{"topic_0", "topic_1"};
+bool result = producer.Publish(err_info, topics);
+```
+
+After publishing, construct the message
+
+```cpp
+#include "tubemq/tubemq_message.h" // header file of tubemq::Message
+
+std::string topic_name = "demo";
+std::string send_data = "hello_world";
+Message message(topic_name, send_data.c_str(), send_data.size()); // type is
const char*
+```
+
+There are two `SendMessage` API: synchronous sending and asynchronous sending.
+
+```cpp
+// sync send
+bool TubeMQProducer::SendMessage(string& err_info, const Message& message);
+// async send
+void TubeMQProducer::SendMessage(const Message& message, const
std::function& callback);
+```
+
+How to use these two `SendMessage`
+
+```cpp
+bool result = producer.SendMessage(err_info, message);
+
+void callback(const ErrorCode&);
+producer.SendMessage(message, callback); // callback can be other invoked
objects
+```
+
+Synchronous sending will block until the result is returned, asynchronous
sending will not, and the returnted result is received through the user-defined
callback function.
+
+Finally, close the producer.
+
+```cpp
+producer.ShutDown();
+```
+
+### Consumer
+Similar with producer，initialize the consumer object and set the config.
+
+```cpp
+#include "tubemq/tubemq_client.h" // header file of TubeMQConsumer
+
+TubeMQConsumer consu
[GitHub] [inlong] healchow commented on a diff in pull request #7036: [INLONG-7035][DataProxy] Fix the id generator not globally unique problem
healchow commented on code in PR #7036:
URL: https://github.com/apache/inlong/pull/7036#discussion_r1056091025
##
inlong-sdk/dataproxy-sdk/src/main/java/org/apache/inlong/sdk/dataproxy/DefaultMessageSender.java:
##
@@ -48,8 +48,8 @@ public class DefaultMessageSender implements MessageSender {
new ConcurrentHashMap<>();
private static final AtomicBoolean MANAGER_FETCHER_THREAD_STARTED = new
AtomicBoolean(false);
private static ManagerFetcherThread managerFetcherThread;
+private static final SequentialID idGenerator;
Review Comment:
```suggestion
private static final SequentialID idGenerator = new
SequentialID(Utils.getLocalIp());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #7036: [INLONG-7035][DataProxy] Fix the id generator not globally unique problem
healchow commented on code in PR #7036:
URL: https://github.com/apache/inlong/pull/7036#discussion_r1056091301
##
inlong-sdk/dataproxy-sdk/src/main/java/org/apache/inlong/sdk/dataproxy/DefaultMessageSender.java:
##
@@ -48,8 +48,8 @@ public class DefaultMessageSender implements MessageSender {
new ConcurrentHashMap<>();
private static final AtomicBoolean MANAGER_FETCHER_THREAD_STARTED = new
AtomicBoolean(false);
private static ManagerFetcherThread managerFetcherThread;
+private static final SequentialID idGenerator;
Review Comment:
The not need a static block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] dockerzhang commented on a diff in pull request #7033: [INLONG-7032][Dashboard] Group creation page split basic information and MQ information
dockerzhang commented on code in PR #7033: URL: https://github.com/apache/inlong/pull/7033#discussion_r1056092853 ## inlong-dashboard/src/locales/cn.json: ## @@ -456,6 +456,8 @@ "pages.GroupDashboard.ConfirmDelete": "确认删除吗", "pages.GroupDashboard.Create": "新建数据流组", "pages.GroupDashboard.SuccessfullyDeleted": "删除成功", + "pages.GroupDetail.Info.Basic": "基础信息", + "pages.GroupDetail.Info.Mq": "MQ信息", Review Comment: ```suggestion "pages.GroupDetail.Info.Mq": "MQ 信息", ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #7029: [INLONG-7028][Manager] Fix query task status list failed
dockerzhang merged PR #7029: URL: https://github.com/apache/inlong/pull/7029 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated (2e865155e -> 3bfbbb2e9)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong.git from 2e865155e [INLONG-7026][Manager] Fixed the problem of not checked before creating the kafka topic (#7027) add 3bfbbb2e9 [INLONG-7028][Manager] Fix query task status list failed (#7029) No new revisions were added by this update. Summary of changes: .../inlong/manager/service/core/impl/SortServiceImpl.java | 11 ++- 1 file changed, 10 insertions(+), 1 deletion(-)
[GitHub] [inlong] dockerzhang merged pull request #7038: [INLONG-7073][Sort] Remove exception when topic does not exist
dockerzhang merged PR #7038: URL: https://github.com/apache/inlong/pull/7038 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-7073][Sort] Remove exception when topic does not exist (#7038)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 6f5bde761 [INLONG-7073][Sort] Remove exception when topic does not
exist (#7038)
6f5bde761 is described below
commit 6f5bde7614f823dfe1fd669e79f08384c3a474a6
Author: vernedeng
AuthorDate: Fri Dec 23 15:25:20 2022 +0800
[INLONG-7073][Sort] Remove exception when topic does not exist (#7038)
---
.../sort/standalone/sink/kafka/KafkaFederationSinkContext.java | 6 ++
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git
a/inlong-sort-standalone/sort-standalone-source/src/main/java/org/apache/inlong/sort/standalone/sink/kafka/KafkaFederationSinkContext.java
b/inlong-sort-standalone/sort-standalone-source/src/main/java/org/apache/inlong/sort/standalone/sink/kafka/KafkaFederationSinkContext.java
index e6a880f54..a1dfa5630 100644
---
a/inlong-sort-standalone/sort-standalone-source/src/main/java/org/apache/inlong/sort/standalone/sink/kafka/KafkaFederationSinkContext.java
+++
b/inlong-sort-standalone/sort-standalone-source/src/main/java/org/apache/inlong/sort/standalone/sink/kafka/KafkaFederationSinkContext.java
@@ -35,6 +35,7 @@ import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Objects;
import java.util.concurrent.ConcurrentHashMap;
/** Context of kafka sink. */
@@ -119,10 +120,7 @@ public class KafkaFederationSinkContext extends
SinkContext {
*/
public String getTopic(String uid) {
KafkaIdConfig idConfig = this.idConfigMap.get(uid);
-if (idConfig == null) {
-throw new NullPointerException("uid " + uid + "got null topic");
-}
-return idConfig.getTopic();
+return Objects.isNull(idConfig) ? null : idConfig.getTopic();
}
/**
[GitHub] [inlong] dockerzhang merged pull request #7020: [INLONG-7019][Sort] Rethrow the system error when flush data to doris
dockerzhang merged PR #7020: URL: https://github.com/apache/inlong/pull/7020 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated (6f5bde761 -> 9a6b4665f)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong.git from 6f5bde761 [INLONG-7073][Sort] Remove exception when topic does not exist (#7038) add 9a6b4665f [INLONG-7019][Sort] Rethrow the system error when flush data to doris (#7020) No new revisions were added by this update. Summary of changes: .../table/DorisDynamicSchemaOutputFormat.java | 27 -- 1 file changed, 15 insertions(+), 12 deletions(-)
[GitHub] [inlong] gosonzhang commented on issue #7039: [Bug][Manager] Unknown method isNoneBlank
gosonzhang commented on issue #7039: URL: https://github.com/apache/inlong/issues/7039#issuecomment-1363692208 @woofyzhao, let me see it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang commented on a diff in pull request #7031: [INLONG-7030][Manager][Sort] Build tool for local debugging environment
dockerzhang commented on code in PR #7031:
URL: https://github.com/apache/inlong/pull/7031#discussion_r1056109281
##
inlong-tools/dev/README.md:
##
@@ -0,0 +1,96 @@
+
+
+# InLong dev toolkit
+
+## Motivation
+
+The InLong project includes modules such as `dashboard`, `dataproxy`,
`manager`, and `tubemq`. When
+debugging in the development environment such as `Intellij IDEA`, it faces the
problem that the
+environment is difficult to build.
+
+The `inlong-manager` project relies on too many configurations and packages.
When debugging locally
+in IDE, multiple default directories need to be created, which is complicated.
+Therefore, a script is urgently needed to quickly support local debugging.
+
+Furthermore, local debugging of other modules also requires similar tools.
Temporarily named devkit,
+looking forward to the addition of more functions.
+
+## Use case
+
+### Help
+
+
+
+### Manager
+
+Rely of Manager:
+
+| Directory/File |
link target
| Rely modules|
+|--||-|
+| `./plugins` |
`inlong-manager/manger-plugins/target/plugins`
| `inlong-sort/sort-plugin` |
+| `./inlong-sort/connectors` | -
| `inlong-sort/sort-connectors/*` |
|
+| `./inlong-sort/connectors/sort-connector-${name}-${project.version}.jar` |
`inlong-sort/sort-connectors/${connector.name}/target/sort-connector-${connector.name}-${project.version}.jar`
| `inlong-sort/sort-connectors/*` |
+| `./sort-dist.jar`|
`sort-dist/target/sort-dist-${project.version}.jar`
| `inlong-sort/sort-dist` |
+
+```shell
+inlong-tools/dev/inlong-dev-toolkit.sh m
+```
+
+
Review Comment:
Please remove the image.
##
inlong-tools/dev/README.md:
##
@@ -0,0 +1,96 @@
+
+
+# InLong dev toolkit
Review Comment:
InLong
->
Apache InLong
and please start with secondary title(##)
##
inlong-tools/dev/inlong-dev-toolkit.sh:
##
@@ -0,0 +1,146 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# Initialize the configuration files of inlong components
+
+script_dir=$(dirname "$0")
+script_file_name=$(basename "$0")
+
+# the absolute dir of project
+base_dir=$(cd $script_dir; cd ../..; pwd)
+
+help_action=(
+ 'help'
+ 'h'
+ 'help'
+ 'welcome'
+)
+
+manager_action=(
+ 'manager'
+ 'm'
+ 'build manager local debugging environment'
+ 'manager'
+)
+
+actions=(
+ help_action
+ manager_action
+)
+
+
+function welcome() {
+ echo
''
+ echo '#Welcome to use InLong dev toolkit !
#'
Review Comment:
InLong
->
Apache InLong
##
inlong-tools/dev/README.md:
##
@@ -0,0 +1,96 @@
+
+
+# InLong dev toolkit
+
+## Motivation
+
+The InLong project includes modules such as `dashboard`, `dataproxy`,
`manager`, and `tubemq`. When
Review Comment:
dashboard -> Dashbaord
dataproxy -> DataProxy
manager -> Manager
tubemq -> TubeMQ
##
inlong-tools/dev/README.md:
##
@@ -0,0 +1,96 @@
+
+
+# InLong dev toolkit
+
+## Motivation
+
+The InLong project includes modules such as `dashboard`, `dataproxy`,
`manager`, and `tubemq`. When
+debugging in the development environment such as `Intellij IDEA`, it faces the
problem that the
+env
[GitHub] [inlong] dockerzhang merged pull request #7033: [INLONG-7032][Dashboard] Group creation page split basic information and MQ information
dockerzhang merged PR #7033: URL: https://github.com/apache/inlong/pull/7033 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-7032][Dashboard] Group creation page split basic information and MQ information (#7033)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 0e1047918 [INLONG-7032][Dashboard] Group creation page split basic
information and MQ information (#7033)
0e1047918 is described below
commit 0e104791855dfafd2289288f339e56fc40346b66
Author: Daniel
AuthorDate: Fri Dec 23 15:51:38 2022 +0800
[INLONG-7032][Dashboard] Group creation page split basic information and MQ
information (#7033)
Co-authored-by: Charles Zhang
---
inlong-dashboard/src/locales/cn.json | 2 ++
inlong-dashboard/src/locales/en.json | 6 +++-
.../src/pages/GroupDetail/Info/config.tsx | 32 --
.../src/pages/GroupDetail/Info/index.tsx | 2 +-
4 files changed, 38 insertions(+), 4 deletions(-)
diff --git a/inlong-dashboard/src/locales/cn.json
b/inlong-dashboard/src/locales/cn.json
index b454a8a1f..54382ab28 100644
--- a/inlong-dashboard/src/locales/cn.json
+++ b/inlong-dashboard/src/locales/cn.json
@@ -456,6 +456,8 @@
"pages.GroupDashboard.ConfirmDelete": "确认删除吗",
"pages.GroupDashboard.Create": "新建数据流组",
"pages.GroupDashboard.SuccessfullyDeleted": "删除成功",
+ "pages.GroupDetail.Info.Basic": "基础信息",
+ "pages.GroupDetail.Info.Mq": "MQ 信息",
"pages.GroupDetail.Info.Back": "返回",
"pages.GroupDetail.Info.NextStep": "下一步",
"pages.GroupDetail.Info.DataStreams": "数据流",
diff --git a/inlong-dashboard/src/locales/en.json
b/inlong-dashboard/src/locales/en.json
index 4a19aa09a..64ae4a6f5 100644
--- a/inlong-dashboard/src/locales/en.json
+++ b/inlong-dashboard/src/locales/en.json
@@ -417,6 +417,9 @@
"meta.Nodes.Iceberg.Url": "URL",
"meta.Nodes.Iceberg.Warehouse": "Warehouse",
"meta.Nodes.Iceberg.CatalogType": "Catalog Type",
+ "meta.Nodes.ClickHouse.Username": "Username",
+ "meta.Nodes.ClickHouse.Password": "Password",
+ "meta.Nodes.ClickHouse.Url": "URL",
"meta.Nodes.StarRocks.Username": "Username",
"meta.Nodes.StarRocks.Password": "Password",
"meta.Nodes.StarRocks.Url": "URL",
@@ -453,6 +456,8 @@
"pages.GroupDashboard.ConfirmDelete": "Confirm delete?",
"pages.GroupDashboard.Create": "Create",
"pages.GroupDashboard.SuccessfullyDeleted": "Successfully deleted",
+ "pages.GroupDetail.Info.Basic": "Basic",
+ "pages.GroupDetail.Info.Mq": "MQ",
"pages.GroupDetail.Info.Back": "Back",
"pages.GroupDetail.Info.NextStep": "Next",
"pages.GroupDetail.Info.DataStreams": "Data streams",
@@ -573,7 +578,6 @@
"pages.ConsumeDashboard.status.Passed": "Passed",
"pages.ConsumeDashboard.status.Cancelled": "Cancelled",
"pages.ConsumeDashboard.NewSubscribe": "Create",
- "pages.ConsumeDetail.Info.config.ConsumerGroupID": "Consumer group ID",
"pages.ConsumeDetail.SubscribeDetails": "Subscription Detail",
"pages.Clusters.Type": "Type",
"pages.Clusters.TypeAll": "All",
diff --git a/inlong-dashboard/src/pages/GroupDetail/Info/config.tsx
b/inlong-dashboard/src/pages/GroupDetail/Info/config.tsx
index 59634c38f..e895cefdf 100644
--- a/inlong-dashboard/src/pages/GroupDetail/Info/config.tsx
+++ b/inlong-dashboard/src/pages/GroupDetail/Info/config.tsx
@@ -17,7 +17,9 @@
* under the License.
*/
-import { useMemo } from 'react';
+import React, { useCallback, useMemo } from 'react';
+import { Divider } from 'antd';
+import i18n from '@/i18n';
import { useLoadMeta, GroupMetaType } from '@/metas';
import { excludeObjectArray } from '@/utils';
@@ -31,7 +33,7 @@ export const useFormContent = ({ mqType, editing, isCreate,
isUpdate }) => {
const excludeKeys = ['ensemble'].concat(isCreate ? 'mqResource' : '');
const fields = excludeObjectArray(excludeKeys, entityFields || []);
- return isCreate
+ const formContent = isCreate
? fields.map(item => {
if (item.name === 'inlongGroupId' && isUpdate) {
return {
@@ -60,6 +62,32 @@ export const useFormContent = ({ mqType, editing, isCreate,
isUpdate }) => {
rules: t === 'text' ? undefined : item.rules,
};
});
+
+ const { Entity: DefaultEntity } = useLoadMeta('group', '');
+
+ const isMqKey = useCallback(
+formName => {
+ const defaultGroupKeysI18nMap = DefaultEntity?.I18nMap || {};
+ return !defaultGroupKeysI18nMap[formName] || formName === 'mqType';
+},
+[DefaultEntity?.I18nMap],
+ );
+
+ const basicFormContent = formContent.filter(item => !isMqKey(item.name));
+ const mqFormContent = formContent.filter(item => isMqKey(item.name));
+
+ return [
+{
+ type: {i18n.t('pages.GroupDetail.Info.Basic')},
+ col: 24,
+},
+...basicFormContent,
+{
+ type: {i18n.t('pages.GroupDetail.Info.Mq')},
+ col: 24,
+},
+...mqFormContent,
+ ];
};
function transType(editing: boolean, conf) {
diff --git a/inlong-dashboard/src/pages/GroupDetail/Info/index.tsx
b/inlong-da
