[PR] [INLONG-10633][Agent] The initialization function of AuditUtils pass in the configuration [inlong]
justinwwhuang opened a new pull request, #10634: URL: https://github.com/apache/inlong/pull/10634 Fixes #10633 ### Motivation The initialization function of AuditUtils needs to pass in the configuration, because the config files of installer and agent are different. ### Modifications Initialize function adds configuration input parameters ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation No doc needed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-10409][Dashboard] Support installing agents by SSH key-based auth [inlong]
shuiqingliu opened a new pull request, #10636: URL: https://github.com/apache/inlong/pull/10636 Fixes #10409 ### Motivation Implement SSH key-based authentication when installing the agent via SSH. This PR will provide a dashboard UI that allows users to retrieve the SSH public key and then manually upload it to the agent node. ### Modifications * Identification Type Addition: Introduced identify type to distinguish between password and SSH key authentication methods. * SSH Key Tips: Added tips to guide users on uploading the SSH authentication key when selecting the SSH key type. * Test Connection Feature: Implemented a 'Test Connection' button, allowing users to verify the correctness of their configuration settings. * Localization Enhancements: Enhanced descriptions for both Chinese (CN) and English (EN) localizations to improve user understanding and accessibility.  ### Verifying this change *(Please pick either of the following options)* - [x] This change is already covered by existing tests, such as: *(please describe tests)* Successfully tested both SSH password type and SSH key type connections to the agent node. This verification confirms that the new identification types and the 'Test Connection' feature are functioning as expected.   ### Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-10409][Dashboard] Support installing agents by SSH key-based auth [inlong]
dockerzhang commented on PR #10636: URL: https://github.com/apache/inlong/pull/10636#issuecomment-2230365840 @wohainilaodou PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-10409][Dashboard] Support installing agents by SSH key-based auth [inlong]
dockerzhang commented on code in PR #10636: URL: https://github.com/apache/inlong/pull/10636#discussion_r1679015747 ## inlong-dashboard/src/ui/locales/cn.json: ## @@ -771,9 +771,12 @@ "pages.Clusters.Node.IsInstall": "安装方式", "pages.Clusters.Node.ManualInstall": "手动安装", "pages.Clusters.Node.SSHInstall": "SSH 安装", + "pages.Clusters.Node.IdentifyType": "认证方式", "pages.Clusters.Node.Username": "SSH 用户名", "pages.Clusters.Node.Password": "SSH 密码", + "pages.Clusters.Node.SSHKey": "SSH 密钥", "pages.Clusters.Node.SSHPort": "SSH 端口", + "pages.Clusters.Node.SSHKeyHelper": "请将公钥上传至 agent 节点的 ~/.ssh/authorized_keys 文件中", Review Comment: ```suggestion "pages.Clusters.Node.SSHKeyHelper": "请将公钥上传至 Agent 节点的 ~/.ssh/authorized_keys 文件中", ``` ## inlong-dashboard/src/ui/locales/en.json: ## @@ -771,9 +771,12 @@ "pages.Clusters.Node.IsInstall": "Installation", "pages.Clusters.Node.ManualInstall": "Manual", "pages.Clusters.Node.SSHInstall": "SSH", + "pages.Clusters.Node.IdentifyType": "Identify Type", "pages.Clusters.Node.Username": "SSH Username", + "pages.Clusters.Node.SSHKey": "SSH Key", "pages.Clusters.Node.Password": "SSH Password", "pages.Clusters.Node.SSHPort": "SSH Port", + "pages.Clusters.Node.SSHKeyHelper": "Please upload the public key to the ~/.ssh/authorized_keys file of the agent node", Review Comment: ```suggestion "pages.Clusters.Node.SSHKeyHelper": "Please upload the public key to the ~/.ssh/authorized_keys file of the Agent node", ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-10635][Manager] Optimize the installer configuration process [inlong]
fuweng11 opened a new pull request, #10637: URL: https://github.com/apache/inlong/pull/10637 Fixes #10635 ### Motivation Optimize the installer configuration process. ### Modifications Optimize the installer configuration process. 1.When setting up the agent node, module config will be loaded into agent_task_config. 2.If you actively install the installer, it will obtain the default module configuration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-10635][Manager] Optimize the installer configuration process [inlong]
vernedeng commented on code in PR #10637:
URL: https://github.com/apache/inlong/pull/10637#discussion_r1679050149
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/cluster/node/AgentClusterNodeOperator.java:
##
@@ -88,4 +164,94 @@ protected void setTargetEntity(ClusterNodeRequest request,
InlongClusterNodeEnti
String.format("serialize extParams of Agent ClusterNode
failure: %s", e.getMessage()));
}
}
+
+public void updateModuleConfig(String ip, String clusterName) {
+try {
+ConfigRequest configRequest = new ConfigRequest();
+LOGGER.info("begin to update module config for cluster name={},
ip={}", clusterName, ip);
+if (StringUtils.isBlank(clusterName) || StringUtils.isBlank(ip)) {
+LOGGER.info("no need to update module config when ip or
cluster name is null");
+return;
+}
+configRequest.setLocalIp(ip);
+configRequest.setClusterName(clusterName);
+ConfigResult configResult = loadModuleConfigs(configRequest);
+AgentTaskConfigEntity existEntity =
agentTaskConfigEntityMapper.selectByIdentifier(ip, clusterName);
+AgentTaskConfigEntity agentTaskConfigEntity = new
AgentTaskConfigEntity();
Review Comment:
```suggestion
AgentTaskConfigEntity agentTaskConfigEntity = existEntity ==
null ? new AgentTaskConfigEntity() :
CommonBeanUtils.copyProperties(existEntity, AgentTaskConfigEntity::new, true);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10635][Manager] Optimize the installer configuration process [inlong]
vernedeng commented on code in PR #10637:
URL: https://github.com/apache/inlong/pull/10637#discussion_r1679052833
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/cluster/node/AgentClusterNodeOperator.java:
##
@@ -88,4 +164,94 @@ protected void setTargetEntity(ClusterNodeRequest request,
InlongClusterNodeEnti
String.format("serialize extParams of Agent ClusterNode
failure: %s", e.getMessage()));
}
}
+
+public void updateModuleConfig(String ip, String clusterName) {
+try {
+ConfigRequest configRequest = new ConfigRequest();
+LOGGER.info("begin to update module config for cluster name={},
ip={}", clusterName, ip);
+if (StringUtils.isBlank(clusterName) || StringUtils.isBlank(ip)) {
+LOGGER.info("no need to update module config when ip or
cluster name is null");
+return;
+}
+configRequest.setLocalIp(ip);
+configRequest.setClusterName(clusterName);
+ConfigResult configResult = loadModuleConfigs(configRequest);
+AgentTaskConfigEntity existEntity =
agentTaskConfigEntityMapper.selectByIdentifier(ip, clusterName);
+AgentTaskConfigEntity agentTaskConfigEntity = new
AgentTaskConfigEntity();
+if (existEntity != null) {
+agentTaskConfigEntity =
CommonBeanUtils.copyProperties(existEntity, AgentTaskConfigEntity::new, true);
+}
+agentTaskConfigEntity.setAgentIp(ip);
+agentTaskConfigEntity.setClusterName(clusterName);
+
agentTaskConfigEntity.setModuleParams(objectMapper.writeValueAsString(configResult));
+if (existEntity == null) {
+agentTaskConfigEntityMapper.insert(agentTaskConfigEntity);
+} else {
+
agentTaskConfigEntityMapper.updateByIdSelective(agentTaskConfigEntity);
+}
+LOGGER.info("success to update module config for cluster name={},
ip={}", clusterName, ip);
+} catch (Exception e) {
+LOGGER.error("load module config failed", e);
+throw new BusinessException("load module config faield");
+}
+}
+
+private ConfigResult loadModuleConfigs(ConfigRequest request) {
+final String clusterName = request.getClusterName();
+final String ip = request.getLocalIp();
+LOGGER.debug("begin to load config for installer = {}", request);
+Preconditions.expectTrue(StringUtils.isNotBlank(clusterName), "cluster
name is blank");
+InlongClusterEntity clusterEntity =
clusterMapper.selectByNameAndType(clusterName, ClusterType.AGENT);
+List configs = new ArrayList<>();
+if (clusterEntity != null) {
+List clusterNodeEntityList =
+
clusterNodeMapper.selectByParentIdAndIp(clusterEntity.getId(), ip);
+if (CollectionUtils.isNotEmpty(clusterNodeEntityList)) {
+AgentClusterNodeDTO dto =
AgentClusterNodeDTO.getFromJson(clusterNodeEntityList.get(0).getExtParams());
+configs = getModuleConfigs(dto);
+}
+}
+String jsonStr = GSON.toJson(configs);
+String configMd5 = DigestUtils.md5Hex(jsonStr);
+
+ConfigResult configResult =
ConfigResult.builder().moduleList(configs).md5(configMd5)
+.code(AgentResponseCode.SUCCESS)
+.build();
+LOGGER.info("success load module config, size = {}",
configResult.getModuleList().size());
+return configResult;
+}
+
+private List getModuleConfigs(AgentClusterNodeDTO dto) {
+List moduleIdList = dto.getModuleIdList();
+List configs = new ArrayList<>();
+if (CollectionUtils.isEmpty(moduleIdList)) {
+return configs;
+}
+for (Integer moduleId : moduleIdList) {
+ModuleConfigEntity moduleConfigEntity =
moduleConfigEntityMapper.selectByPrimaryKey(moduleId);
+ModuleConfig moduleConfig =
CommonBeanUtils.copyProperties(moduleConfigEntity, ModuleConfig::new);
Review Comment:
may produce NPE?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10635][Manager] Optimize the installer configuration process [inlong]
fuweng11 commented on code in PR #10637:
URL: https://github.com/apache/inlong/pull/10637#discussion_r1679139431
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/cluster/node/AgentClusterNodeOperator.java:
##
@@ -88,4 +164,94 @@ protected void setTargetEntity(ClusterNodeRequest request,
InlongClusterNodeEnti
String.format("serialize extParams of Agent ClusterNode
failure: %s", e.getMessage()));
}
}
+
+public void updateModuleConfig(String ip, String clusterName) {
+try {
+ConfigRequest configRequest = new ConfigRequest();
+LOGGER.info("begin to update module config for cluster name={},
ip={}", clusterName, ip);
+if (StringUtils.isBlank(clusterName) || StringUtils.isBlank(ip)) {
+LOGGER.info("no need to update module config when ip or
cluster name is null");
+return;
+}
+configRequest.setLocalIp(ip);
+configRequest.setClusterName(clusterName);
+ConfigResult configResult = loadModuleConfigs(configRequest);
+AgentTaskConfigEntity existEntity =
agentTaskConfigEntityMapper.selectByIdentifier(ip, clusterName);
+AgentTaskConfigEntity agentTaskConfigEntity = new
AgentTaskConfigEntity();
+if (existEntity != null) {
+agentTaskConfigEntity =
CommonBeanUtils.copyProperties(existEntity, AgentTaskConfigEntity::new, true);
+}
+agentTaskConfigEntity.setAgentIp(ip);
+agentTaskConfigEntity.setClusterName(clusterName);
+
agentTaskConfigEntity.setModuleParams(objectMapper.writeValueAsString(configResult));
+if (existEntity == null) {
+agentTaskConfigEntityMapper.insert(agentTaskConfigEntity);
+} else {
+
agentTaskConfigEntityMapper.updateByIdSelective(agentTaskConfigEntity);
+}
+LOGGER.info("success to update module config for cluster name={},
ip={}", clusterName, ip);
+} catch (Exception e) {
+LOGGER.error("load module config failed", e);
+throw new BusinessException("load module config faield");
+}
+}
+
+private ConfigResult loadModuleConfigs(ConfigRequest request) {
+final String clusterName = request.getClusterName();
+final String ip = request.getLocalIp();
+LOGGER.debug("begin to load config for installer = {}", request);
+Preconditions.expectTrue(StringUtils.isNotBlank(clusterName), "cluster
name is blank");
+InlongClusterEntity clusterEntity =
clusterMapper.selectByNameAndType(clusterName, ClusterType.AGENT);
+List configs = new ArrayList<>();
+if (clusterEntity != null) {
+List clusterNodeEntityList =
+
clusterNodeMapper.selectByParentIdAndIp(clusterEntity.getId(), ip);
+if (CollectionUtils.isNotEmpty(clusterNodeEntityList)) {
+AgentClusterNodeDTO dto =
AgentClusterNodeDTO.getFromJson(clusterNodeEntityList.get(0).getExtParams());
+configs = getModuleConfigs(dto);
+}
+}
+String jsonStr = GSON.toJson(configs);
+String configMd5 = DigestUtils.md5Hex(jsonStr);
+
+ConfigResult configResult =
ConfigResult.builder().moduleList(configs).md5(configMd5)
+.code(AgentResponseCode.SUCCESS)
+.build();
+LOGGER.info("success load module config, size = {}",
configResult.getModuleList().size());
+return configResult;
+}
+
+private List getModuleConfigs(AgentClusterNodeDTO dto) {
+List moduleIdList = dto.getModuleIdList();
+List configs = new ArrayList<>();
+if (CollectionUtils.isEmpty(moduleIdList)) {
+return configs;
+}
+for (Integer moduleId : moduleIdList) {
+ModuleConfigEntity moduleConfigEntity =
moduleConfigEntityMapper.selectByPrimaryKey(moduleId);
+ModuleConfig moduleConfig =
CommonBeanUtils.copyProperties(moduleConfigEntity, ModuleConfig::new);
Review Comment:
Fixed.
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/cluster/node/AgentClusterNodeOperator.java:
##
@@ -88,4 +164,94 @@ protected void setTargetEntity(ClusterNodeRequest request,
InlongClusterNodeEnti
String.format("serialize extParams of Agent ClusterNode
failure: %s", e.getMessage()));
}
}
+
+public void updateModuleConfig(String ip, String clusterName) {
+try {
+ConfigRequest configRequest = new ConfigRequest();
+LOGGER.info("begin to update module config for cluster name={},
ip={}", clusterName, ip);
+if (StringUtils.isBlank(clusterName) || StringUtils.isBlank(ip)) {
+LOGGER.info("no need to update module config when ip or
cluster name is null");
+
Re: [PR] [INLONG-10635][Manager] Optimize the installer configuration process [inlong]
aloyszhang commented on code in PR #10637:
URL: https://github.com/apache/inlong/pull/10637#discussion_r1679186254
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/cluster/node/AgentClusterNodeOperator.java:
##
@@ -88,4 +163,95 @@ protected void setTargetEntity(ClusterNodeRequest request,
InlongClusterNodeEnti
String.format("serialize extParams of Agent ClusterNode
failure: %s", e.getMessage()));
}
}
+
+public void updateModuleConfig(String ip, String clusterName) {
+try {
+ConfigRequest configRequest = new ConfigRequest();
+LOGGER.info("begin to update module config for cluster name={},
ip={}", clusterName, ip);
+if (StringUtils.isBlank(clusterName) || StringUtils.isBlank(ip)) {
+LOGGER.info("no need to update module config when ip or
cluster name is null");
+return;
+}
Review Comment:
blank check move to the top of the method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10635][Manager] Optimize the installer configuration process [inlong]
aloyszhang commented on code in PR #10637:
URL: https://github.com/apache/inlong/pull/10637#discussion_r1679189618
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/AgentServiceImpl.java:
##
@@ -323,6 +300,15 @@ public void reload() {
agentConfigInfo.setVersion(agentTaskConfigEntity.getVersion());
newAgentConfigMap.putIfAbsent(key, agentConfigInfo);
}
+ConfigResult configResult =
+
JsonUtils.parseObject(agentTaskConfigEntity.getModuleParams(),
ConfigResult.class);
+if (configResult != null) {
+
configResult.setVersion(agentTaskConfigEntity.getVersion());
+newInstallerConfigMap.putIfAbsent(key, configResult);
+if (Objects.equals(agentTaskConfigEntity.getAgentIp(),
"9.135.95.77")) {
+LOGGER.info("Test get md5={}, configresult={}",
configResult.getMd5(), configResult);
Review Comment:
```suggestion
LOGGER.info("Test get md5={}, configResult={}",
configResult.getMd5(), configResult);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10635][Manager] Optimize the installer configuration process [inlong]
fuweng11 commented on code in PR #10637:
URL: https://github.com/apache/inlong/pull/10637#discussion_r1679197893
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/AgentServiceImpl.java:
##
@@ -323,6 +300,15 @@ public void reload() {
agentConfigInfo.setVersion(agentTaskConfigEntity.getVersion());
newAgentConfigMap.putIfAbsent(key, agentConfigInfo);
}
+ConfigResult configResult =
+
JsonUtils.parseObject(agentTaskConfigEntity.getModuleParams(),
ConfigResult.class);
+if (configResult != null) {
+
configResult.setVersion(agentTaskConfigEntity.getVersion());
+newInstallerConfigMap.putIfAbsent(key, configResult);
+if (Objects.equals(agentTaskConfigEntity.getAgentIp(),
"9.135.95.77")) {
+LOGGER.info("Test get md5={}, configresult={}",
configResult.getMd5(), configResult);
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10635][Manager] Optimize the installer configuration process [inlong]
fuweng11 commented on code in PR #10637:
URL: https://github.com/apache/inlong/pull/10637#discussion_r1679199887
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/cluster/node/AgentClusterNodeOperator.java:
##
@@ -88,4 +163,95 @@ protected void setTargetEntity(ClusterNodeRequest request,
InlongClusterNodeEnti
String.format("serialize extParams of Agent ClusterNode
failure: %s", e.getMessage()));
}
}
+
+public void updateModuleConfig(String ip, String clusterName) {
+try {
+ConfigRequest configRequest = new ConfigRequest();
+LOGGER.info("begin to update module config for cluster name={},
ip={}", clusterName, ip);
+if (StringUtils.isBlank(clusterName) || StringUtils.isBlank(ip)) {
+LOGGER.info("no need to update module config when ip or
cluster name is null");
+return;
+}
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10638][Manager] Data preview supports filtering function [inlong]
vernedeng commented on code in PR #10639:
URL: https://github.com/apache/inlong/pull/10639#discussion_r1679257959
##
inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/inner/client/InlongStreamClient.java:
##
@@ -295,11 +300,15 @@ public List parseFields(String method,
String statement) {
return parseFields(request);
}
-public List queryMessage(String groupId, String streamId,
Integer messageCount) {
-Preconditions.expectNotBlank(groupId, ErrorCodeEnum.GROUP_ID_IS_EMPTY);
-Preconditions.expectNotBlank(streamId,
ErrorCodeEnum.STREAM_ID_IS_EMPTY);
+public List queryMessage(QueryMessageRequest request) {
+Preconditions.expectNotBlank(request.getGroupId(),
ErrorCodeEnum.GROUP_ID_IS_EMPTY);
+Preconditions.expectNotBlank(request.getStreamId(),
ErrorCodeEnum.STREAM_ID_IS_EMPTY);
+Map requestMap =
JsonUtils.OBJECT_MAPPER.convertValue(request,
Review Comment:
JsonUtils.parseObject
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10638][Manager] Data preview supports filtering function [inlong]
vernedeng commented on code in PR #10639:
URL: https://github.com/apache/inlong/pull/10639#discussion_r1679261325
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/message/InlongMsgDeserializeOperator.java:
##
@@ -87,6 +86,10 @@ public List decodeMsg(InlongStreamInfo
streamInfo, byte[] msgByt
DataTypeOperator dataTypeOperator =
dataTypeOperatorFactory.getInstance(DataTypeEnum.forType(streamInfo.getDataType()));
List streamFieldList =
dataTypeOperator.parseFields(body, streamInfo);
+boolean ifFilter = checkIfFilter(request, streamFieldList);
+if (!ifFilter) {
Review Comment:
```suggestion
if (!checkIfFilter(request, streamFieldList))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10638][Manager] Data preview supports filtering function [inlong]
vernedeng commented on code in PR #10639: URL: https://github.com/apache/inlong/pull/10639#discussion_r1679261639 ## inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/message/RawMsgDeserializeOperator.java: ## @@ -59,6 +60,10 @@ public List decodeMsg(InlongStreamInfo streamInfo, DataTypeOperator dataTypeOperator = dataTypeOperatorFactory.getInstance(DataTypeEnum.forType(streamInfo.getDataType())); List fieldList = dataTypeOperator.parseFields(body, streamInfo); +boolean ifFilter = checkIfFilter(request, fieldList); Review Comment: ditto -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-10638][Manager] Data preview supports filtering function [inlong]
fuweng11 commented on code in PR #10639:
URL: https://github.com/apache/inlong/pull/10639#discussion_r1679264686
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/message/InlongMsgDeserializeOperator.java:
##
@@ -87,6 +86,10 @@ public List decodeMsg(InlongStreamInfo
streamInfo, byte[] msgByt
DataTypeOperator dataTypeOperator =
dataTypeOperatorFactory.getInstance(DataTypeEnum.forType(streamInfo.getDataType()));
List streamFieldList =
dataTypeOperator.parseFields(body, streamInfo);
+boolean ifFilter = checkIfFilter(request, streamFieldList);
+if (!ifFilter) {
Review Comment:
Fixed.
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/message/RawMsgDeserializeOperator.java:
##
@@ -59,6 +60,10 @@ public List decodeMsg(InlongStreamInfo
streamInfo,
DataTypeOperator dataTypeOperator =
dataTypeOperatorFactory.getInstance(DataTypeEnum.forType(streamInfo.getDataType()));
List fieldList = dataTypeOperator.parseFields(body,
streamInfo);
+boolean ifFilter = checkIfFilter(request, fieldList);
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10638][Manager] Data preview supports filtering function [inlong]
fuweng11 commented on code in PR #10639:
URL: https://github.com/apache/inlong/pull/10639#discussion_r1679266496
##
inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/inner/client/InlongStreamClient.java:
##
@@ -295,11 +300,15 @@ public List parseFields(String method,
String statement) {
return parseFields(request);
}
-public List queryMessage(String groupId, String streamId,
Integer messageCount) {
-Preconditions.expectNotBlank(groupId, ErrorCodeEnum.GROUP_ID_IS_EMPTY);
-Preconditions.expectNotBlank(streamId,
ErrorCodeEnum.STREAM_ID_IS_EMPTY);
+public List queryMessage(QueryMessageRequest request) {
+Preconditions.expectNotBlank(request.getGroupId(),
ErrorCodeEnum.GROUP_ID_IS_EMPTY);
+Preconditions.expectNotBlank(request.getStreamId(),
ErrorCodeEnum.STREAM_ID_IS_EMPTY);
+Map requestMap =
JsonUtils.OBJECT_MAPPER.convertValue(request,
Review Comment:
The purpose here is to convert the object into a map, which
`JsonUtils.parseObject` cannot achieve.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] [Improve][Sort] New so many MetricsController obejects [inlong]
github-actions[bot] commented on issue #10240: URL: https://github.com/apache/inlong/issues/10240#issuecomment-2232173874 This issue is stale because it has been open for 60 days with no activity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-10628][Sort] Implement the end2end test env on flink1.18 [inlong]
EMsnap merged PR #10629: URL: https://github.com/apache/inlong/pull/10629 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-10628][Sort] Implement the end2end test env on flink1.18 (#10629)
This is an automated email from the ASF dual-hosted git repository.
zirui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 043626c5ee [INLONG-10628][Sort] Implement the end2end test env on
flink1.18 (#10629)
043626c5ee is described below
commit 043626c5ee7b149adfc7afbe29e9fa97a1e9f595
Author: XiaoYou201 <58425449+xiaoyou...@users.noreply.github.com>
AuthorDate: Wed Jul 17 10:27:37 2024 +0800
[INLONG-10628][Sort] Implement the end2end test env on flink1.18 (#10629)
---
inlong-sort/sort-end-to-end-tests/pom.xml | 6 +
.../sort-end-to-end-tests-v1.18/pom.xml| 113 ++
.../sort/tests/utils/FlinkContainerTestEnv.java| 241 +
.../tests/utils/FlinkContainerTestEnvJRE11.java| 55 +
.../tests/utils/FlinkContainerTestEnvJRE8.java | 55 +
.../sort/tests/utils/PlaceholderResolver.java | 150 +
.../apache/inlong/sort/tests/utils/TestUtils.java | 124 +++
.../src/main/resources/log4j2-test.properties | 82 +++
8 files changed, 826 insertions(+)
diff --git a/inlong-sort/sort-end-to-end-tests/pom.xml
b/inlong-sort/sort-end-to-end-tests/pom.xml
index 8109574f20..be4a7418ee 100644
--- a/inlong-sort/sort-end-to-end-tests/pom.xml
+++ b/inlong-sort/sort-end-to-end-tests/pom.xml
@@ -52,6 +52,12 @@
sort-end-to-end-tests-v1.15
+
+v1.18
+
+sort-end-to-end-tests-v1.18
+
+
diff --git
a/inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.18/pom.xml
b/inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.18/pom.xml
new file mode 100644
index 00..59ecfe2886
--- /dev/null
+++ b/inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.18/pom.xml
@@ -0,0 +1,113 @@
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+org.apache.inlong
+sort-end-to-end-tests
+1.13.0-SNAPSHOT
+
+
+sort-end-to-end-tests-v1.18
+Apache InLong - Sort End to End Tests v1.18
+
+
+
${project.parent.parent.parent.basedir}
+1.18.1
+
+
+
+
+org.testcontainers
+testcontainers
+
+
+
+org.apache.inlong
+sort-dist
+${project.version}
+test
+
+
+org.apache.flink
+flink-test-utils
+${flink.version}
+test
+
+
+org.apache.logging.log4j
+log4j-slf4j-impl
+
+
+org.apache.logging.log4j
+log4j-core
+
+
+
+
+org.apache.inlong
+sort-flink-dependencies-v1.18
+${project.version}
+test
+
+
+org.apache.flink
+flink-core
+${flink.version}
+test
+
+
+org.apache.flink
+flink-json
+${flink.version}
+test
+
+
+org.apache.flink
+flink-avro
+${flink.version}
+test
+
+
+org.apache.flink
+flink-csv
+${flink.version}
+test
+
+
+org.apache.flink
+flink-sql-avro
+${flink.version}
+test
+
+
+org.apache.flink
+flink-runtime
+${flink.version}
+test
+
+
+org.apache.flink
+flink-table-common
+${flink.version}
+test
+
+
+
diff --git
a/inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.18/src/main/java/org/apache/inlong/sort/tests/utils/FlinkContainerTestEnv.java
b/inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.18/src/main/java/org/apache/inlong/sort/tests/utils/FlinkContainerTestEnv.java
new file mode 100644
index 00..de6166442e
--- /dev/null
+++
b/inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.18/src/main/java/org/apache/inlong/sort/tests/utils/FlinkContainerTestEnv.java
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
Re: [PR] [INLONG-10635][Manager] Optimize the installer configuration process [inlong]
aloyszhang merged PR #10637: URL: https://github.com/apache/inlong/pull/10637 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-10635][Manager] Optimize the installer configuration process (#10637)
This is an automated email from the ASF dual-hosted git repository.
aloyszhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 0c02cfdee8 [INLONG-10635][Manager] Optimize the installer
configuration process (#10637)
0c02cfdee8 is described below
commit 0c02cfdee8476a8123b10903863574c46d0556bb
Author: fuweng11 <76141879+fuwen...@users.noreply.github.com>
AuthorDate: Wed Jul 17 11:40:28 2024 +0800
[INLONG-10635][Manager] Optimize the installer configuration process
(#10637)
---
.../common/pojo/agent/installer/ConfigResult.java | 2 +
.../common/pojo/agent/installer/ModuleConfig.java | 4 +
.../manager/dao/entity/AgentTaskConfigEntity.java | 1 +
.../dao/mapper/ModuleConfigEntityMapper.java | 8 +
.../dao/mapper/PackageConfigEntityMapper.java | 8 +
.../mappers/AgentTaskConfigEntityMapper.xml| 12 +-
.../main/resources/mappers/ModuleConfigMapper.xml | 8 +
.../main/resources/mappers/PackageConfigMapper.xml | 8 +
.../cluster/node/AgentClusterNodeOperator.java | 168 +++-
.../inlong/manager/service/core/ConfigLoader.java | 16 ++
.../service/core/impl/AgentServiceImpl.java| 218 -
.../service/core/impl/ConfigLoaderImpl.java| 27 +++
.../main/resources/h2/apache_inlong_manager.sql| 3 +-
.../manager-web/sql/apache_inlong_manager.sql | 3 +-
inlong-manager/manager-web/sql/changes-1.13.0.sql | 3 +-
.../src/main/resources/application-dev.properties | 3 +-
16 files changed, 389 insertions(+), 103 deletions(-)
diff --git
a/inlong-common/src/main/java/org/apache/inlong/common/pojo/agent/installer/ConfigResult.java
b/inlong-common/src/main/java/org/apache/inlong/common/pojo/agent/installer/ConfigResult.java
index b2c5426a81..781b0eb090 100644
---
a/inlong-common/src/main/java/org/apache/inlong/common/pojo/agent/installer/ConfigResult.java
+++
b/inlong-common/src/main/java/org/apache/inlong/common/pojo/agent/installer/ConfigResult.java
@@ -45,6 +45,8 @@ public class ConfigResult {
*/
private String md5;
+private Integer version;
+
/**
* The list of module config list
*/
diff --git
a/inlong-common/src/main/java/org/apache/inlong/common/pojo/agent/installer/ModuleConfig.java
b/inlong-common/src/main/java/org/apache/inlong/common/pojo/agent/installer/ModuleConfig.java
index c76e42522d..517903b2dc 100644
---
a/inlong-common/src/main/java/org/apache/inlong/common/pojo/agent/installer/ModuleConfig.java
+++
b/inlong-common/src/main/java/org/apache/inlong/common/pojo/agent/installer/ModuleConfig.java
@@ -32,6 +32,10 @@ import lombok.NoArgsConstructor;
public class ModuleConfig {
private Integer id;
+/**
+ * The primary key ID of the manager module config table
+ */
+private Integer entityId;
private String name;
/**
* The md5 of the module config
diff --git
a/inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/entity/AgentTaskConfigEntity.java
b/inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/entity/AgentTaskConfigEntity.java
index cc8c3ab529..ed539e370b 100644
---
a/inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/entity/AgentTaskConfigEntity.java
+++
b/inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/entity/AgentTaskConfigEntity.java
@@ -36,6 +36,7 @@ public class AgentTaskConfigEntity implements Serializable {
private String configParams;
private String taskParams;
+private String moduleParams;
private Integer isDeleted;
private String creator;
diff --git
a/inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/mapper/ModuleConfigEntityMapper.java
b/inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/mapper/ModuleConfigEntityMapper.java
index cec1881377..253fe297ff 100644
---
a/inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/mapper/ModuleConfigEntityMapper.java
+++
b/inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/mapper/ModuleConfigEntityMapper.java
@@ -17,10 +17,14 @@
package org.apache.inlong.manager.dao.mapper;
+import org.apache.inlong.manager.common.tenant.MultiTenantQuery;
import org.apache.inlong.manager.dao.entity.ModuleConfigEntity;
import org.apache.inlong.manager.pojo.module.ModulePageRequest;
+import org.apache.ibatis.annotations.Options;
import org.apache.ibatis.annotations.Param;
+import org.apache.ibatis.cursor.Cursor;
+import org.apache.ibatis.mapping.ResultSetType;
import org.springframework.stereotype.Repository;
import java.util.List;
@@ -36,4 +40,8 @@ public interface ModuleConfigEntityMapper {
List selectByCondition(@Param("request")
ModulePageRequest request);
+@MultiTenantQuery(with = false)
+@Options(resultSetType = ResultSetType.FORWARD
Re: [PR] [INLONG-10633][Agent] The initialization function of AuditUtils pass in the configuration [inlong]
aloyszhang merged PR #10634: URL: https://github.com/apache/inlong/pull/10634 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-10633][Agent] The initialization function of AuditUtils needs to pass in the configuration (#10634)
This is an automated email from the ASF dual-hosted git repository.
aloyszhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 5e9112fab5 [INLONG-10633][Agent] The initialization function of
AuditUtils needs to pass in the configuration (#10634)
5e9112fab5 is described below
commit 5e9112fab5d36226cefadda25e6bfe0af93ef08e
Author: justinwwhuang
AuthorDate: Wed Jul 17 14:07:55 2024 +0800
[INLONG-10633][Agent] The initialization function of AuditUtils needs to
pass in the configuration (#10634)
---
.../main/java/org/apache/inlong/agent/metrics/audit/AuditUtils.java | 6 +++---
.../src/main/java/org/apache/inlong/agent/core/AgentMain.java | 2 +-
.../src/main/java/org/apache/inlong/agent/installer/Main.java | 2 +-
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git
a/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/metrics/audit/AuditUtils.java
b/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/metrics/audit/AuditUtils.java
index 3dc920ed5e..9cc5e5fd6e 100644
---
a/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/metrics/audit/AuditUtils.java
+++
b/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/metrics/audit/AuditUtils.java
@@ -17,7 +17,7 @@
package org.apache.inlong.agent.metrics.audit;
-import org.apache.inlong.agent.conf.AgentConfiguration;
+import org.apache.inlong.agent.conf.AbstractConfiguration;
import org.apache.inlong.audit.AuditOperator;
import org.apache.inlong.audit.entity.AuditComponent;
@@ -59,12 +59,12 @@ public class AuditUtils {
public static int AUDIT_ID_AGENT_ADD_INSTANCE_MEM_FAILED = 1073741842;
public static int AUDIT_ID_AGENT_DEL_INSTANCE_MEM_UNUSUAL = 1073741843;
private static boolean IS_AUDIT = true;
+private static AbstractConfiguration conf;
/**
* Init audit config
*/
-public static void initAudit() {
-AgentConfiguration conf = AgentConfiguration.getAgentConf();
+public static void initAudit(AbstractConfiguration conf) {
IS_AUDIT = conf.getBoolean(AUDIT_ENABLE, DEFAULT_AUDIT_ENABLE);
if (IS_AUDIT) {
AuditOperator.getInstance().setAuditProxy(AuditComponent.AGENT,
conf.get(AGENT_MANAGER_ADDR),
diff --git
a/inlong-agent/agent-core/src/main/java/org/apache/inlong/agent/core/AgentMain.java
b/inlong-agent/agent-core/src/main/java/org/apache/inlong/agent/core/AgentMain.java
index 3584ac033f..695c27e231 100644
---
a/inlong-agent/agent-core/src/main/java/org/apache/inlong/agent/core/AgentMain.java
+++
b/inlong-agent/agent-core/src/main/java/org/apache/inlong/agent/core/AgentMain.java
@@ -110,7 +110,7 @@ public class AgentMain {
CommandLine cl = initOptions(args);
assert cl != null;
initAgentConf(cl);
-AuditUtils.initAudit();
+AuditUtils.initAudit(AgentConfiguration.getAgentConf());
AgentManager manager = new AgentManager();
try {
manager.start();
diff --git
a/inlong-agent/agent-installer/src/main/java/org/apache/inlong/agent/installer/Main.java
b/inlong-agent/agent-installer/src/main/java/org/apache/inlong/agent/installer/Main.java
index f03739b3d3..121282f5d3 100644
---
a/inlong-agent/agent-installer/src/main/java/org/apache/inlong/agent/installer/Main.java
+++
b/inlong-agent/agent-installer/src/main/java/org/apache/inlong/agent/installer/Main.java
@@ -110,7 +110,7 @@ public class Main {
CommandLine cl = initOptions(args);
assert cl != null;
initAgentConf(cl);
-AuditUtils.initAudit();
+AuditUtils.initAudit(InstallerConfiguration.getInstallerConf());
Manager manager = new Manager();
try {
manager.start();
