[GitHub] [inlong] dockerzhang merged pull request #5886: [INLONG-5885][Manager] Refactor the task issue logic to simply code complexity
dockerzhang merged PR #5886: URL: https://github.com/apache/inlong/pull/5886 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-5885][Manager] Refactor the task issue logic to simply code complexity (#5886)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 69e17cbcb [INLONG-5885][Manager] Refactor the task issue logic to
simply code complexity (#5886)
69e17cbcb is described below
commit 69e17cbcb3ac1bbceeb5336147c266987c4b4dcd
Author: woofyzhao <490467...@qq.com>
AuthorDate: Wed Sep 14 15:31:55 2022 +0800
[INLONG-5885][Manager] Refactor the task issue logic to simply code
complexity (#5886)
Co-authored-by: healchow
---
.../service/core/impl/AgentServiceImpl.java| 102 +++--
.../inlong/manager/service/ServiceBaseTest.java| 1 +
.../service/core/impl/AgentServiceTest.java| 68 ++
3 files changed, 123 insertions(+), 48 deletions(-)
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/AgentServiceImpl.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/AgentServiceImpl.java
index 2258581a9..111035c87 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/AgentServiceImpl.java
+++
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/AgentServiceImpl.java
@@ -97,7 +97,7 @@ public class AgentServiceImpl implements AgentService {
// Update task status, other tasks with status 20x will change to 30x
in next request
if (CollectionUtils.isEmpty(request.getCommandInfo())) {
-LOGGER.warn("task result was empty, just return");
+LOGGER.info("task result was empty in request: {}, just return",
request);
return;
}
for (CommandEntity command : request.getCommandInfo()) {
@@ -211,61 +211,17 @@ public class AgentServiceImpl implements AgentService {
}
final String agentIp = taskRequest.getAgentIp();
final String agentClusterName = taskRequest.getClusterName();
-final String uuid = taskRequest.getUuid();
Preconditions.checkTrue(StringUtils.isNotBlank(agentIp) ||
StringUtils.isNotBlank(agentClusterName),
"both agent ip and cluster name are blank when fetching file
task");
List sourceEntities =
sourceMapper.selectByAgentIpOrCluster(needAddStatusList,
Lists.newArrayList(SourceType.FILE), agentIp,
agentClusterName);
List fileTasks = Lists.newArrayList();
for (StreamSourceEntity sourceEntity : sourceEntities) {
-FileSourceDTO fileSourceDTO =
FileSourceDTO.getFromJson(sourceEntity.getExtParams());
-final String destIp = sourceEntity.getAgentIp();
-final String destClusterName = sourceEntity.getInlongClusterName();
-
-// Use ip directly if it is not empty
-if (StringUtils.isNotBlank(destIp)) {
-if (destIp.equals(agentIp)) {
-int op = getOp(sourceEntity.getStatus());
-int nextStatus = getNextStatus(sourceEntity.getStatus());
-sourceEntity.setUuid(uuid);
-sourceEntity.setStatus(nextStatus);
-if (sourceMapper.updateByPrimaryKeySelective(sourceEntity)
== 1) {
-sourceEntity.setVersion(sourceEntity.getVersion() + 1);
-fileTasks.add(getDataConfig(sourceEntity, op));
-}
-}
+DataConfig taskConfig = getFileTaskFromEntity(taskRequest,
sourceEntity);
+if (taskConfig == null) {
continue;
}
-
-// Cluster name is not blank, split subtask if necessary
-// The template task's id is assigned to the subtask's template id
field
-if (StringUtils.isNotBlank(destClusterName) &&
destClusterName.equals(agentClusterName)
-&& Objects.isNull(sourceEntity.getTemplateId())) {
-
-// Is the task already fetched by this agent ?
-List subSources =
sourceMapper.selectByTemplateId(sourceEntity.getId());
-if (subSources.stream().anyMatch(subSource ->
subSource.getAgentIp().equals(agentIp))) {
-continue;
-}
-
-// If not, clone a sub task for the new agent
-// Note that a new source name with random suffix is generated
to adhere to the unique constraint
-StreamSourceEntity fileEntity =
CommonBeanUtils.copyProperties(sourceEntity, StreamSourceEntity::new);
-FileSourceDTO childFileSourceDTO =
CommonBeanUtils.copyProperties(fileSourceDTO, FileSourceDTO::new);
-
fileEntity.setExtParams(JsonUtils.toJsonString(childFileSourceDTO));
-fileEntity.setAgentIp(agentIp);
-fileEntity.setUui
[GitHub] [inlong] healchow merged pull request #5878: [INLONG-5876][Manager] Remove the check of data separator
healchow merged PR #5878: URL: https://github.com/apache/inlong/pull/5878 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-5876][Manager] Remove the check of data separator (#5878)
This is an automated email from the ASF dual-hosted git repository.
healchow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 72f5ef7af [INLONG-5876][Manager] Remove the check of data separator
(#5878)
72f5ef7af is described below
commit 72f5ef7afc79586cdf117c2297a4f01280823baa
Author: emhui <111486498+e-m...@users.noreply.github.com>
AuthorDate: Wed Sep 14 15:41:51 2022 +0800
[INLONG-5876][Manager] Remove the check of data separator (#5878)
---
.../src/test/java/org/apache/inlong/manager/client/BaseExample.java | 5 ++---
.../src/test/java/org/apache/inlong/manager/client/ut/BaseTest.java | 5 ++---
.../java/org/apache/inlong/manager/common/enums/DataSeparator.java | 3 ++-
.../java/org/apache/inlong/manager/pojo/sink/hive/HiveSink.java | 2 +-
.../java/org/apache/inlong/manager/pojo/sink/hive/HiveSinkDTO.java | 2 +-
.../org/apache/inlong/manager/pojo/sink/hive/HiveSinkRequest.java | 2 +-
.../org/apache/inlong/manager/pojo/sort/util/ExtractNodeUtils.java | 5 +
.../apache/inlong/manager/pojo/source/autopush/AutoPushSource.java | 4 ++--
.../inlong/manager/pojo/source/autopush/AutoPushSourceDTO.java | 4 ++--
.../inlong/manager/pojo/source/autopush/AutoPushSourceRequest.java | 6 +++---
.../org/apache/inlong/manager/pojo/source/kafka/KafkaSource.java| 4 ++--
.../org/apache/inlong/manager/pojo/source/kafka/KafkaSourceDTO.java | 4 ++--
.../apache/inlong/manager/pojo/source/kafka/KafkaSourceRequest.java | 6 +++---
.../org/apache/inlong/manager/pojo/source/pulsar/PulsarSource.java | 4 ++--
.../apache/inlong/manager/pojo/source/pulsar/PulsarSourceDTO.java | 4 ++--
.../inlong/manager/pojo/source/pulsar/PulsarSourceRequest.java | 6 +++---
.../org/apache/inlong/manager/pojo/stream/InlongStreamInfo.java | 4 ++--
.../org/apache/inlong/manager/pojo/stream/InlongStreamRequest.java | 6 +++---
.../org/apache/inlong/manager/pojo/stream/InlongStreamResponse.java | 4 ++--
.../inlong/manager/service/source/pulsar/PulsarSourceOperator.java | 3 +--
.../java/org/apache/inlong/manager/service/ServiceBaseTest.java | 2 +-
.../apache/inlong/manager/service/sort/DisableZkForSortTest.java| 2 +-
.../manager-test/src/main/resources/h2/apache_inlong_manager.sql| 2 +-
inlong-manager/manager-web/sql/apache_inlong_manager.sql| 2 +-
24 files changed, 43 insertions(+), 48 deletions(-)
diff --git
a/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/BaseExample.java
b/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/BaseExample.java
index a8ca418d2..1abc7ae40 100644
---
a/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/BaseExample.java
+++
b/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/BaseExample.java
@@ -20,7 +20,6 @@ package org.apache.inlong.manager.client;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.Data;
import org.apache.inlong.manager.common.auth.DefaultAuthentication;
-import org.apache.inlong.manager.common.enums.DataSeparator;
import org.apache.inlong.manager.common.enums.FieldType;
import org.apache.inlong.manager.common.enums.FileFormat;
import org.apache.inlong.manager.common.consts.InlongConstants;
@@ -109,7 +108,7 @@ public class BaseExample {
streamInfo.setName(this.getStreamId());
streamInfo.setInlongStreamId(this.getStreamId());
streamInfo.setDataEncoding(StandardCharsets.UTF_8.toString());
-streamInfo.setDataSeparator(DataSeparator.VERTICAL_BAR.getSeparator());
+streamInfo.setDataSeparator("|");
// if you need strictly order for data, set to 1
streamInfo.setSyncSend(InlongConstants.SYNC_SEND);
streamInfo.setMqResource(this.getTopic());
@@ -126,7 +125,7 @@ public class BaseExample {
hiveSink.setAuthentication(new DefaultAuthentication("hive", "hive"));
hiveSink.setDataEncoding(StandardCharsets.UTF_8.toString());
hiveSink.setFileFormat(FileFormat.TextFile.name());
-hiveSink.setDataSeparator(DataSeparator.VERTICAL_BAR.getSeparator());
+hiveSink.setDataSeparator("|");
hiveSink.setDataPath("hdfs://{ip:port}/usr/hive/warehouse/{db.name}");
hiveSink.setHiveConfDir("{hive.conf.dir}");
diff --git
a/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/ut/BaseTest.java
b/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/ut/BaseTest.java
index 015591878..4c8c122da 100644
---
a/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/ut/BaseTest.java
+++
b/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/ut/BaseTest.java
@@ -24,7 +24,6 @@ import
org.apache.inlong.manager.clien
[inlong] branch release-1.3.0 updated: [INLONG-5885][Manager] Refactor the task issue logic to simply code complexity (#5886)
This is an automated email from the ASF dual-hosted git repository.
healchow pushed a commit to branch release-1.3.0
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/release-1.3.0 by this push:
new 649d09508 [INLONG-5885][Manager] Refactor the task issue logic to
simply code complexity (#5886)
649d09508 is described below
commit 649d0950805799efd8922ec41e999e0b08cd6235
Author: woofyzhao <490467...@qq.com>
AuthorDate: Wed Sep 14 15:31:55 2022 +0800
[INLONG-5885][Manager] Refactor the task issue logic to simply code
complexity (#5886)
Co-authored-by: healchow
---
.../service/core/impl/AgentServiceImpl.java| 102 +++--
.../inlong/manager/service/ServiceBaseTest.java| 1 +
.../service/core/impl/AgentServiceTest.java| 68 ++
3 files changed, 123 insertions(+), 48 deletions(-)
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/AgentServiceImpl.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/AgentServiceImpl.java
index 2258581a9..111035c87 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/AgentServiceImpl.java
+++
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/AgentServiceImpl.java
@@ -97,7 +97,7 @@ public class AgentServiceImpl implements AgentService {
// Update task status, other tasks with status 20x will change to 30x
in next request
if (CollectionUtils.isEmpty(request.getCommandInfo())) {
-LOGGER.warn("task result was empty, just return");
+LOGGER.info("task result was empty in request: {}, just return",
request);
return;
}
for (CommandEntity command : request.getCommandInfo()) {
@@ -211,61 +211,17 @@ public class AgentServiceImpl implements AgentService {
}
final String agentIp = taskRequest.getAgentIp();
final String agentClusterName = taskRequest.getClusterName();
-final String uuid = taskRequest.getUuid();
Preconditions.checkTrue(StringUtils.isNotBlank(agentIp) ||
StringUtils.isNotBlank(agentClusterName),
"both agent ip and cluster name are blank when fetching file
task");
List sourceEntities =
sourceMapper.selectByAgentIpOrCluster(needAddStatusList,
Lists.newArrayList(SourceType.FILE), agentIp,
agentClusterName);
List fileTasks = Lists.newArrayList();
for (StreamSourceEntity sourceEntity : sourceEntities) {
-FileSourceDTO fileSourceDTO =
FileSourceDTO.getFromJson(sourceEntity.getExtParams());
-final String destIp = sourceEntity.getAgentIp();
-final String destClusterName = sourceEntity.getInlongClusterName();
-
-// Use ip directly if it is not empty
-if (StringUtils.isNotBlank(destIp)) {
-if (destIp.equals(agentIp)) {
-int op = getOp(sourceEntity.getStatus());
-int nextStatus = getNextStatus(sourceEntity.getStatus());
-sourceEntity.setUuid(uuid);
-sourceEntity.setStatus(nextStatus);
-if (sourceMapper.updateByPrimaryKeySelective(sourceEntity)
== 1) {
-sourceEntity.setVersion(sourceEntity.getVersion() + 1);
-fileTasks.add(getDataConfig(sourceEntity, op));
-}
-}
+DataConfig taskConfig = getFileTaskFromEntity(taskRequest,
sourceEntity);
+if (taskConfig == null) {
continue;
}
-
-// Cluster name is not blank, split subtask if necessary
-// The template task's id is assigned to the subtask's template id
field
-if (StringUtils.isNotBlank(destClusterName) &&
destClusterName.equals(agentClusterName)
-&& Objects.isNull(sourceEntity.getTemplateId())) {
-
-// Is the task already fetched by this agent ?
-List subSources =
sourceMapper.selectByTemplateId(sourceEntity.getId());
-if (subSources.stream().anyMatch(subSource ->
subSource.getAgentIp().equals(agentIp))) {
-continue;
-}
-
-// If not, clone a sub task for the new agent
-// Note that a new source name with random suffix is generated
to adhere to the unique constraint
-StreamSourceEntity fileEntity =
CommonBeanUtils.copyProperties(sourceEntity, StreamSourceEntity::new);
-FileSourceDTO childFileSourceDTO =
CommonBeanUtils.copyProperties(fileSourceDTO, FileSourceDTO::new);
-
fileEntity.setExtParams(JsonUtils.toJsonString(childFileSourceDTO));
-fileEntity.setAgentIp(agentIp);
-fileEn
[inlong] branch release-1.3.0 updated: [INLONG-5876][Manager] Remove the check of data separator (#5878)
This is an automated email from the ASF dual-hosted git repository.
healchow pushed a commit to branch release-1.3.0
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/release-1.3.0 by this push:
new 006489dbf [INLONG-5876][Manager] Remove the check of data separator
(#5878)
006489dbf is described below
commit 006489dbf135e9f8758c190ff93c2209324613fc
Author: emhui <111486498+e-m...@users.noreply.github.com>
AuthorDate: Wed Sep 14 15:41:51 2022 +0800
[INLONG-5876][Manager] Remove the check of data separator (#5878)
---
.../src/test/java/org/apache/inlong/manager/client/BaseExample.java | 5 ++---
.../src/test/java/org/apache/inlong/manager/client/ut/BaseTest.java | 5 ++---
.../java/org/apache/inlong/manager/common/enums/DataSeparator.java | 3 ++-
.../java/org/apache/inlong/manager/pojo/sink/hive/HiveSink.java | 2 +-
.../java/org/apache/inlong/manager/pojo/sink/hive/HiveSinkDTO.java | 2 +-
.../org/apache/inlong/manager/pojo/sink/hive/HiveSinkRequest.java | 2 +-
.../org/apache/inlong/manager/pojo/sort/util/ExtractNodeUtils.java | 5 +
.../apache/inlong/manager/pojo/source/autopush/AutoPushSource.java | 4 ++--
.../inlong/manager/pojo/source/autopush/AutoPushSourceDTO.java | 4 ++--
.../inlong/manager/pojo/source/autopush/AutoPushSourceRequest.java | 6 +++---
.../org/apache/inlong/manager/pojo/source/kafka/KafkaSource.java| 4 ++--
.../org/apache/inlong/manager/pojo/source/kafka/KafkaSourceDTO.java | 4 ++--
.../apache/inlong/manager/pojo/source/kafka/KafkaSourceRequest.java | 6 +++---
.../org/apache/inlong/manager/pojo/source/pulsar/PulsarSource.java | 4 ++--
.../apache/inlong/manager/pojo/source/pulsar/PulsarSourceDTO.java | 4 ++--
.../inlong/manager/pojo/source/pulsar/PulsarSourceRequest.java | 6 +++---
.../org/apache/inlong/manager/pojo/stream/InlongStreamInfo.java | 4 ++--
.../org/apache/inlong/manager/pojo/stream/InlongStreamRequest.java | 6 +++---
.../org/apache/inlong/manager/pojo/stream/InlongStreamResponse.java | 4 ++--
.../inlong/manager/service/source/pulsar/PulsarSourceOperator.java | 3 +--
.../java/org/apache/inlong/manager/service/ServiceBaseTest.java | 2 +-
.../apache/inlong/manager/service/sort/DisableZkForSortTest.java| 2 +-
.../manager-test/src/main/resources/h2/apache_inlong_manager.sql| 2 +-
inlong-manager/manager-web/sql/apache_inlong_manager.sql| 2 +-
24 files changed, 43 insertions(+), 48 deletions(-)
diff --git
a/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/BaseExample.java
b/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/BaseExample.java
index a8ca418d2..1abc7ae40 100644
---
a/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/BaseExample.java
+++
b/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/BaseExample.java
@@ -20,7 +20,6 @@ package org.apache.inlong.manager.client;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.Data;
import org.apache.inlong.manager.common.auth.DefaultAuthentication;
-import org.apache.inlong.manager.common.enums.DataSeparator;
import org.apache.inlong.manager.common.enums.FieldType;
import org.apache.inlong.manager.common.enums.FileFormat;
import org.apache.inlong.manager.common.consts.InlongConstants;
@@ -109,7 +108,7 @@ public class BaseExample {
streamInfo.setName(this.getStreamId());
streamInfo.setInlongStreamId(this.getStreamId());
streamInfo.setDataEncoding(StandardCharsets.UTF_8.toString());
-streamInfo.setDataSeparator(DataSeparator.VERTICAL_BAR.getSeparator());
+streamInfo.setDataSeparator("|");
// if you need strictly order for data, set to 1
streamInfo.setSyncSend(InlongConstants.SYNC_SEND);
streamInfo.setMqResource(this.getTopic());
@@ -126,7 +125,7 @@ public class BaseExample {
hiveSink.setAuthentication(new DefaultAuthentication("hive", "hive"));
hiveSink.setDataEncoding(StandardCharsets.UTF_8.toString());
hiveSink.setFileFormat(FileFormat.TextFile.name());
-hiveSink.setDataSeparator(DataSeparator.VERTICAL_BAR.getSeparator());
+hiveSink.setDataSeparator("|");
hiveSink.setDataPath("hdfs://{ip:port}/usr/hive/warehouse/{db.name}");
hiveSink.setHiveConfDir("{hive.conf.dir}");
diff --git
a/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/ut/BaseTest.java
b/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/ut/BaseTest.java
index 015591878..4c8c122da 100644
---
a/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/ut/BaseTest.java
+++
b/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/ut/BaseTest.java
@@ -24,7 +24,6 @@ import
org.apache.inlong
[GitHub] [inlong] leosanqing commented on pull request #5866: [INLONG-5865][Manager] Support maintenance of inlong group
leosanqing commented on PR #5866: URL: https://github.com/apache/inlong/pull/5866#issuecomment-1246384711 #5843 ，Please fix this first, the two problems are similar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] dockerzhang opened a new issue, #553: [Improve] Optimize the standalone deployment guide
dockerzhang opened a new issue, #553: URL: https://github.com/apache/inlong-website/issues/553 ### Description _No response_ ### Are you willing to submit PR? - [X] Yes, I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] dockerzhang opened a new pull request, #554: [INLONG-553][Doc] Optimize the standalone deployment guide
dockerzhang opened a new pull request, #554: URL: https://github.com/apache/inlong-website/pull/554 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #553 ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve.* ### Modifications *Describe the modifications you've done.* ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a followup issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang commented on issue #5872: [Improve]建议增加standalone模式启动建议
dockerzhang commented on issue #5872: URL: https://github.com/apache/inlong/issues/5872#issuecomment-1246478561 please change the `local_ip=` to actual IP in `conf/inlong.conf`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang opened a new pull request, #5891: [INLONG-5890][Script] Add tcp protocol when connecting to MySQL
dockerzhang opened a new pull request, #5891: URL: https://github.com/apache/inlong/pull/5891 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #5890 ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve?* ### Modifications *Describe the modifications you've done.* ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] dockerzhang merged pull request #554: [INLONG-553][Doc] Optimize the standalone deployment guide
dockerzhang merged PR #554: URL: https://github.com/apache/inlong-website/pull/554 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] dockerzhang closed issue #553: [Improve] Optimize the standalone deployment guide
dockerzhang closed issue #553: [Improve] Optimize the standalone deployment guide URL: https://github.com/apache/inlong-website/issues/553 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong-website] branch master updated: [INLONG-553][Doc] Optimize the standalone deployment guide (#554)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/inlong-website.git The following commit(s) were added to refs/heads/master by this push: new 1af9b06e74 [INLONG-553][Doc] Optimize the standalone deployment guide (#554) 1af9b06e74 is described below commit 1af9b06e746a3c72d127afbf0277ff551140debc Author: Charles Zhang AuthorDate: Wed Sep 14 17:23:56 2022 +0800 [INLONG-553][Doc] Optimize the standalone deployment guide (#554) --- docs/deployment/standalone.md| 20 +--- .../current/deployment/standalone.md | 20 +--- .../version-1.3.0/deployment/standalone.md | 20 +--- .../version-1.3.0/deployment/standalone.md | 20 +--- 4 files changed, 68 insertions(+), 12 deletions(-) diff --git a/docs/deployment/standalone.md b/docs/deployment/standalone.md index 1d58b156a3..7a08b04ffd 100644 --- a/docs/deployment/standalone.md +++ b/docs/deployment/standalone.md @@ -5,7 +5,7 @@ sidebar_position: 1 ## Environment Requirements - MySQL 5.7+ -- Flink 1.13.5 +- [Flink 1.13.5](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/try-flink/local_installation/) - [Docker](https://docs.docker.com/engine/install/) 19.03.1+ ## Prepare Message Queue @@ -16,7 +16,7 @@ InLong Support the following Message Queue services now, you can choose one of t ## Download the Binary Package You can get binary package from [Download Page](https://inlong.apache.org/download) ,or you can build the InLong refer to [How to Build](quick_start/how_to_build.md). -Extract `apache-inlong-[version]-bin.tar.gz` and `apache-inlong-[version]-sort-connectors.tar.gz`, and make sure the `inlong-sort/connectors/` directory contains Connectors. +Extract `apache-inlong-[version]-bin.tar.gz` and `apache-inlong-[version]-sort-connectors.tar.gz`, and make sure the `inlong-sort/connectors/` directory contains `sort-connector-[type]-[version].jar`. ## DB Dependencies - If the backend database is MySQL, please download [mysql-connector-java-8.0.27.jar](https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.27/mysql-connector-java-8.0.27.jar) and put it into the following directories: @@ -29,7 +29,21 @@ inlong-tubemq-manager/lib/ - If the backend database is PostgreSQL, there's no need for additional dependencies. ## Configure -In `conf/inlong.conf`, configure the parameters according to the actual situation. +In `conf/inlong.conf`, configure the parameters according to the actual situation, mainly include: +```shell +# local IP +local_ip= +# Configure MySQL +spring_datasource_hostname= +spring_datasource_port=3306 +spring_datasource_username=root +spring_datasource_password=inlong +# Configure Pulsar or TubeMQ Address +# the REST server address for Flink +flink_rest_address= +# the REST server Port for Flink +flink_rest_port=8081 +``` ## Start ```shell diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/deployment/standalone.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/deployment/standalone.md index f3511c098d..5fc4ac9c9c 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/deployment/standalone.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/deployment/standalone.md @@ -4,7 +4,7 @@ sidebar_position: 1 --- ## 环境要求 - MySQL 5.7+ -- Flink 1.13.5 +- [Flink 1.13.5](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/try-flink/local_installation/) - [Docker](https://docs.docker.com/engine/install/) 19.03.1+ ## 准备消息队列 @@ -15,7 +15,7 @@ InLong 当前支持以下消息队列，根据使用情况**选择其一**即可 ## 下载安装包 可以从 [下载页面](https://inlong.apache.org/download) 获取二进制包，或参考 [如何编译](quick_start/how_to_build.md) 编译需要的版本。 -解压 `apache-inlong-[version]-bin.tar.gz` 和 `apache-inlong-[version]-sort-connectors.tar.gz`，并确保 `inlong-sort/connectors/` 目录包含 Connectors。 +解压 `apache-inlong-[version]-bin.tar.gz` 和 `apache-inlong-[version]-sort-connectors.tar.gz`，并确保 `inlong-sort/connectors/` 目录包含 `sort-connector-[type]-[version].jar`。 ## DB 依赖 - 如果后端连接 MySQL 数据库，请下载 [mysql-connector-java-8.0.27.jar](https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.27/mysql-connector-java-8.0.27.jar), 并将其放入以下目录： @@ -29,7 +29,21 @@ inlong-tubemq-manager/lib/ - 如果后端连接 PostgreSQL 数据库，不需要引入额外依赖。 ## 配置 -在`conf/inlong.conf`文件中根据实际情况配置参数。 +在`conf/inlong.conf`文件中根据实际情况配置参数，主要包括： +```shell +# 本地机器 IP +local_ip= +# MySQL 配置 +spring_datasource_hostname= +spring_datasource_port=3306 +spring_datasource_username=root +spring_datasource_password=inlong +# 配置 Pulsar 或者 TubeMQ 地址 +# Flink REST server 地址 +flink_rest_address= +# Flink REST server 端口 +flink_rest_port=8081 +``` ## 启动 ```shell diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-1.3.0/deployment/standalone.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-1.3.0/deployment/standalone
[GitHub] [inlong-website] dockerzhang opened a new issue, #555: [Bug] the title is error for Doris Load Node
dockerzhang opened a new issue, #555: URL: https://github.com/apache/inlong-website/issues/555 ### What happened  ### What you expected to happen the title is error for Doris Load Node ### How to reproduce the title is error for Doris Load Node ### Environment _No response_ ### Are you willing to submit PR? - [X] Yes, I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] dockerzhang opened a new pull request, #556: [INLONG-555][Doc] Change the title for Doris
dockerzhang opened a new pull request, #556: URL: https://github.com/apache/inlong-website/pull/556 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #555 ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve.* ### Modifications *Describe the modifications you've done.* ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a followup issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] dockerzhang merged pull request #556: [INLONG-555][Doc] Change the title for Doris
dockerzhang merged PR #556: URL: https://github.com/apache/inlong-website/pull/556 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong-website] branch master updated: [INLONG-555][Doc] Change the title for Doris (#556)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/inlong-website.git The following commit(s) were added to refs/heads/master by this push: new 4b0e226692 [INLONG-555][Doc] Change the title for Doris (#556) 4b0e226692 is described below commit 4b0e226692aa5bdc726fd6aec4e381c141fc4123 Author: Charles Zhang AuthorDate: Wed Sep 14 18:29:16 2022 +0800 [INLONG-555][Doc] Change the title for Doris (#556) --- docs/data_node/load_node/doris.md | 2 +- .../docusaurus-plugin-content-docs/current/data_node/load_node/doris.md | 2 +- .../version-1.3.0/data_node/load_node/doris.md | 2 +- versioned_docs/version-1.3.0/data_node/load_node/doris.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/data_node/load_node/doris.md b/docs/data_node/load_node/doris.md index b2fb3b64bf..2facf8e0dd 100644 --- a/docs/data_node/load_node/doris.md +++ b/docs/data_node/load_node/doris.md @@ -1,5 +1,5 @@ --- -title: ClickHouse +title: Doris sidebar_position: 16 --- diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/load_node/doris.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/load_node/doris.md index 78760f8975..8298516f6f 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/load_node/doris.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/load_node/doris.md @@ -1,5 +1,5 @@ --- -title: ClickHouse +title: Doris sidebar_position: 16 --- diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-1.3.0/data_node/load_node/doris.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-1.3.0/data_node/load_node/doris.md index 78760f8975..8298516f6f 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-1.3.0/data_node/load_node/doris.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-1.3.0/data_node/load_node/doris.md @@ -1,5 +1,5 @@ --- -title: ClickHouse +title: Doris sidebar_position: 16 --- diff --git a/versioned_docs/version-1.3.0/data_node/load_node/doris.md b/versioned_docs/version-1.3.0/data_node/load_node/doris.md index b2fb3b64bf..2facf8e0dd 100644 --- a/versioned_docs/version-1.3.0/data_node/load_node/doris.md +++ b/versioned_docs/version-1.3.0/data_node/load_node/doris.md @@ -1,5 +1,5 @@ --- -title: ClickHouse +title: Doris sidebar_position: 16 ---
[GitHub] [inlong-website] dockerzhang closed issue #555: [Bug] the title is error for Doris Load Node
dockerzhang closed issue #555: [Bug] the title is error for Doris Load Node URL: https://github.com/apache/inlong-website/issues/555 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] EMsnap opened a new pull request, #5892: [INLONG-5889][Sort] fix mysql load node jdbc for RCE Vulnerability
EMsnap opened a new pull request, #5892: URL: https://github.com/apache/inlong/pull/5892 - Fixes #5889 ### Motivation fix mysql load node jdbc for RCE Vulnerability JDBC URL of MySQL was vulnerable to security attacks. See https://su18.org/post/jdbc-connection-url-attack, or https://pyn3rd.github.io/2022/06/06/Make-JDBC-Attacks-Brillian-Again-I.### Modifications fix mysql load node jdbc for RCE Vulnerability ### Verifying this change add unit test ### Documentation - Does this pull request introduce a new feature? (no) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] bluewang opened a new issue, #557: [Improve] Optimize the navigation bar document column
bluewang opened a new issue, #557: URL: https://github.com/apache/inlong-website/issues/557 ### Description _No response_ ### Are you willing to submit PR? - [X] Yes, I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] bluewang opened a new pull request, #558: [INLONG-557] Optimize the navigation bar document column
bluewang opened a new pull request, #558: URL: https://github.com/apache/inlong-website/pull/558 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes https://github.com/apache/inlong-website/issues/557 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] healchow opened a new pull request, #5893: [INLONG-5881][Manager] Fix the vulnerability for the MySQL JDBC URL (addendum)
healchow opened a new pull request, #5893: URL: https://github.com/apache/inlong/pull/5893 ### Prepare a Pull Request - Fixes #5881 ### Motivation Fix the vulnerability for the MySQL JDBC URL. ### Verifying this change - [x] This change added tests and can be verified as follows: - org.apache.inlong.manager.pojo.sink.mysql.MySQLSinkDTOTest -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #5891: [INLONG-5890][Script] Add tcp protocol when connecting to MySQL
dockerzhang merged PR #5891: URL: https://github.com/apache/inlong/pull/5891 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated (72f5ef7af -> 831bb26a3)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong.git from 72f5ef7af [INLONG-5876][Manager] Remove the check of data separator (#5878) add 831bb26a3 [INLONG-5890][Script] Add tcp protocol when connecting to MySQL (#5891) No new revisions were added by this update. Summary of changes: bin/inlong-daemon | 8 1 file changed, 4 insertions(+), 4 deletions(-)
[GitHub] [inlong] EMsnap closed pull request #5892: [INLONG-5889][Sort] Fix the RCE vulnerability for MySQL node JDBC URL
EMsnap closed pull request #5892: [INLONG-5889][Sort] Fix the RCE vulnerability for MySQL node JDBC URL URL: https://github.com/apache/inlong/pull/5892 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang opened a new pull request, #5895: [INLONG-5894][Dashboard] Remove the password required restrict for test/development clickhouse cluster
dockerzhang opened a new pull request, #5895: URL: https://github.com/apache/inlong/pull/5895 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #5894 ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve?* ### Modifications *Describe the modifications you've done.* ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] EMsnap opened a new pull request, #5896: [INLONG-5889][Sort] Fix MySQL Node JDBC Url for RCE vulnerability
EMsnap opened a new pull request, #5896: URL: https://github.com/apache/inlong/pull/5896 - Fixes #5889 ### Motivation Fix MySQL Node JDBC Url for RCE vulnerability ### Modifications Fix MySQL Node JDBC Url for RCE vulnerability ### Verifying this change run ut ### Documentation - Does this pull request introduce a new feature? (no) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] healchow merged pull request #5893: [INLONG-5881][Manager] Fix the vulnerability for the MySQL JDBC URL (addendum)
healchow merged PR #5893: URL: https://github.com/apache/inlong/pull/5893 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-5881][Manager] Fix the vulnerability for the MySQL JDBC URL (addendum) (#5893)
This is an automated email from the ASF dual-hosted git repository.
healchow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 0c2e9fe83 [INLONG-5881][Manager] Fix the vulnerability for the MySQL
JDBC URL (addendum) (#5893)
0c2e9fe83 is described below
commit 0c2e9fe8376e52c754d9932542014632d326a48a
Author: healchow
AuthorDate: Wed Sep 14 20:20:42 2022 +0800
[INLONG-5881][Manager] Fix the vulnerability for the MySQL JDBC URL
(addendum) (#5893)
---
.../manager/pojo/sink/mysql/MySQLSinkDTO.java | 24 --
.../manager/pojo/sink/mysql/MySQLSinkDTOTest.java | 17 +++
2 files changed, 15 insertions(+), 26 deletions(-)
diff --git
a/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sink/mysql/MySQLSinkDTO.java
b/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sink/mysql/MySQLSinkDTO.java
index 070456809..e2f206212 100644
---
a/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sink/mysql/MySQLSinkDTO.java
+++
b/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sink/mysql/MySQLSinkDTO.java
@@ -45,13 +45,11 @@ import java.util.Map;
@AllArgsConstructor
public class MySQLSinkDTO {
-@VisibleForTesting
-protected static final char SYMBOL = '&';
/**
* The sensitive param may lead the attack.
*/
-@VisibleForTesting
-protected static final String SENSITIVE_PARAM = "autoDeserialize=true";
+private static final String SENSITIVE_PARAM_TRUE = "autoDeserialize=true";
+private static final String SENSITIVE_PARAM_FALSE =
"autoDeserialize=false";
private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
private static final Logger LOGGER =
LoggerFactory.getLogger(MySQLSinkDTO.class);
@@ -181,23 +179,17 @@ public class MySQLSinkDTO {
*/
@VisibleForTesting
protected static String filterSensitive(String url) {
-if (StringUtils.isBlank(url) || !url.contains(SENSITIVE_PARAM)) {
-LOGGER.info("string was empty or not contains sensitive for [{}]",
url);
+if (StringUtils.isBlank(url)) {
return url;
}
-String originUrl = url;
-int index = url.indexOf(SENSITIVE_PARAM);
-String tmp = SENSITIVE_PARAM;
-if (index == 0) {
-tmp = tmp + SYMBOL;
-} else if (url.charAt(index - 1) == SYMBOL) {
-tmp = SYMBOL + tmp;
+String resultUrl = url;
+if (StringUtils.containsIgnoreCase(url, SENSITIVE_PARAM_TRUE)) {
+resultUrl = StringUtils.replaceIgnoreCase(url,
SENSITIVE_PARAM_TRUE, SENSITIVE_PARAM_FALSE);
}
-url = url.replace(tmp, "");
-LOGGER.debug("the origin url [{}] was filter to: [{}]", originUrl,
url);
-return url;
+LOGGER.debug("the origin url [{}] was replaced to: [{}]", url,
resultUrl);
+return resultUrl;
}
}
diff --git
a/inlong-manager/manager-pojo/src/test/java/org/apache/inlong/manager/pojo/sink/mysql/MySQLSinkDTOTest.java
b/inlong-manager/manager-pojo/src/test/java/org/apache/inlong/manager/pojo/sink/mysql/MySQLSinkDTOTest.java
index fabd6842d..82d6129b4 100644
---
a/inlong-manager/manager-pojo/src/test/java/org/apache/inlong/manager/pojo/sink/mysql/MySQLSinkDTOTest.java
+++
b/inlong-manager/manager-pojo/src/test/java/org/apache/inlong/manager/pojo/sink/mysql/MySQLSinkDTOTest.java
@@ -20,28 +20,25 @@ package org.apache.inlong.manager.pojo.sink.mysql;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
-import static
org.apache.inlong.manager.pojo.sink.mysql.MySQLSinkDTO.SENSITIVE_PARAM;
-import static org.apache.inlong.manager.pojo.sink.mysql.MySQLSinkDTO.SYMBOL;
-
/**
* Test for {@link MySQLSinkDTO}
*/
public class MySQLSinkDTOTest {
@Test
-public void testFilterOther() {
+public void testFilterSensitive() {
// the sensitive params at the first
-String originUrl = MySQLSinkDTO.filterSensitive(SENSITIVE_PARAM +
SYMBOL + "autoReconnect=true");
-Assertions.assertEquals("autoReconnect=true", originUrl);
+String originUrl =
MySQLSinkDTO.filterSensitive("autoDeserialize=TRue&autoReconnect=true");
+Assertions.assertEquals("autoDeserialize=false&autoReconnect=true",
originUrl);
// the sensitive params at the end
-originUrl = MySQLSinkDTO.filterSensitive("autoReconnect=true" + SYMBOL
+ SENSITIVE_PARAM);
-Assertions.assertEquals("autoReconnect=true", originUrl);
+originUrl =
MySQLSinkDTO.filterSensitive("autoReconnect=true&autoDeserialize=trUE");
+Assertions.assertEquals("autoReconnect=true&autoDeserialize=false",
originUrl);
// the sensitive params in the middle
originUrl = MySQLSinkDTO.filterSensitive(
-"useSSL=false" + SY
[GitHub] [inlong] healchow merged pull request #5895: [INLONG-5894][Dashboard] Remove the password required restrict for test/development clickhouse cluster
healchow merged PR #5895: URL: https://github.com/apache/inlong/pull/5895 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated (0c2e9fe83 -> c3f6d11c1)
This is an automated email from the ASF dual-hosted git repository. healchow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong.git from 0c2e9fe83 [INLONG-5881][Manager] Fix the vulnerability for the MySQL JDBC URL (addendum) (#5893) add c3f6d11c1 [INLONG-5894][Dashboard] Remove the password required restrict for ClickHouse cluster (#5895) No new revisions were added by this update. Summary of changes: inlong-dashboard/src/metas/sinks/clickhouse.tsx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[GitHub] [inlong-website] EMsnap opened a new pull request, #559: [INLONG-549][Bug] fix release doc bug on release-1.3.0
EMsnap opened a new pull request, #559: URL: https://github.com/apache/inlong-website/pull/559 - Fixes #549 ### Motivation [INLONG-549][Bug] fix release doc bug on release-1.3.0 ### Modifications [INLONG-549][Bug] fix release doc bug on release-1.3.0 ### Verifying this change -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #5896: [INLONG-5889][Sort] Fix MySQL Node JDBC Url For RCE Vulnerability
gong commented on code in PR #5896:
URL: https://github.com/apache/inlong/pull/5896#discussion_r970766182
##
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/util/JdbcUrlUtils.java:
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.util;
+
+import static org.apache.inlong.sort.base.Constants.AUTO_DESERIALIZE_FALSE;
+import static org.apache.inlong.sort.base.Constants.AUTO_DESERIALIZE_TRUE;
+
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * utils for jdbc url
+ */
+public class JdbcUrlUtils {
+
+private static final Logger LOG =
LoggerFactory.getLogger(JdbcUrlUtils.class);
+
+/**
+ * see https://su18.org/post/jdbc-connection-url-attack/
+ * @param url
+ * @return url after filtering out the invalid property
+ */
+public static String replaceInvalidUrlProperty(String url) {
+if (StringUtils.containsIgnoreCase(url, AUTO_DESERIALIZE_TRUE)) {
+LOG.info("url {} contains invalid property {}, replace it to {}",
url,
Review Comment:
Maybe, LOG.warn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #5896: [INLONG-5889][Sort] Fix MySQL Node JDBC Url For RCE Vulnerability
gong commented on code in PR #5896:
URL: https://github.com/apache/inlong/pull/5896#discussion_r970767201
##
inlong-sort/sort-connectors/jdbc/src/main/java/org/apache/inlong/sort/jdbc/table/JdbcDynamicTableFactory.java:
##
@@ -219,6 +221,7 @@ public DynamicTableSource createDynamicTableSource(Context
context) {
private JdbcOptions getJdbcOptions(ReadableConfig readableConfig) {
final String url = readableConfig.get(URL);
+JdbcUrlUtils.replaceInvalidUrlProperty(url);
Review Comment:
`JdbcUrlUtils.replaceInvalidUrlProperty(url);` return value should be used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[inlong-website] branch master updated: [INLONG-557][Website] Optimize the navigation bar document column (#558)
This is an automated email from the ASF dual-hosted git repository.
healchow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new 1ad798fc40 [INLONG-557][Website] Optimize the navigation bar document
column (#558)
1ad798fc40 is described below
commit 1ad798fc407897e3091284847c7d5c56be1c1411
Author: Lizhen <88174078+bluew...@users.noreply.github.com>
AuthorDate: Wed Sep 14 21:34:16 2022 +0800
[INLONG-557][Website] Optimize the navigation bar document column (#558)
---
docusaurus.config.js | 16
1 file changed, 16 deletions(-)
diff --git a/docusaurus.config.js b/docusaurus.config.js
index bcd9e20a1b..ba2c0cf585 100644
--- a/docusaurus.config.js
+++ b/docusaurus.config.js
@@ -83,22 +83,6 @@ const darkCodeTheme =
require('prism-react-renderer/themes/dracula');
label: "1.2.0",
to: "/docs/1.2.0/introduction",
},
- {
-label: "1.1.0",
-to: "/docs/1.1.0/introduction",
- },
- {
-label: "1.0.0",
-to: "/docs/1.0.0/introduction",
- },
- {
-label: "0.12.0",
-to: "/docs/0.12.0/introduction",
- },
- {
-label: "0.11.0",
-to: "/docs/0.11.0/user_guide/quick_start",
- },
{
label: "All versions",
to: "/versions/",
[GitHub] [inlong-website] healchow closed issue #557: [Improve] Optimize the navigation bar document column
healchow closed issue #557: [Improve] Optimize the navigation bar document column URL: https://github.com/apache/inlong-website/issues/557 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] healchow merged pull request #558: [INLONG-557][Website] Optimize the navigation bar document column
healchow merged PR #558: URL: https://github.com/apache/inlong-website/pull/558 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] iamsee123 commented on a diff in pull request #5780: [INLONG-5722][Agent] Support Redis Source
iamsee123 commented on code in PR #5780:
URL: https://github.com/apache/inlong/pull/5780#discussion_r970846441
##
inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/RedisReader.java:
##
@@ -0,0 +1,445 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.agent.plugin.sources.reader;
+
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+import com.google.gson.TypeAdapter;
+import com.google.gson.stream.JsonReader;
+import com.google.gson.stream.JsonWriter;
+import com.moilioncircle.redis.replicator.RedisReplicator;
+import com.moilioncircle.redis.replicator.Replicator;
+import com.moilioncircle.redis.replicator.cmd.CommandName;
+import com.moilioncircle.redis.replicator.cmd.impl.DefaultCommand;
+import com.moilioncircle.redis.replicator.cmd.parser.DefaultCommandParser;
+import com.moilioncircle.redis.replicator.event.Event;
+import com.moilioncircle.redis.replicator.event.EventListener;
+import com.moilioncircle.redis.replicator.event.PostRdbSyncEvent;
+import com.moilioncircle.redis.replicator.rdb.datatype.KeyStringValueHash;
+import com.moilioncircle.redis.replicator.rdb.datatype.KeyStringValueList;
+import com.moilioncircle.redis.replicator.rdb.datatype.KeyStringValueSet;
+import com.moilioncircle.redis.replicator.rdb.datatype.KeyStringValueString;
+import com.moilioncircle.redis.replicator.rdb.datatype.KeyStringValueZSet;
+import com.moilioncircle.redis.replicator.rdb.datatype.KeyValuePair;
+import com.moilioncircle.redis.replicator.rdb.datatype.ZSetEntry;
+import org.apache.inlong.agent.conf.JobProfile;
+import org.apache.inlong.agent.message.DefaultMessage;
+import org.apache.inlong.agent.metrics.audit.AuditUtils;
+import org.apache.inlong.agent.plugin.Message;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+/**
+ * Redis data reader
+ */
+public class RedisReader extends AbstractReader {
+
+public static final String REDIS_READER_TAG_NAME = "AgentRedisMetric";
+public static final String JOB_REDIS_PORT = "job.redisJob.port";
+public static final String JOB_REDIS_HOSTNAME = "job.redisJob.hostname";
+public static final String JOB_REDIS_SSL = "job.redisJob.ssl";
+public static final String JOB_REDIS_AUTHUSER = "job.redisJob.authUser";
+public static final String JOB_REDIS_AUTHPASSWORD =
"job.redisJob.authPassword";
+public static final String JOB_REDIS_READTIMEOUT =
"job.redisJob.readTimeout";
+public static final String JOB_REDIS_QUEUE_SIZE = "job.redisJob.queueSize";
+public static final String JOB_REDIS_REPLID = "job.redisJob.replId";
+public static final String JOB_REDIS_OFFSET = "job.redisJob.offset";
+private static final Logger LOGGER =
LoggerFactory.getLogger(RedisReader.class);
+private String port;
+private String hostName;
+private boolean ssl;
+private String authUser;
+private String authPassword;
+private String readTimeout;
+private String instanceId;
+private String replId;
+private String snapShot;
+private boolean destroyed;
+private Replicator redisReplicator;
+private LinkedBlockingQueue redisMessageQueue;
+private boolean finished = false;
+private ExecutorService executor;
+private Gson gson;
+
Review Comment:
By using lua-replicate-commands configuration directive and
[redis.replicate_commands()](https://redis.io/topics/lua-api#redis.replicate_commands)
Lua API can enable effects replication, then redisReader can read the command
directly. However if using verbatim replication, we just read the eval command
but the change of data can be read by rdb scan at the beginning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] iamsee123 commented on a diff in pull request #5780: [INLONG-5722][Agent] Support Redis Source
iamsee123 commented on code in PR #5780:
URL: https://github.com/apache/inlong/pull/5780#discussion_r970846441
##
inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/RedisReader.java:
##
@@ -0,0 +1,445 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.agent.plugin.sources.reader;
+
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+import com.google.gson.TypeAdapter;
+import com.google.gson.stream.JsonReader;
+import com.google.gson.stream.JsonWriter;
+import com.moilioncircle.redis.replicator.RedisReplicator;
+import com.moilioncircle.redis.replicator.Replicator;
+import com.moilioncircle.redis.replicator.cmd.CommandName;
+import com.moilioncircle.redis.replicator.cmd.impl.DefaultCommand;
+import com.moilioncircle.redis.replicator.cmd.parser.DefaultCommandParser;
+import com.moilioncircle.redis.replicator.event.Event;
+import com.moilioncircle.redis.replicator.event.EventListener;
+import com.moilioncircle.redis.replicator.event.PostRdbSyncEvent;
+import com.moilioncircle.redis.replicator.rdb.datatype.KeyStringValueHash;
+import com.moilioncircle.redis.replicator.rdb.datatype.KeyStringValueList;
+import com.moilioncircle.redis.replicator.rdb.datatype.KeyStringValueSet;
+import com.moilioncircle.redis.replicator.rdb.datatype.KeyStringValueString;
+import com.moilioncircle.redis.replicator.rdb.datatype.KeyStringValueZSet;
+import com.moilioncircle.redis.replicator.rdb.datatype.KeyValuePair;
+import com.moilioncircle.redis.replicator.rdb.datatype.ZSetEntry;
+import org.apache.inlong.agent.conf.JobProfile;
+import org.apache.inlong.agent.message.DefaultMessage;
+import org.apache.inlong.agent.metrics.audit.AuditUtils;
+import org.apache.inlong.agent.plugin.Message;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+/**
+ * Redis data reader
+ */
+public class RedisReader extends AbstractReader {
+
+public static final String REDIS_READER_TAG_NAME = "AgentRedisMetric";
+public static final String JOB_REDIS_PORT = "job.redisJob.port";
+public static final String JOB_REDIS_HOSTNAME = "job.redisJob.hostname";
+public static final String JOB_REDIS_SSL = "job.redisJob.ssl";
+public static final String JOB_REDIS_AUTHUSER = "job.redisJob.authUser";
+public static final String JOB_REDIS_AUTHPASSWORD =
"job.redisJob.authPassword";
+public static final String JOB_REDIS_READTIMEOUT =
"job.redisJob.readTimeout";
+public static final String JOB_REDIS_QUEUE_SIZE = "job.redisJob.queueSize";
+public static final String JOB_REDIS_REPLID = "job.redisJob.replId";
+public static final String JOB_REDIS_OFFSET = "job.redisJob.offset";
+private static final Logger LOGGER =
LoggerFactory.getLogger(RedisReader.class);
+private String port;
+private String hostName;
+private boolean ssl;
+private String authUser;
+private String authPassword;
+private String readTimeout;
+private String instanceId;
+private String replId;
+private String snapShot;
+private boolean destroyed;
+private Replicator redisReplicator;
+private LinkedBlockingQueue redisMessageQueue;
+private boolean finished = false;
+private ExecutorService executor;
+private Gson gson;
+
Review Comment:
By using lua-replicate-commands configuration directive and
[redis.replicate_commands()](https://redis.io/topics/lua-api#redis.replicate_commands)
Lua API can enable effects replication, then redisReader can read the command
directly. However if using verbatim replication, we just read the eval command
but the change of data can be read by rdb scan at the beginning. @GanfengTan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] ciscozhou commented on pull request #5815: [INLONG-5810][Manager] Supplement the workflow event API in the manager client
ciscozhou commented on PR #5815: URL: https://github.com/apache/inlong/pull/5815#issuecomment-1246875929 > @haifxu @ciscozhou PTAL, thanks. OK, I checked and give my +1 to it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] luchunliang closed pull request #5619: [INLONG-5618][SortStandalone] Support sink runner of configurable backoff sleeping time
luchunliang closed pull request #5619: [INLONG-5618][SortStandalone] Support sink runner of configurable backoff sleeping time URL: https://github.com/apache/inlong/pull/5619 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch release-1.3.0 updated (006489dbf -> 097c5935b)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch release-1.3.0 in repository https://gitbox.apache.org/repos/asf/inlong.git from 006489dbf [INLONG-5876][Manager] Remove the check of data separator (#5878) new c999a90da [INLONG-5890][Script] Add tcp protocol when connecting to MySQL (#5891) new 097c5935b [INLONG-5894][Dashboard] Remove the password required restrict for ClickHouse cluster (#5895) The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: bin/inlong-daemon | 8 inlong-dashboard/src/metas/sinks/clickhouse.tsx | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-)
[inlong] 02/02: [INLONG-5894][Dashboard] Remove the password required restrict for ClickHouse cluster (#5895)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch release-1.3.0
in repository https://gitbox.apache.org/repos/asf/inlong.git
commit 097c5935bc086d53208ec36f011641a2e76ae042
Author: Charles Zhang
AuthorDate: Wed Sep 14 20:21:12 2022 +0800
[INLONG-5894][Dashboard] Remove the password required restrict for
ClickHouse cluster (#5895)
---
inlong-dashboard/src/metas/sinks/clickhouse.tsx | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/inlong-dashboard/src/metas/sinks/clickhouse.tsx
b/inlong-dashboard/src/metas/sinks/clickhouse.tsx
index 8941fa3bc..69ad65a46 100644
--- a/inlong-dashboard/src/metas/sinks/clickhouse.tsx
+++ b/inlong-dashboard/src/metas/sinks/clickhouse.tsx
@@ -104,7 +104,7 @@ const getForm: GetStorageFormFieldsType = (
name: 'password',
type: 'password',
label: i18n.t('meta.Sinks.Password'),
- rules: [{ required: true }],
+ rules: [{ required: false }],
props: {
disabled: isEdit && [110, 130].includes(currentValues?.status),
style: {
[inlong] 01/02: [INLONG-5890][Script] Add tcp protocol when connecting to MySQL (#5891)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch release-1.3.0
in repository https://gitbox.apache.org/repos/asf/inlong.git
commit c999a90da85c953baa863cb2b06f7a9481a1d223
Author: Charles Zhang
AuthorDate: Wed Sep 14 19:33:30 2022 +0800
[INLONG-5890][Script] Add tcp protocol when connecting to MySQL (#5891)
---
bin/inlong-daemon | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/bin/inlong-daemon b/bin/inlong-daemon
index 646a42b70..b70bfef26 100644
--- a/bin/inlong-daemon
+++ b/bin/inlong-daemon
@@ -68,10 +68,10 @@ start_inlong_audit() {
echo "init apache_inlong_audit"
cd $INLONG_HOME/inlong-audit
select_db_sql="SELECT COUNT(*) FROM information_schema.TABLES WHERE
table_schema = 'apache_inlong_audit'"
- inlong_audit_count=$(mysql -h${spring_datasource_hostname}
-P${spring_datasource_port} -u${spring_datasource_username}
-p${spring_datasource_password} -e "${select_db_sql}")
+ inlong_audit_count=$(mysql --protocol=tcp -h${spring_datasource_hostname}
-P${spring_datasource_port} -u${spring_datasource_username}
-p${spring_datasource_password} -e "${select_db_sql}")
inlong_num=$(echo $inlong_audit_count | tr -cd "[0-9]")
if [ $inlong_num -eq 0 ]; then
-mysql -h${spring_datasource_hostname} -P${spring_datasource_port}
-u${spring_datasource_username} -p${spring_datasource_password}
[GitHub] [inlong] EMsnap commented on a diff in pull request #5896: [INLONG-5889][Sort] Fix MySQL Node JDBC Url For RCE Vulnerability
EMsnap commented on code in PR #5896:
URL: https://github.com/apache/inlong/pull/5896#discussion_r971450030
##
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/util/JdbcUrlUtils.java:
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.util;
+
+import static org.apache.inlong.sort.base.Constants.AUTO_DESERIALIZE_FALSE;
+import static org.apache.inlong.sort.base.Constants.AUTO_DESERIALIZE_TRUE;
+
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * utils for jdbc url
+ */
+public class JdbcUrlUtils {
+
+private static final Logger LOG =
LoggerFactory.getLogger(JdbcUrlUtils.class);
+
+/**
+ * see https://su18.org/post/jdbc-connection-url-attack/
+ * @param url
+ * @return url after filtering out the invalid property
+ */
+public static String replaceInvalidUrlProperty(String url) {
+if (StringUtils.containsIgnoreCase(url, AUTO_DESERIALIZE_TRUE)) {
+LOG.info("url {} contains invalid property {}, replace it to {}",
url,
Review Comment:
will do thx
##
inlong-sort/sort-connectors/jdbc/src/main/java/org/apache/inlong/sort/jdbc/table/JdbcDynamicTableFactory.java:
##
@@ -219,6 +221,7 @@ public DynamicTableSource createDynamicTableSource(Context
context) {
private JdbcOptions getJdbcOptions(ReadableConfig readableConfig) {
final String url = readableConfig.get(URL);
+JdbcUrlUtils.replaceInvalidUrlProperty(url);
Review Comment:
noted thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #5864: [INLONG-5863][Manager] Extend Redis extract node
healchow commented on code in PR #5864:
URL: https://github.com/apache/inlong/pull/5864#discussion_r971451447
##
inlong-common/src/main/java/org/apache/inlong/common/enums/TaskTypeEnum.java:
##
@@ -29,7 +29,8 @@ public enum TaskTypeEnum {
ORACLE(7),
SQLSERVER(8),
MONGODB(9),
-TUBEMQ(10)
+TUBEMQ(10),
+REDIS(11)
Review Comment:
Please add a comma (,) after `REDIS(11)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] github-actions[bot] commented on issue #3690: [Improve] Initialize database by SQL file automatic
github-actions[bot] commented on issue #3690: URL: https://github.com/apache/inlong/issues/3690#issuecomment-1247490565 This issue is stale because it has been open for 60 days with no activity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] xuesongxs commented on issue #5814: [Bug][Dashboard] Supports setting the precision of a field when adding a table field
xuesongxs commented on issue #5814: URL: https://github.com/apache/inlong/issues/5814#issuecomment-1247502621 If the front page does not provide field length, an error will be reported when creating a table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] healchow merged pull request #5777: [INLONG-5775][Manager] Add unit tests for the Inlong Consume
healchow merged PR #5777: URL: https://github.com/apache/inlong/pull/5777 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-5775][Manager] Add unit tests for the Inlong Consume (#5777)
This is an automated email from the ASF dual-hosted git repository.
healchow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new cc63c57e3 [INLONG-5775][Manager] Add unit tests for the Inlong Consume
(#5777)
cc63c57e3 is described below
commit cc63c57e396fe4d09a46d17338e44977ea3d580f
Author: ciscozhou <45899072+ciscoz...@users.noreply.github.com>
AuthorDate: Thu Sep 15 11:13:51 2022 +0800
[INLONG-5775][Manager] Add unit tests for the Inlong Consume (#5777)
Co-authored-by: healchow
---
.../dao/mapper/InlongConsumeEntityMapper.java | 4 +-
.../resources/mappers/ConsumptionEntityMapper.xml | 2 +-
.../mappers/InlongConsumeEntityMapper.xml | 8 +-
.../mappers/WorkflowProcessEntityMapper.xml| 2 +-
.../resources/mappers/WorkflowTaskEntityMapper.xml | 2 +-
.../pojo/consume/InlongConsumeCountInfo.java | 8 +-
.../service/consume/InlongConsumeServiceImpl.java | 26 ++--
.../service/consume/InlongConsumeServiceTest.java | 165 +
.../service/core/impl/InlongStreamServiceTest.java | 21 +--
.../service/group/InlongGroupServiceTest.java | 21 ++-
10 files changed, 207 insertions(+), 52 deletions(-)
diff --git
a/inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/mapper/InlongConsumeEntityMapper.java
b/inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/mapper/InlongConsumeEntityMapper.java
index feaf28a50..ab9615e70 100644
---
a/inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/mapper/InlongConsumeEntityMapper.java
+++
b/inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/mapper/InlongConsumeEntityMapper.java
@@ -19,11 +19,11 @@ package org.apache.inlong.manager.dao.mapper;
import org.apache.ibatis.annotations.Param;
import org.apache.inlong.manager.dao.entity.InlongConsumeEntity;
+import org.apache.inlong.manager.pojo.common.CountInfo;
import org.apache.inlong.manager.pojo.consume.InlongConsumePageRequest;
import org.springframework.stereotype.Repository;
import java.util.List;
-import java.util.Map;
@Repository
public interface InlongConsumeEntityMapper {
@@ -32,7 +32,7 @@ public interface InlongConsumeEntityMapper {
InlongConsumeEntity selectById(Integer id);
-List> countByUser(@Param(value = "username") String
username);
+List countByUser(@Param(value = "username") String username);
List selectByCondition(InlongConsumePageRequest
request);
diff --git
a/inlong-manager/manager-dao/src/main/resources/mappers/ConsumptionEntityMapper.xml
b/inlong-manager/manager-dao/src/main/resources/mappers/ConsumptionEntityMapper.xml
index 885cc22e6..9865ca0a1 100644
---
a/inlong-manager/manager-dao/src/main/resources/mappers/ConsumptionEntityMapper.xml
+++
b/inlong-manager/manager-dao/src/main/resources/mappers/ConsumptionEntityMapper.xml
@@ -264,7 +264,7 @@
-select status as `key`, count(1) as value
+select status as `key`, count(1) as `value`
from consumption
where is_deleted=0
diff --git
a/inlong-manager/manager-dao/src/main/resources/mappers/InlongConsumeEntityMapper.xml
b/inlong-manager/manager-dao/src/main/resources/mappers/InlongConsumeEntityMapper.xml
index 435d6ca3c..0eaee7686 100644
---
a/inlong-manager/manager-dao/src/main/resources/mappers/InlongConsumeEntityMapper.xml
+++
b/inlong-manager/manager-dao/src/main/resources/mappers/InlongConsumeEntityMapper.xml
@@ -51,12 +51,12 @@
mq_type, topic, inlong_group_id,
filter_enabled, inlong_stream_id,
ext_params, in_charges, status,
-is_deleted, creator, modifier)
+creator, modifier)
values (#{id, jdbcType=INTEGER}, #{consumerGroup, jdbcType=VARCHAR},
#{description, jdbcType=VARCHAR},
#{mqType, jdbcType=VARCHAR}, #{topic, jdbcType=VARCHAR},
#{inlongGroupId, jdbcType=VARCHAR},
#{filterEnabled, jdbcType=INTEGER}, #{inlongStreamId,
jdbcType=VARCHAR},
#{extParams, jdbcType=LONGVARCHAR}, #{inCharges,
jdbcType=VARCHAR}, #{status, jdbcType=INTEGER},
-#{isDeleted, jdbcType=INTEGER}, #{creator, jdbcType=VARCHAR},
#{modifier, jdbcType=VARCHAR})
+#{creator, jdbcType=VARCHAR}, #{modifier, jdbcType=VARCHAR})
@@ -65,8 +65,8 @@
from inlong_consume
where id = #{id, jdbcType=INTEGER}
-
-select status, count(1) as total
+
+select status as `key`, count(1) as `value`
from inlong_consume
where is_deleted = 0
and (creator = #{username, jdbcType=VARCHAR} or
FIND_IN_SET(#{username, jdbcType=VARCHAR}, in_charges))
diff --git
a/inlong-manager/man
[GitHub] [inlong] dockerzhang merged pull request #5896: [INLONG-5889][Sort] Fix MySQL Node JDBC Url For RCE Vulnerability
dockerzhang merged PR #5896: URL: https://github.com/apache/inlong/pull/5896 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-5889][Sort] Fix MySQL Node JDBC Url For RCE Vulnerability (#5896)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 0719f95d3 [INLONG-5889][Sort] Fix MySQL Node JDBC Url For RCE
Vulnerability (#5896)
0719f95d3 is described below
commit 0719f95d3362540d69c6a7924d33e16a34671826
Author: Schnapps
AuthorDate: Thu Sep 15 11:27:23 2022 +0800
[INLONG-5889][Sort] Fix MySQL Node JDBC Url For RCE Vulnerability (#5896)
---
.../org/apache/inlong/sort/base/Constants.java | 11 +
.../apache/inlong/sort/base/util/JdbcUrlUtils.java | 50 ++
.../inlong/sort/base/metric/TestUrlValidate.java | 46
.../sort/jdbc/table/JdbcDynamicTableFactory.java | 4 +-
.../inlong/sort/cdc/mysql/table/JdbcUrlUtils.java | 9 +++-
5 files changed, 118 insertions(+), 2 deletions(-)
diff --git
a/inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java
b/inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java
index 9dd124284..45023d38b 100644
---
a/inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java
+++
b/inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java
@@ -94,4 +94,15 @@ public final class Constants {
.defaultValue(false)
.withDescription("Regard upsert delete as insert kind.");
+
+/**
+ * It is used for jdbc url filter for avoiding url attack
+ * see also in https://su18.org/post/jdbc-connection-url-attack/
+ */
+public static final String AUTO_DESERIALIZE = "autoDeserialize";
+
+public static final String AUTO_DESERIALIZE_TRUE = "autoDeserialize=true";
+
+public static final String AUTO_DESERIALIZE_FALSE =
"autoDeserialize=false";
+
}
diff --git
a/inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/util/JdbcUrlUtils.java
b/inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/util/JdbcUrlUtils.java
new file mode 100644
index 0..253f783ca
--- /dev/null
+++
b/inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/util/JdbcUrlUtils.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.util;
+
+import static org.apache.inlong.sort.base.Constants.AUTO_DESERIALIZE_FALSE;
+import static org.apache.inlong.sort.base.Constants.AUTO_DESERIALIZE_TRUE;
+
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * utils for jdbc url
+ */
+public class JdbcUrlUtils {
+
+private static final Logger LOG =
LoggerFactory.getLogger(JdbcUrlUtils.class);
+
+/**
+ * see https://su18.org/post/jdbc-connection-url-attack/
+ * @param url
+ * @return url after filtering out the invalid property
+ */
+public static String replaceInvalidUrlProperty(String url) {
+if (StringUtils.containsIgnoreCase(url, AUTO_DESERIALIZE_TRUE)) {
+LOG.warn("url {} contains invalid property {}, replace it to {}",
url,
+AUTO_DESERIALIZE_TRUE, AUTO_DESERIALIZE_FALSE);
+return StringUtils.replaceIgnoreCase(url, AUTO_DESERIALIZE_TRUE,
+AUTO_DESERIALIZE_FALSE);
+}
+return url;
+}
+
+}
diff --git
a/inlong-sort/sort-connectors/base/src/test/java/org/apache/inlong/sort/base/metric/TestUrlValidate.java
b/inlong-sort/sort-connectors/base/src/test/java/org/apache/inlong/sort/base/metric/TestUrlValidate.java
new file mode 100644
index 0..8e090e5d3
--- /dev/null
+++
b/inlong-sort/sort-connectors/base/src/test/java/org/apache/inlong/sort/base/metric/TestUrlValidate.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not us
[GitHub] [inlong] woofyzhao opened a new pull request, #5898: [INLONG-5897][Manager] Support query audit data by user role
woofyzhao opened a new pull request, #5898: URL: https://github.com/apache/inlong/pull/5898 - Fixes #5897 ### Motivation In some cases, admin and ordinary user may need different audit data view, e.g : only admin needs the complete audit data view, and ordinary customer user needs only to know data collection count and sort output out -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #5898: [INLONG-5897][Manager] Support query audit data by user role
healchow commented on code in PR #5898:
URL: https://github.com/apache/inlong/pull/5898#discussion_r971500758
##
inlong-common/src/main/java/org/apache/inlong/common/constant/Constants.java:
##
@@ -23,7 +23,15 @@
public class Constants {
public static final int RESULT_SUCCESS = 0;
-
public static final int RESULT_FAIL = 1;
Review Comment:
Suggest creating a new AuditConstants class for those audit constants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #5898: [INLONG-5897][Manager] Support query audit data by user role
healchow commented on code in PR #5898:
URL: https://github.com/apache/inlong/pull/5898#discussion_r971507573
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/AuditServiceImpl.java:
##
@@ -128,6 +148,24 @@ public List listByCondition(AuditRequest request)
throws IOException {
return aggregateByTimeDim(result, request.getTimeStaticsDim());
}
+@Override
+public List listByRole(AuditRequest request) throws IOException {
+UserInfo userInfo = LoginUserUtils.getLoginUser();
+List adminAuditIds = Arrays.asList(
Review Comment:
Suggest extracting to an immutable constant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] gosonzhang opened a new pull request, #5900: [INLONG-5899][DataProxy] Optimize Http's SimpleMessageHandler class
gosonzhang opened a new pull request, #5900: URL: https://github.com/apache/inlong/pull/5900 - Fixes #5899 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] EMsnap commented on issue #5617: [Umbrella] Release InLong 1.3.0
EMsnap commented on issue #5617: URL: https://github.com/apache/inlong/issues/5617#issuecomment-1247640642 see https://inlong.apache.org/blog/2022/09/05/release-1.3.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
