[GitHub] [inlong] GanfengTan opened a new pull request, #5862: [INLONG-5861][Agent] Support symbolic links for file
GanfengTan opened a new pull request, #5862: URL: https://github.com/apache/inlong/pull/5862 Support symbolic links for file, to reset postion is 0 when change symbolic links for file. - Fixes #5861 ### Motivation No message ### Modifications No message ### Verifying this change *(Please pick either of the following options)* - [x] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] iamsee123 opened a new pull request, #5864: [INLONG-5863][Manager] extend redis extract node
iamsee123 opened a new pull request, #5864: URL: https://github.com/apache/inlong/pull/5864 ### Prepare a Pull Request - Fixes #5863 ### Motivation extend redis ExtractNode in Manager module ### Modifications Follow the [extend extractNode](https://inlong.apache.org/zh-CN/docs/design_and_concept/how_to_extend_data_node_for_manager) + Add RedisSource, RedisSourceDTO, RedisRequestDTO + Add Redis type in TaskTypeEnum and SourceType + Add RedisSourceOperator + Add unit test for Redis source service ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [x] This change added tests and can be verified as follows: - RedisSourceServiceTest for testing Redis source service ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] luchunliang opened a new pull request, #5866: [INLONG-5865][Manager] Support maintenance of inlong group
luchunliang opened a new pull request, #5866: URL: https://github.com/apache/inlong/pull/5866 ### Prepare a Pull Request - Title : [INLONG-5865][Manager] Support maintenance of inlong group - Fixes #5865 ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve?* ### Modifications *Describe the modifications you've done.* ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang commented on pull request #5780: [INLONG-5722][Agent] Support Redis Source
dockerzhang commented on PR #5780: URL: https://github.com/apache/inlong/pull/5780#issuecomment-1243880491 @iamsee123 please fix the comment and conficts -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #5855: [INLONG-5848][TubeMQ] Get duplicated call when calling TubeMQConsumer::Start()
dockerzhang merged PR #5855: URL: https://github.com/apache/inlong/pull/5855 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated (a2eeee4f2 -> b179c2bea)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong.git from a24f2 [INLONG-5853][TubeMQ] Fix typo in AddTopicExecutor (#5854) add b179c2bea [INLONG-5848][TubeMQ] Get duplicated call when calling TubeMQConsumer::Start() (#5855) No new revisions were added by this update. Summary of changes: .../tubemq-client-twins/tubemq-client-cpp/src/baseconsumer.cc | 3 +++ .../tubemq-client-twins/tubemq-client-cpp/src/tubemq_client.cc | 2 ++ inlong-tubemq/tubemq-client-twins/tubemq-client-cpp/src/version.h | 2 +- 3 files changed, 6 insertions(+), 1 deletion(-)
[GitHub] [inlong] dockerzhang commented on pull request #5862: [INLONG-5861][Agent] Support symbolic links for file
dockerzhang commented on PR #5862: URL: https://github.com/apache/inlong/pull/5862#issuecomment-1244807704 @pocozh PTAL, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] gosonzhang commented on a diff in pull request #5852: [INLONG-5851][TubeMQ] Optimize while-sleep to ScheduledExecutorService
gosonzhang commented on code in PR #5852: URL: https://github.com/apache/inlong/pull/5852#discussion_r969075692 ## inlong-tubemq/tubemq-manager/pom.xml: ## @@ -176,6 +176,10 @@ jakarta.validation jakarta.validation-api + Review Comment: Need to add the LICENSE and NOTICE (if any) that the third party depends on in [1] https://github.com/apache/inlong/blob/master/licenses/inlong-tubemq-manager/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #5860: [INLONG-5859][TubeMQ] Remove the redundant codes in DefaultOffsetManager
dockerzhang merged PR #5860: URL: https://github.com/apache/inlong/pull/5860 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated (b179c2bea -> a87dd8a60)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong.git from b179c2bea [INLONG-5848][TubeMQ] Get duplicated call when calling TubeMQConsumer::Start() (#5855) add a87dd8a60 [INLONG-5859][TubeMQ] Remove the redundant codes in DefaultOffsetManager (#5860) No new revisions were added by this update. Summary of changes: .../tubemq/server/broker/offset/DefaultOffsetManager.java | 15 +-- 1 file changed, 1 insertion(+), 14 deletions(-)
[GitHub] [inlong] github-actions[bot] commented on issue #5038: [Improve][Manager] Support to create resources using sql for managerctl
github-actions[bot] commented on issue #5038: URL: https://github.com/apache/inlong/issues/5038#issuecomment-1244810623 This issue is stale because it has been open for 60 days with no activity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] github-actions[bot] commented on issue #5028: [Feature][Manager] Adaptation to increase the union conversion node
github-actions[bot] commented on issue #5028: URL: https://github.com/apache/inlong/issues/5028#issuecomment-1244810638 This issue is stale because it has been open for 60 days with no activity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] github-actions[bot] commented on issue #4986: [Feature][Agent] Support MQTT Source
github-actions[bot] commented on issue #4986: URL: https://github.com/apache/inlong/issues/4986#issuecomment-1244810663 This issue is stale because it has been open for 60 days with no activity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] github-actions[bot] commented on issue #4985: [Improve][Agent] Improve File Collection Capabilities
github-actions[bot] commented on issue #4985: URL: https://github.com/apache/inlong/issues/4985#issuecomment-1244810682 This issue is stale because it has been open for 60 days with no activity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] github-actions[bot] commented on issue #4982: [Feature][Dashboard] Extract / Load Node Management
github-actions[bot] commented on issue #4982: URL: https://github.com/apache/inlong/issues/4982#issuecomment-1244810708 This issue is stale because it has been open for 60 days with no activity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] github-actions[bot] commented on issue #4979: [Feature][Dashboard] Data Stream Task Status Management
github-actions[bot] commented on issue #4979: URL: https://github.com/apache/inlong/issues/4979#issuecomment-1244810739 This issue is stale because it has been open for 60 days with no activity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] github-actions[bot] commented on issue #4981: [Feature][Dashboard] Add Workflow Management for Data Stream
github-actions[bot] commented on issue #4981: URL: https://github.com/apache/inlong/issues/4981#issuecomment-1244810727 This issue is stale because it has been open for 60 days with no activity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] github-actions[bot] commented on issue #4975: [Feature][Manager] Support Agent Management
github-actions[bot] commented on issue #4975: URL: https://github.com/apache/inlong/issues/4975#issuecomment-1244810755 This issue is stale because it has been open for 60 days with no activity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] github-actions[bot] commented on issue #4967: [Feature][Sort] Expose Metrics
github-actions[bot] commented on issue #4967: URL: https://github.com/apache/inlong/issues/4967#issuecomment-1244810772 This issue is stale because it has been open for 60 days with no activity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] github-actions[bot] commented on issue #2306: [Feature][Manag] Batch Data Stream Management
github-actions[bot] commented on issue #2306: URL: https://github.com/apache/inlong/issues/2306#issuecomment-1244810842 This issue is stale because it has been open for 60 days with no activity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] github-actions[bot] commented on issue #3680: [Feature][Manager] Make the default admin account configurable
github-actions[bot] commented on issue #3680: URL: https://github.com/apache/inlong/issues/3680#issuecomment-1244810825 This issue is stale because it has been open for 60 days with no activity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] gosonzhang opened a new pull request, #5868: [INLONG-5867][TubeMQ] Add registered partition information printing
gosonzhang opened a new pull request, #5868: URL: https://github.com/apache/inlong/pull/5868 - Fixes #5867 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] healchow commented on issue #5869: [Bug][Manager] Error in /api/inlong/manager/consumption/save
healchow commented on issue #5869: URL: https://github.com/apache/inlong/issues/5869#issuecomment-1244820227 Hello, thanks for your concern and test. Those APIs were changed. Please use the master branch or the latest release (1.3.0) to test again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch release-1.3.0 updated: [INLONG-5844][Manager] Sub sources should be filtered when querying task status (#5845)
This is an automated email from the ASF dual-hosted git repository.
healchow pushed a commit to branch release-1.3.0
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/release-1.3.0 by this push:
new e67adc4e7 [INLONG-5844][Manager] Sub sources should be filtered when
querying task status (#5845)
e67adc4e7 is described below
commit e67adc4e75c8c5b294c6ebc91fa71aee1934f38e
Author: woofyzhao <490467...@qq.com>
AuthorDate: Sat Sep 10 12:31:01 2022 +0800
[INLONG-5844][Manager] Sub sources should be filtered when querying task
status (#5845)
Co-authored-by: healchow
---
.../java/org/apache/inlong/manager/client/api/InlongGroupContext.java | 4 +++-
.../main/java/org/apache/inlong/manager/pojo/source/StreamSource.java | 3 +++
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git
a/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/InlongGroupContext.java
b/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/InlongGroupContext.java
index 3ba3a0aec..52cca26a7 100644
---
a/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/InlongGroupContext.java
+++
b/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/InlongGroupContext.java
@@ -90,7 +90,9 @@ public class InlongGroupContext implements Serializable {
if (MapUtils.isNotEmpty(sources)) {
for (Map.Entry entry :
sources.entrySet()) {
StreamSource source = entry.getValue();
-if (source != null) {
+// when template id is null it is considered as normal
source other than template source
+// sub sources are filtered because they are already
collected in template source's sub source list
+if (source != null && source.getTemplateId() == null) {
groupSources.add(source);
}
}
diff --git
a/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/source/StreamSource.java
b/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/source/StreamSource.java
index 9c07d7969..a95dd6d5b 100644
---
a/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/source/StreamSource.java
+++
b/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/source/StreamSource.java
@@ -104,6 +104,9 @@ public abstract class StreamSource extends StreamNode {
@Builder.Default
private Map properties = new LinkedHashMap<>();
+@ApiModelProperty("Null if not a sub source")
+private Integer templateId;
+
@ApiModelProperty("Sub source information of existing agents")
private List subSourceList;
[GitHub] [inlong-website] dockerzhang merged pull request #542: [INLONG-541][Sort] Modify doc for importing all changelog data into Kafka
dockerzhang merged PR #542: URL: https://github.com/apache/inlong-website/pull/542 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] dockerzhang closed issue #541: [Feature][Sort] Modify doc for importing all changelog data into Kafka
dockerzhang closed issue #541: [Feature][Sort] Modify doc for importing all changelog data into Kafka URL: https://github.com/apache/inlong-website/issues/541 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong-website] branch master updated: [INLONG-541][Sort] Modify doc for importing all changelog data into Kafka (#542)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/inlong-website.git The following commit(s) were added to refs/heads/master by this push: new 2fe22f8764 [INLONG-541][Sort] Modify doc for importing all changelog data into Kafka (#542) 2fe22f8764 is described below commit 2fe22f87648e4368285e5171d7795ce88db924e8 Author: Oneal65 AuthorDate: Tue Sep 13 10:49:17 2022 +0800 [INLONG-541][Sort] Modify doc for importing all changelog data into Kafka (#542) --- docs/data_node/extract_node/kafka.md | 1 + .../current/data_node/extract_node/kafka.md | 1 + 2 files changed, 2 insertions(+) diff --git a/docs/data_node/extract_node/kafka.md b/docs/data_node/extract_node/kafka.md index 2a7f784e47..d843c1dec5 100644 --- a/docs/data_node/extract_node/kafka.md +++ b/docs/data_node/extract_node/kafka.md @@ -110,6 +110,7 @@ TODO: It will be supported in the future. | scan.startup.specific-offsets | optional | (none) | String | Specify offsets for each partition in case of 'specific-offsets' startup mode, e.g. 'partition:0,offset:42;partition:1,offset:300'. | | scan.startup.timestamp-millis | optional | (none) | Long | Start from the specified epoch timestamp (milliseconds) used in case of 'timestamp' startup mode. | | scan.topic-partition-discovery.interval | optional | (none) | Duration | Interval for consumer to discover dynamically created Kafka topics and partitions periodically. | +| sink.ignore.changelog | optional | false | Boolean | Importing all changelog mode data ingest into Kafka . | ## Available Metadata diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/kafka.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/kafka.md index 196a95d65a..2acd5dcb85 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/kafka.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/kafka.md @@ -108,6 +108,7 @@ TODO: 将在未来支持此功能。 | scan.startup.specific-offsets | 可选 | (none) | String | 在使用 'specific-offsets' 启动模式时为每个 partition 指定 offset，例如 'partition:0,offset:42;partition:1,offset:300'。 | | scan.startup.timestamp-millis | 可选 | (none) | Long | 在使用 'timestamp' 启动模式时指定启动的时间戳（单位毫秒）。 | | scan.topic-partition-discovery.interval | 可选 | (none) | Duration | Consumer 定期探测动态创建的 Kafka topic 和 partition 的时间间隔。 | +| sink.ignore.changelog | 可选 | false | 布尔型 | 支持所有类型的 changelog 流 ingest 到 Kafka。 | ## 可用的元数据字段
[GitHub] [inlong] iamsee123 commented on a diff in pull request #5780: [INLONG-5722][Agent] Support Redis Source
iamsee123 commented on code in PR #5780:
URL: https://github.com/apache/inlong/pull/5780#discussion_r969101231
##
inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/RedisReader.java:
##
@@ -0,0 +1,445 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.agent.plugin.sources.reader;
+
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+import com.google.gson.TypeAdapter;
+import com.google.gson.stream.JsonReader;
+import com.google.gson.stream.JsonWriter;
+import com.moilioncircle.redis.replicator.RedisReplicator;
+import com.moilioncircle.redis.replicator.Replicator;
+import com.moilioncircle.redis.replicator.cmd.CommandName;
+import com.moilioncircle.redis.replicator.cmd.impl.DefaultCommand;
+import com.moilioncircle.redis.replicator.cmd.parser.DefaultCommandParser;
+import com.moilioncircle.redis.replicator.event.Event;
+import com.moilioncircle.redis.replicator.event.EventListener;
+import com.moilioncircle.redis.replicator.event.PostRdbSyncEvent;
+import com.moilioncircle.redis.replicator.rdb.datatype.KeyStringValueHash;
+import com.moilioncircle.redis.replicator.rdb.datatype.KeyStringValueList;
+import com.moilioncircle.redis.replicator.rdb.datatype.KeyStringValueSet;
+import com.moilioncircle.redis.replicator.rdb.datatype.KeyStringValueString;
+import com.moilioncircle.redis.replicator.rdb.datatype.KeyStringValueZSet;
+import com.moilioncircle.redis.replicator.rdb.datatype.KeyValuePair;
+import com.moilioncircle.redis.replicator.rdb.datatype.ZSetEntry;
+import org.apache.inlong.agent.conf.JobProfile;
+import org.apache.inlong.agent.message.DefaultMessage;
+import org.apache.inlong.agent.metrics.audit.AuditUtils;
+import org.apache.inlong.agent.plugin.Message;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+/**
+ * Redis data reader
+ */
+public class RedisReader extends AbstractReader {
+
+public static final String REDIS_READER_TAG_NAME = "AgentRedisMetric";
+public static final String JOB_REDIS_PORT = "job.redisJob.port";
+public static final String JOB_REDIS_HOSTNAME = "job.redisJob.hostname";
+public static final String JOB_REDIS_SSL = "job.redisJob.ssl";
+public static final String JOB_REDIS_AUTHUSER = "job.redisJob.authUser";
+public static final String JOB_REDIS_AUTHPASSWORD =
"job.redisJob.authPassword";
+public static final String JOB_REDIS_READTIMEOUT =
"job.redisJob.readTimeout";
+public static final String JOB_REDIS_QUEUE_SIZE = "job.redisJob.queueSize";
+public static final String JOB_REDIS_REPLID = "job.redisJob.replId";
+public static final String JOB_REDIS_OFFSET = "job.redisJob.offset";
+private static final Logger LOGGER =
LoggerFactory.getLogger(RedisReader.class);
+private String port;
+private String hostName;
+private boolean ssl;
+private String authUser;
+private String authPassword;
+private String readTimeout;
+private String instanceId;
+private String replId;
+private String snapShot;
+private boolean destroyed;
+private Replicator redisReplicator;
+private LinkedBlockingQueue redisMessageQueue;
+private boolean finished = false;
+private ExecutorService executor;
+private Gson gson;
+
Review Comment:
what kind of filter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #5862: [INLONG-5861][Agent] Support symbolic links for file
dockerzhang merged PR #5862: URL: https://github.com/apache/inlong/pull/5862 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-5861][Agent] Support symbolic links for file (#5862)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 539fdb3ea [INLONG-5861][Agent] Support symbolic links for file (#5862)
539fdb3ea is described below
commit 539fdb3ea425e13d93b5d98b35eb925ccf84cb5e
Author: ganfengtan
AuthorDate: Tue Sep 13 11:01:29 2022 +0800
[INLONG-5861][Agent] Support symbolic links for file (#5862)
---
.../sources/reader/file/MonitorTextFile.java | 28 --
1 file changed, 21 insertions(+), 7 deletions(-)
diff --git
a/inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/file/MonitorTextFile.java
b/inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/file/MonitorTextFile.java
index dfad65450..0c6f6a72e 100644
---
a/inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/file/MonitorTextFile.java
+++
b/inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/file/MonitorTextFile.java
@@ -82,6 +82,7 @@ public final class MonitorTextFile {
private final TextFileReader textFileReader;
private final Long interval;
private final long startTime = System.currentTimeMillis();
+private String path;
/**
* the last modify time of the file
*/
@@ -95,6 +96,7 @@ public final class MonitorTextFile {
try {
this.attributesBefore = Files
.readAttributes(fileReaderOperator.file.toPath(),
BasicFileAttributes.class);
+this.path = this.fileReaderOperator.file.getCanonicalPath();
} catch (IOException e) {
LOGGER.error("get {} last modify time error:",
fileReaderOperator.file.getName(), e);
}
@@ -121,17 +123,29 @@ public final class MonitorTextFile {
}
private void listen() throws IOException {
-BasicFileAttributes attributesAfter = Files
-.readAttributes(this.fileReaderOperator.file.toPath(),
BasicFileAttributes.class);
+BasicFileAttributes attributesAfter;
+String currentPath;
+try {
+attributesAfter = Files
+.readAttributes(this.fileReaderOperator.file.toPath(),
BasicFileAttributes.class);
+currentPath = this.fileReaderOperator.file.getCanonicalPath();
+} catch (Exception e) {
+// Set position 0 when split file
+this.fileReaderOperator.position = 0;
+LOGGER.error("monitor {} error, reset position is 0:",
this.fileReaderOperator.file.getName(), e);
+return;
+}
+// If change symbolic links
+if (attributesAfter.isSymbolicLink() && !path.equals(currentPath))
{
+this.fileReaderOperator.position = 0;
+path = currentPath;
+}
if
(attributesBefore.lastModifiedTime().compareTo(attributesAfter.lastModifiedTime())
< 0) {
// Not triggered during data sending
-if (Objects.nonNull(this.fileReaderOperator.iterator) &&
this.fileReaderOperator.iterator.hasNext()) {
+if (Objects.nonNull(this.fileReaderOperator.iterator) &&
this.fileReaderOperator.iterator
+.hasNext()) {
return;
}
-// Set position 0 when split file
-if
(attributesBefore.creationTime().compareTo(attributesAfter.creationTime()) < 0)
{
-this.fileReaderOperator.position = 0;
-}
this.textFileReader.getData();
this.textFileReader.mergeData(this.fileReaderOperator);
this.attributesBefore = attributesAfter;
[GitHub] [inlong] dockerzhang merged pull request #5868: [INLONG-5867][TubeMQ] Add registered partition information printing
dockerzhang merged PR #5868: URL: https://github.com/apache/inlong/pull/5868 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated (539fdb3ea -> ee16aa60f)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong.git from 539fdb3ea [INLONG-5861][Agent] Support symbolic links for file (#5862) add ee16aa60f [INLONG-5867][TubeMQ] Add registered partition information printing (#5868) No new revisions were added by this update. Summary of changes: .../tubemq-client-cpp/src/baseconsumer.cc | 17 - .../tubemq-client-cpp/src/baseconsumer.h| 2 +- 2 files changed, 13 insertions(+), 6 deletions(-)
[GitHub] [inlong] dockerzhang merged pull request #5850: [INLONG-5849][Manager] Update the inlong group ext info by the client
dockerzhang merged PR #5850: URL: https://github.com/apache/inlong/pull/5850 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-5849][Manager] Update the inlong group ext info by the client (#5850)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new c85b1bc2f [INLONG-5849][Manager] Update the inlong group ext info by
the client (#5850)
c85b1bc2f is described below
commit c85b1bc2f04d8844e787f5fc0c52f39399a3296d
Author: healchow
AuthorDate: Tue Sep 13 11:03:22 2022 +0800
[INLONG-5849][Manager] Update the inlong group ext info by the client
(#5850)
---
.../inlong/manager/client/cli/CreateCommand.java | 4 +-
.../manager/client/api/impl/InlongClientImpl.java | 20 +-
.../manager/client/api/impl/InlongGroupImpl.java | 45 +++---
.../client/api/util/InlongGroupTransfer.java | 7 +---
4 files changed, 37 insertions(+), 39 deletions(-)
diff --git
a/inlong-manager/manager-client-tools/src/main/java/org/apache/inlong/manager/client/cli/CreateCommand.java
b/inlong-manager/manager-client-tools/src/main/java/org/apache/inlong/manager/client/cli/CreateCommand.java
index 2a099f958..24b46f34a 100644
---
a/inlong-manager/manager-client-tools/src/main/java/org/apache/inlong/manager/client/cli/CreateCommand.java
+++
b/inlong-manager/manager-client-tools/src/main/java/org/apache/inlong/manager/client/cli/CreateCommand.java
@@ -70,9 +70,9 @@ public class CreateCommand extends AbstractCommand {
return;
}
}
-// first extract groupconfig from the file passed in
+// first extract group config from the file passed in
CreateGroupConf groupConf = objectMapper.readValue(content,
CreateGroupConf.class);
-// get the correspodning inlonggroup, a.k.a the task to execute
+// get the corresponding inlong group, aka the task to execute
InlongClient inlongClient = ClientUtils.getClient();
InlongGroup group =
inlongClient.forGroup(groupConf.getGroupInfo());
InlongStreamBuilder streamBuilder =
group.createStream(groupConf.getStreamInfo());
diff --git
a/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/impl/InlongClientImpl.java
b/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/impl/InlongClientImpl.java
index eac22c0fa..9e84a30ef 100644
---
a/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/impl/InlongClientImpl.java
+++
b/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/impl/InlongClientImpl.java
@@ -28,12 +28,12 @@ import
org.apache.inlong.manager.client.api.ClientConfiguration;
import org.apache.inlong.manager.client.api.InlongClient;
import org.apache.inlong.manager.client.api.InlongCluster;
import org.apache.inlong.manager.client.api.InlongGroup;
-import org.apache.inlong.manager.common.enums.SimpleGroupStatus;
-import org.apache.inlong.manager.common.enums.SimpleSourceStatus;
import org.apache.inlong.manager.client.api.inner.client.ClientFactory;
import org.apache.inlong.manager.client.api.inner.client.InlongClusterClient;
import org.apache.inlong.manager.client.api.inner.client.InlongGroupClient;
import org.apache.inlong.manager.client.api.util.ClientUtils;
+import org.apache.inlong.manager.common.enums.SimpleGroupStatus;
+import org.apache.inlong.manager.common.enums.SimpleSourceStatus;
import org.apache.inlong.manager.common.util.HttpUtils;
import org.apache.inlong.manager.common.util.Preconditions;
import org.apache.inlong.manager.pojo.cluster.BindTagRequest;
@@ -108,17 +108,17 @@ public class InlongClientImpl implements InlongClient {
@Override
public List listGroup(String expr, int status, int pageNum,
int pageSize) {
-PageResult pageInfo =
groupClient.listGroups(expr, status, pageNum,
-pageSize);
+PageResult pageInfo =
groupClient.listGroups(expr, status, pageNum, pageSize);
if (CollectionUtils.isEmpty(pageInfo.getList())) {
return Lists.newArrayList();
-} else {
-return pageInfo.getList().stream().map(briefInfo -> {
-String groupId = briefInfo.getInlongGroupId();
-InlongGroupInfo groupInfo = groupClient.getGroupInfo(groupId);
-return new InlongGroupImpl(groupInfo, configuration);
-}).collect(Collectors.toList());
}
+
+return pageInfo.getList().stream()
+.map(info -> {
+String groupId = info.getInlongGroupId();
+InlongGroupInfo groupInfo =
groupClient.getGroupInfo(groupId);
+return new InlongGroupImpl(groupInfo, configuration);
+}).collect(Collectors.toList());
}
@Override
diff --git
a/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/impl/
[GitHub] [inlong] vernedeng commented on a diff in pull request #5843: [INLONG-5842][Manager] Support maintenance of message queue cluster
vernedeng commented on code in PR #5843:
URL: https://github.com/apache/inlong/pull/5843#discussion_r969099966
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/cluster/queue/MessageQueueServiceImpl.java:
##
@@ -0,0 +1,555 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.service.cluster.queue;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.gson.Gson;
+import com.google.gson.reflect.TypeToken;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.math.NumberUtils;
+import org.apache.inlong.common.msg.AttributeConstants;
+import org.apache.inlong.manager.common.consts.MQType;
+import org.apache.inlong.manager.dao.entity.InlongClusterEntity;
+import org.apache.inlong.manager.dao.entity.InlongGroupEntity;
+import org.apache.inlong.manager.dao.mapper.InlongClusterEntityMapper;
+import org.apache.inlong.manager.dao.mapper.InlongGroupEntityMapper;
+import
org.apache.inlong.manager.pojo.cluster.queue.MessageQueueClearTopicRequest;
+import org.apache.inlong.manager.pojo.cluster.queue.MessageQueueControlRequest;
+import org.apache.inlong.manager.pojo.cluster.queue.MessageQueueOfflineRequest;
+import org.apache.inlong.manager.pojo.cluster.queue.MessageQueueOnlineRequest;
+import
org.apache.inlong.manager.pojo.cluster.queue.MessageQueueSynchronizeTopicRequest;
+import org.apache.inlong.manager.pojo.cluster.tubemq.TubeClusterInfo;
+import org.apache.inlong.manager.pojo.group.InlongGroupPageRequest;
+import org.apache.inlong.manager.service.resource.queue.tubemq.TubeMQOperator;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.CreateTopicsResult;
+import org.apache.kafka.clients.admin.DeleteTopicsResult;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.pulsar.client.admin.PulsarAdmin;
+import org.apache.pulsar.client.admin.PulsarAdminBuilder;
+import org.apache.pulsar.client.admin.Topics;
+import org.apache.pulsar.client.api.AuthenticationFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Inlong message queue cluster operator.
+ */
+@Service
+public class MessageQueueServiceImpl implements MessageQueueService {
+
+public static final Logger LOG =
LoggerFactory.getLogger(MessageQueueServiceImpl.class);
+
+public static final Splitter.MapSplitter MAP_SPLITTER =
Splitter.on(AttributeConstants.SEPARATOR)
+
.trimResults().withKeyValueSeparator(AttributeConstants.KEY_VALUE_SEPARATOR);
+public static final Joiner.MapJoiner MAP_JOINER =
Joiner.on(AttributeConstants.SEPARATOR)
+.withKeyValueSeparator(AttributeConstants.KEY_VALUE_SEPARATOR);
+public static final String KEY_PRODUCER = "producer";
+public static final String KEY_CONSUMER = "consumer";
+public static final String KEY_ADMIN_URL = "adminUrl";
+public static final String KEY_AUTHENTICATION = "authentication";
+public static final String KEY_TENANT = "tenant";
+public static final String KEY_NAMESPACE = "namespace";
+public static final String KEY_NUM_PARTITIONS = "numPartitions";
+public static final String KEY_REPLICATION_FACTOR = "replicationFactor";
+public static final int DEFAULT_NUM_PARTITIONS = 10;
+public static final short DEFAULT_REPLICATION_FACTOR = 2;
+
+@Autowired
+private InlongClusterEntityMapper clusterMapper;
+@Autowired
+private InlongGroupEntityMapper groupMapper;
+@Autowired
+private TubeMQOperator tubeMQOperator;
+
+/**
+ * Control produce operation and consume operation of Inlong message queue
cluster
+ */
+@Override
+public String control(MessageQueueControlRequest request) {
+String name = request.getName();
+// check parameters
+if (StringUtils.isEmpty(name)) {
+return "miss cluster name
[inlong] branch release-1.3.0 updated: [INLONG-5861][Agent] Support symbolic links for file (#5862)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch release-1.3.0
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/release-1.3.0 by this push:
new bce8b5432 [INLONG-5861][Agent] Support symbolic links for file (#5862)
bce8b5432 is described below
commit bce8b543208351b557fb384c11d34d1ad0d3a505
Author: ganfengtan
AuthorDate: Tue Sep 13 11:01:29 2022 +0800
[INLONG-5861][Agent] Support symbolic links for file (#5862)
---
.../sources/reader/file/MonitorTextFile.java | 28 --
1 file changed, 21 insertions(+), 7 deletions(-)
diff --git
a/inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/file/MonitorTextFile.java
b/inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/file/MonitorTextFile.java
index dfad65450..0c6f6a72e 100644
---
a/inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/file/MonitorTextFile.java
+++
b/inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/file/MonitorTextFile.java
@@ -82,6 +82,7 @@ public final class MonitorTextFile {
private final TextFileReader textFileReader;
private final Long interval;
private final long startTime = System.currentTimeMillis();
+private String path;
/**
* the last modify time of the file
*/
@@ -95,6 +96,7 @@ public final class MonitorTextFile {
try {
this.attributesBefore = Files
.readAttributes(fileReaderOperator.file.toPath(),
BasicFileAttributes.class);
+this.path = this.fileReaderOperator.file.getCanonicalPath();
} catch (IOException e) {
LOGGER.error("get {} last modify time error:",
fileReaderOperator.file.getName(), e);
}
@@ -121,17 +123,29 @@ public final class MonitorTextFile {
}
private void listen() throws IOException {
-BasicFileAttributes attributesAfter = Files
-.readAttributes(this.fileReaderOperator.file.toPath(),
BasicFileAttributes.class);
+BasicFileAttributes attributesAfter;
+String currentPath;
+try {
+attributesAfter = Files
+.readAttributes(this.fileReaderOperator.file.toPath(),
BasicFileAttributes.class);
+currentPath = this.fileReaderOperator.file.getCanonicalPath();
+} catch (Exception e) {
+// Set position 0 when split file
+this.fileReaderOperator.position = 0;
+LOGGER.error("monitor {} error, reset position is 0:",
this.fileReaderOperator.file.getName(), e);
+return;
+}
+// If change symbolic links
+if (attributesAfter.isSymbolicLink() && !path.equals(currentPath))
{
+this.fileReaderOperator.position = 0;
+path = currentPath;
+}
if
(attributesBefore.lastModifiedTime().compareTo(attributesAfter.lastModifiedTime())
< 0) {
// Not triggered during data sending
-if (Objects.nonNull(this.fileReaderOperator.iterator) &&
this.fileReaderOperator.iterator.hasNext()) {
+if (Objects.nonNull(this.fileReaderOperator.iterator) &&
this.fileReaderOperator.iterator
+.hasNext()) {
return;
}
-// Set position 0 when split file
-if
(attributesBefore.creationTime().compareTo(attributesAfter.creationTime()) < 0)
{
-this.fileReaderOperator.position = 0;
-}
this.textFileReader.getData();
this.textFileReader.mergeData(this.fileReaderOperator);
this.attributesBefore = attributesAfter;
[inlong] branch release-1.3.0 updated: [INLONG-5849][Manager] Update the inlong group ext info by the client (#5850)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch release-1.3.0
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/release-1.3.0 by this push:
new d35a4df5b [INLONG-5849][Manager] Update the inlong group ext info by
the client (#5850)
d35a4df5b is described below
commit d35a4df5b9b1f0a9716e205023f1562828e6b2ab
Author: healchow
AuthorDate: Tue Sep 13 11:03:22 2022 +0800
[INLONG-5849][Manager] Update the inlong group ext info by the client
(#5850)
---
.../inlong/manager/client/cli/CreateCommand.java | 4 +-
.../manager/client/api/impl/InlongClientImpl.java | 20 +-
.../manager/client/api/impl/InlongGroupImpl.java | 45 +++---
.../client/api/util/InlongGroupTransfer.java | 7 +---
4 files changed, 37 insertions(+), 39 deletions(-)
diff --git
a/inlong-manager/manager-client-tools/src/main/java/org/apache/inlong/manager/client/cli/CreateCommand.java
b/inlong-manager/manager-client-tools/src/main/java/org/apache/inlong/manager/client/cli/CreateCommand.java
index 2a099f958..24b46f34a 100644
---
a/inlong-manager/manager-client-tools/src/main/java/org/apache/inlong/manager/client/cli/CreateCommand.java
+++
b/inlong-manager/manager-client-tools/src/main/java/org/apache/inlong/manager/client/cli/CreateCommand.java
@@ -70,9 +70,9 @@ public class CreateCommand extends AbstractCommand {
return;
}
}
-// first extract groupconfig from the file passed in
+// first extract group config from the file passed in
CreateGroupConf groupConf = objectMapper.readValue(content,
CreateGroupConf.class);
-// get the correspodning inlonggroup, a.k.a the task to execute
+// get the corresponding inlong group, aka the task to execute
InlongClient inlongClient = ClientUtils.getClient();
InlongGroup group =
inlongClient.forGroup(groupConf.getGroupInfo());
InlongStreamBuilder streamBuilder =
group.createStream(groupConf.getStreamInfo());
diff --git
a/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/impl/InlongClientImpl.java
b/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/impl/InlongClientImpl.java
index eac22c0fa..9e84a30ef 100644
---
a/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/impl/InlongClientImpl.java
+++
b/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/impl/InlongClientImpl.java
@@ -28,12 +28,12 @@ import
org.apache.inlong.manager.client.api.ClientConfiguration;
import org.apache.inlong.manager.client.api.InlongClient;
import org.apache.inlong.manager.client.api.InlongCluster;
import org.apache.inlong.manager.client.api.InlongGroup;
-import org.apache.inlong.manager.common.enums.SimpleGroupStatus;
-import org.apache.inlong.manager.common.enums.SimpleSourceStatus;
import org.apache.inlong.manager.client.api.inner.client.ClientFactory;
import org.apache.inlong.manager.client.api.inner.client.InlongClusterClient;
import org.apache.inlong.manager.client.api.inner.client.InlongGroupClient;
import org.apache.inlong.manager.client.api.util.ClientUtils;
+import org.apache.inlong.manager.common.enums.SimpleGroupStatus;
+import org.apache.inlong.manager.common.enums.SimpleSourceStatus;
import org.apache.inlong.manager.common.util.HttpUtils;
import org.apache.inlong.manager.common.util.Preconditions;
import org.apache.inlong.manager.pojo.cluster.BindTagRequest;
@@ -108,17 +108,17 @@ public class InlongClientImpl implements InlongClient {
@Override
public List listGroup(String expr, int status, int pageNum,
int pageSize) {
-PageResult pageInfo =
groupClient.listGroups(expr, status, pageNum,
-pageSize);
+PageResult pageInfo =
groupClient.listGroups(expr, status, pageNum, pageSize);
if (CollectionUtils.isEmpty(pageInfo.getList())) {
return Lists.newArrayList();
-} else {
-return pageInfo.getList().stream().map(briefInfo -> {
-String groupId = briefInfo.getInlongGroupId();
-InlongGroupInfo groupInfo = groupClient.getGroupInfo(groupId);
-return new InlongGroupImpl(groupInfo, configuration);
-}).collect(Collectors.toList());
}
+
+return pageInfo.getList().stream()
+.map(info -> {
+String groupId = info.getInlongGroupId();
+InlongGroupInfo groupInfo =
groupClient.getGroupInfo(groupId);
+return new InlongGroupImpl(groupInfo, configuration);
+}).collect(Collectors.toList());
}
@Override
diff --git
a/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/cl
[GitHub] [inlong-website] thesumery commented on a diff in pull request #538: [INLONG-537][Sort][Agent] Add TubeMQ connector docs and update Agent docs
thesumery commented on code in PR #538:
URL: https://github.com/apache/inlong-website/pull/538#discussion_r969114631
##
docs/data_node/extract_node/tube.md:
##
@@ -0,0 +1,70 @@
+---
+title: TubeMQ
+sidebar_position: 11
+---
+
+import {siteVariables} from '../../version';
Review Comment:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong-website] thesumery commented on a diff in pull request #538: [INLONG-537][Sort][Agent] Add TubeMQ connector docs and update Agent docs
thesumery commented on code in PR #538:
URL: https://github.com/apache/inlong-website/pull/538#discussion_r969114752
##
docs/data_node/extract_node/tube.md:
##
@@ -0,0 +1,70 @@
+---
+title: TubeMQ
+sidebar_position: 11
+---
+
+import {siteVariables} from '../../version';
Review Comment:
what does it means?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong-website] thesumery commented on a diff in pull request #538: [INLONG-537][Sort][Agent] Add TubeMQ connector docs and update Agent docs
thesumery commented on code in PR #538:
URL: https://github.com/apache/inlong-website/pull/538#discussion_r969115052
##
docs/data_node/extract_node/tube.md:
##
@@ -0,0 +1,70 @@
+---
+title: TubeMQ
+sidebar_position: 11
+---
+
+import {siteVariables} from '../../version';
+
+## Overview
+
+[Apache InLong TubeMQ](https://inlong.apache.org/docs/modules/tubemq/overview)
is a distributed, open source pub-sub messaging and steaming platform for
real-time workloads, trillions of massive data precipitation.
+
+## Version
+
+| Extract Node | Version
|
+| - |
|
+| [TubeMQ](./tube.md) |
[TubeMQ](https://inlong.apache.org/docs/next/modules/tubemq/overview):
>=0.1.0 |
+
+## Dependencies
+
+In order to set up the `TubeMQ Extract Node`, the following provides
dependency information for both projects using a
+build automation tool (such as Maven or SBT) and SQL Client with Sort
Connectors JAR bundles.
+
+### Maven dependency
+
+
+{`
Review Comment:
wrong format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong-website] thesumery commented on a diff in pull request #538: [INLONG-537][Sort][Agent] Add TubeMQ connector docs and update Agent docs
thesumery commented on code in PR #538:
URL: https://github.com/apache/inlong-website/pull/538#discussion_r969115316
##
docs/data_node/extract_node/tube.md:
##
@@ -0,0 +1,70 @@
+---
+title: TubeMQ
+sidebar_position: 11
+---
+
+import {siteVariables} from '../../version';
+
+## Overview
+
+[Apache InLong TubeMQ](https://inlong.apache.org/docs/modules/tubemq/overview)
is a distributed, open source pub-sub messaging and steaming platform for
real-time workloads, trillions of massive data precipitation.
+
+## Version
+
+| Extract Node | Version
|
+| - |
|
+| [TubeMQ](./tube.md) |
[TubeMQ](https://inlong.apache.org/docs/next/modules/tubemq/overview):
>=0.1.0 |
Review Comment:
Extra `:`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong-website] Oneal65 commented on a diff in pull request #540: [INLONG-539][Sort] Modify doc for support routing field for Elasticsearch connector
Oneal65 commented on code in PR #540: URL: https://github.com/apache/inlong-website/pull/540#discussion_r969117049 ## i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/load_node/elasticsearch.md: ## @@ -229,6 +229,13 @@ TODO: 将在未来支持这个特性。 String 添加到每个 REST 通信中的前缀字符串，例如，'/v1'。 + + routing.filed-name + 可选 + (none) + String + 使用 field 值来生成该field的动态路由 Review Comment: OK -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] thesumery commented on a diff in pull request #538: [INLONG-537][Sort][Agent] Add TubeMQ connector docs and update Agent docs
thesumery commented on code in PR #538:
URL: https://github.com/apache/inlong-website/pull/538#discussion_r969117480
##
docs/data_node/extract_node/tube.md:
##
@@ -0,0 +1,70 @@
+---
+title: TubeMQ
+sidebar_position: 11
+---
+
+import {siteVariables} from '../../version';
+
+## Overview
+
+[Apache InLong TubeMQ](https://inlong.apache.org/docs/modules/tubemq/overview)
is a distributed, open source pub-sub messaging and steaming platform for
real-time workloads, trillions of massive data precipitation.
+
+## Version
+
+| Extract Node | Version
|
+| - |
|
+| [TubeMQ](./tube.md) |
[TubeMQ](https://inlong.apache.org/docs/next/modules/tubemq/overview):
>=0.1.0 |
+
+## Dependencies
+
+In order to set up the `TubeMQ Extract Node`, the following provides
dependency information for both projects using a
+build automation tool (such as Maven or SBT) and SQL Client with Sort
Connectors JAR bundles.
+
+### Maven dependency
+
+
+{`
Review Comment:
Please correct the above format errors in the Chinese document in the same
time.
##
docs/data_node/extract_node/tube.md:
##
@@ -0,0 +1,70 @@
+---
+title: TubeMQ
+sidebar_position: 11
+---
+
+import {siteVariables} from '../../version';
+
+## Overview
+
+[Apache InLong TubeMQ](https://inlong.apache.org/docs/modules/tubemq/overview)
is a distributed, open source pub-sub messaging and steaming platform for
real-time workloads, trillions of massive data precipitation.
+
+## Version
+
+| Extract Node | Version
|
+| - |
|
+| [TubeMQ](./tube.md) |
[TubeMQ](https://inlong.apache.org/docs/next/modules/tubemq/overview):
>=0.1.0 |
Review Comment:
Please correct the above format errors in the Chinese document in the same
time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong-website] thesumery commented on a diff in pull request #538: [INLONG-537][Sort][Agent] Add TubeMQ connector docs and update Agent docs
thesumery commented on code in PR #538: URL: https://github.com/apache/inlong-website/pull/538#discussion_r969117989 ## docs/modules/agent/overview.md: ## @@ -25,6 +25,21 @@ User-configured path monitoring, able to monitor the created file information Directory regular filtering, support MMDD+regular expression path configuration Breakpoint retransmission, when InLong-Agent restarts, it can automatically re-read from the last read position to ensure no reread or missed reading. + File options +| Parameter | Required | Default value | Type | Description | +| - | | - | -- | | +| pattern | required | (none)| String | File pattern. For example: /root/[*].log | +| timeOffset| optional | (none)| String | File name includes time, for example: *** MMDDHH *** represents the year, MM represents the month, DD represents the day, and HH represents the hour, *** is any character. '1m' means one minute after, '-1m' means one minute before. '1h' means one hour after, '-1h' means one hour before. '1d' means one day after, '-1d' means one day before.| Review Comment: *** MMDDHH *** should be ` MMDDHH ` ？ ## docs/modules/agent/overview.md: ## @@ -25,6 +25,21 @@ User-configured path monitoring, able to monitor the created file information Directory regular filtering, support MMDD+regular expression path configuration Breakpoint retransmission, when InLong-Agent restarts, it can automatically re-read from the last read position to ensure no reread or missed reading. + File options +| Parameter | Required | Default value | Type | Description | +| - | | - | -- | | +| pattern | required | (none)| String | File pattern. For example: /root/[*].log | +| timeOffset| optional | (none)| String | File name includes time, for example: *** MMDDHH *** represents the year, MM represents the month, DD represents the day, and HH represents the hour, *** is any character. '1m' means one minute after, '-1m' means one minute before. '1h' means one hour after, '-1h' means one hour before. '1d' means one day after, '-1d' means one day before.| Review Comment: Please correct the above format errors in the Chinese document in the same time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] thesumery commented on a diff in pull request #536: [INLONG-535][Sort] Add doris docs
thesumery commented on code in PR #536:
URL: https://github.com/apache/inlong-website/pull/536#discussion_r969121589
##
docs/data_node/extract_node/doris.md:
##
@@ -0,0 +1,152 @@
+---
+title: Doris
+sidebar_position: 11
+---
+
+import {siteVariables} from '../../version';
+
+## Overview
+
+`Doris Extract` node supports reading data from Doris. This chapter describes
how to set up a Doris Extract
+node to run SQL queries against the Doris database.
+
+## Supported Version
+
+| Extract Node| Doris version |
+|-||
+| [Doris](./doris.md) | 0.13+ |
+
+## Dependencies
+
+In order to set up the Doris Extract node, the dependency information needed
to use build automation tools
+such as Maven or SBT is provided below.
+
+### Maven dependency
+
+
+{`
Review Comment:
wrong format
##
docs/data_node/extract_node/doris.md:
##
@@ -0,0 +1,152 @@
+---
+title: Doris
+sidebar_position: 11
+---
+
+import {siteVariables} from '../../version';
+
+## Overview
+
+`Doris Extract` node supports reading data from Doris. This chapter describes
how to set up a Doris Extract
+node to run SQL queries against the Doris database.
+
+## Supported Version
+
+| Extract Node| Doris version |
+|-||
+| [Doris](./doris.md) | 0.13+ |
+
+## Dependencies
+
+In order to set up the Doris Extract node, the dependency information needed
to use build automation tools
+such as Maven or SBT is provided below.
+
+### Maven dependency
+
+
+{`
Review Comment:
Please correct the above format errors in the Chinese document in the same
time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong-website] dockerzhang merged pull request #548: [INLONG-521][Release] Modify download & release notes for 1.3.0
dockerzhang merged PR #548: URL: https://github.com/apache/inlong-website/pull/548 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] dockerzhang closed issue #521: [Release] Modify download & release notes for 1.3.0
dockerzhang closed issue #521: [Release] Modify download & release notes for 1.3.0 URL: https://github.com/apache/inlong-website/issues/521 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] lucaspeng12138 opened a new pull request, #5871: [INLONG-5870] Change sh to bash
lucaspeng12138 opened a new pull request, #5871: URL: https://github.com/apache/inlong/pull/5871 ### Prepare a Pull Request [[INLONG-5870] Change sh to bash - Fixes #5870 ### Motivation Change sh to bash ### Modifications Change sh to bash ### Verifying this change Change sh to bash -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] thesumery commented on a diff in pull request #538: [INLONG-537][Sort][Agent] Add TubeMQ connector docs and update Agent docs
thesumery commented on code in PR #538:
URL: https://github.com/apache/inlong-website/pull/538#discussion_r969114752
##
docs/data_node/extract_node/tube.md:
##
@@ -0,0 +1,70 @@
+---
+title: TubeMQ
+sidebar_position: 11
+---
+
+import {siteVariables} from '../../version';
Review Comment:
what does it means?
##
docs/data_node/extract_node/tube.md:
##
@@ -0,0 +1,70 @@
+---
+title: TubeMQ
+sidebar_position: 11
+---
+
+import {siteVariables} from '../../version';
Review Comment:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong-website] thesumery commented on a diff in pull request #538: [INLONG-537][Sort][Agent] Add TubeMQ connector docs and update Agent docs
thesumery commented on code in PR #538:
URL: https://github.com/apache/inlong-website/pull/538#discussion_r969117480
##
docs/data_node/extract_node/tube.md:
##
@@ -0,0 +1,70 @@
+---
+title: TubeMQ
+sidebar_position: 11
+---
+
+import {siteVariables} from '../../version';
+
+## Overview
+
+[Apache InLong TubeMQ](https://inlong.apache.org/docs/modules/tubemq/overview)
is a distributed, open source pub-sub messaging and steaming platform for
real-time workloads, trillions of massive data precipitation.
+
+## Version
+
+| Extract Node | Version
|
+| - |
|
+| [TubeMQ](./tube.md) |
[TubeMQ](https://inlong.apache.org/docs/next/modules/tubemq/overview):
>=0.1.0 |
+
+## Dependencies
+
+In order to set up the `TubeMQ Extract Node`, the following provides
dependency information for both projects using a
+build automation tool (such as Maven or SBT) and SQL Client with Sort
Connectors JAR bundles.
+
+### Maven dependency
+
+
+{`
Review Comment:
Please correct the above format errors in the Chinese document in the same
time.
##
docs/data_node/extract_node/tube.md:
##
@@ -0,0 +1,70 @@
+---
+title: TubeMQ
+sidebar_position: 11
+---
+
+import {siteVariables} from '../../version';
+
+## Overview
+
+[Apache InLong TubeMQ](https://inlong.apache.org/docs/modules/tubemq/overview)
is a distributed, open source pub-sub messaging and steaming platform for
real-time workloads, trillions of massive data precipitation.
+
+## Version
+
+| Extract Node | Version
|
+| - |
|
+| [TubeMQ](./tube.md) |
[TubeMQ](https://inlong.apache.org/docs/next/modules/tubemq/overview):
>=0.1.0 |
+
+## Dependencies
+
+In order to set up the `TubeMQ Extract Node`, the following provides
dependency information for both projects using a
+build automation tool (such as Maven or SBT) and SQL Client with Sort
Connectors JAR bundles.
+
+### Maven dependency
+
+
+{`
Review Comment:
wrong format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong-website] thesumery commented on a diff in pull request #536: [INLONG-535][Sort] Add doris docs
thesumery commented on code in PR #536:
URL: https://github.com/apache/inlong-website/pull/536#discussion_r969121618
##
docs/data_node/extract_node/doris.md:
##
@@ -0,0 +1,152 @@
+---
+title: Doris
+sidebar_position: 11
+---
+
+import {siteVariables} from '../../version';
+
+## Overview
+
+`Doris Extract` node supports reading data from Doris. This chapter describes
how to set up a Doris Extract
+node to run SQL queries against the Doris database.
+
+## Supported Version
+
+| Extract Node| Doris version |
+|-||
+| [Doris](./doris.md) | 0.13+ |
+
+## Dependencies
+
+In order to set up the Doris Extract node, the dependency information needed
to use build automation tools
+such as Maven or SBT is provided below.
+
+### Maven dependency
+
+
+{`
Review Comment:
Please correct the above format errors in the Chinese document in the same
time.
##
docs/data_node/extract_node/doris.md:
##
@@ -0,0 +1,152 @@
+---
+title: Doris
+sidebar_position: 11
+---
+
+import {siteVariables} from '../../version';
+
+## Overview
+
+`Doris Extract` node supports reading data from Doris. This chapter describes
how to set up a Doris Extract
+node to run SQL queries against the Doris database.
+
+## Supported Version
+
+| Extract Node| Doris version |
+|-||
+| [Doris](./doris.md) | 0.13+ |
+
+## Dependencies
+
+In order to set up the Doris Extract node, the dependency information needed
to use build automation tools
+such as Maven or SBT is provided below.
+
+### Maven dependency
+
+
+{`
Review Comment:
wrong format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] EMsnap commented on issue #5671: [Release] Modify download & release notes for 1.3.0
EMsnap commented on issue #5671: URL: https://github.com/apache/inlong/issues/5671#issuecomment-1244897334 see https://github.com/apache/inlong-website/pull/548 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] seedscoder commented on pull request #5674: [INLONG-5045][Agent] Support collect data from MongoDB
seedscoder commented on PR #5674: URL: https://github.com/apache/inlong/pull/5674#issuecomment-1244939627 @EMsnap Please review this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang commented on issue #5872: [Improve]建议增加standalone模式启动建议
dockerzhang commented on issue #5872: URL: https://github.com/apache/inlong/issues/5872#issuecomment-1244948264 thanks, @yg9538 . please add more details for your improvement suggestion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] yg9538 commented on issue #5872: [Improve]建议增加standalone模式启动建议
yg9538 commented on issue #5872: URL: https://github.com/apache/inlong/issues/5872#issuecomment-1244970739 我启动standalone模式时候dashboard会爆出502错误，经过排查是因为dashboard中的nginx配置文件中default.conf指向了127.0.0.1:8083，这个地址没法走通 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
