Re: [PR] [INLONG-9467][Agent] improve code exception detection to ensure task and instance state transitions [inlong]
EMsnap merged PR #9468: URL: https://github.com/apache/inlong/pull/9468 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-9467][Agent] Improve code exception detection to ensure task and instance state transitions (#9468)
This is an automated email from the ASF dual-hosted git repository.
zirui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new ef594ccf4a [INLONG-9467][Agent] Improve code exception detection to
ensure task and instance state transitions (#9468)
ef594ccf4a is described below
commit ef594ccf4a5e2f48fbea039e5db68e1ce225516e
Author: justinwwhuang
AuthorDate: Thu Dec 14 10:28:46 2023 +0800
[INLONG-9467][Agent] Improve code exception detection to ensure task and
instance state transitions (#9468)
---
.../inlong/agent/metrics/audit/AuditUtils.java | 1 +
.../org/apache/inlong/agent/plugin/Instance.java | 2 +-
.../agent/core/instance/InstanceManager.java | 25 +++--
.../inlong/agent/core/instance/MockInstance.java | 3 +-
.../inlong/agent/plugin/instance/FileInstance.java | 22 +++--
.../inlong/agent/plugin/sources/LogFileSource.java | 110 +++--
.../task/filecollect/LogFileCollectTask.java | 10 +-
7 files changed, 104 insertions(+), 69 deletions(-)
diff --git
a/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/metrics/audit/AuditUtils.java
b/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/metrics/audit/AuditUtils.java
index 290d3b71bb..c2d946b923 100644
---
a/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/metrics/audit/AuditUtils.java
+++
b/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/metrics/audit/AuditUtils.java
@@ -55,6 +55,7 @@ public class AuditUtils {
public static final int AUDIT_ID_AGENT_INSTANCE_HEARTBEAT = 30010;
public static final int AUDIT_ID_AGENT_SEND_FAILED_REAL_TIME = 30011;
public static final int AUDIT_ID_AGENT_DEL_INSTANCE_MEM_UNUSUAL = 30014;
+public static final int AUDIT_ID_AGENT_ADD_INSTANCE_MEM_FAILED = 30015;
private static boolean IS_AUDIT = true;
diff --git
a/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/plugin/Instance.java
b/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/plugin/Instance.java
index 90bac4c94f..990d7e60b2 100755
---
a/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/plugin/Instance.java
+++
b/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/plugin/Instance.java
@@ -32,7 +32,7 @@ public abstract class Instance extends AbstractStateWrapper {
*
* @throws IOException
*/
-public abstract void init(Object instanceManager, InstanceProfile profile);
+public abstract boolean init(Object instanceManager, InstanceProfile
profile);
/**
* destroy instance.
diff --git
a/inlong-agent/agent-core/src/main/java/org/apache/inlong/agent/core/instance/InstanceManager.java
b/inlong-agent/agent-core/src/main/java/org/apache/inlong/agent/core/instance/InstanceManager.java
index 3a86f32fc2..3b74cf4e48 100644
---
a/inlong-agent/agent-core/src/main/java/org/apache/inlong/agent/core/instance/InstanceManager.java
+++
b/inlong-agent/agent-core/src/main/java/org/apache/inlong/agent/core/instance/InstanceManager.java
@@ -429,15 +429,22 @@ public class InstanceManager extends AbstractDaemon {
try {
Class taskClass =
Class.forName(instanceProfile.getInstanceClass());
Instance instance = (Instance) taskClass.newInstance();
-instance.init(this, instanceProfile);
-instanceMap.put(instanceProfile.getInstanceId(), instance);
-EXECUTOR_SERVICE.submit(instance);
-LOGGER.info(
-"add instance to memory instanceId {} instanceMap size {},
runningPool instance total {}, runningPool instance active {}",
-instance.getInstanceId(), instanceMap.size(),
EXECUTOR_SERVICE.getTaskCount(),
-EXECUTOR_SERVICE.getActiveCount());
-AuditUtils.add(AuditUtils.AUDIT_ID_AGENT_ADD_INSTANCE_MEM,
inlongGroupId, inlongStreamId,
-instanceProfile.getSinkDataTime(), 1, 1);
+boolean initSuc = instance.init(this, instanceProfile);
+if (initSuc) {
+instanceMap.put(instanceProfile.getInstanceId(), instance);
+EXECUTOR_SERVICE.submit(instance);
+LOGGER.info(
+"add instance to memory instanceId {} instanceMap size
{}, runningPool instance total {}, runningPool instance active {}",
+instance.getInstanceId(), instanceMap.size(),
EXECUTOR_SERVICE.getTaskCount(),
+EXECUTOR_SERVICE.getActiveCount());
+AuditUtils.add(AuditUtils.AUDIT_ID_AGENT_ADD_INSTANCE_MEM,
inlongGroupId, inlongStreamId,
+instanceProfile.getSinkDataTime(), 1, 1);
+} else {
+LOGGER.error(
+"add instance to memory init failed instanceId {}",
instance.getInstanceId());
+
Re: [PR] [INLONG-9470][Manager] Fix the problem of failed to verify if the namespace exists [inlong]
vernedeng commented on code in PR #9471:
URL: https://github.com/apache/inlong/pull/9471#discussion_r1426214534
##
inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/util/HttpUtils.java:
##
@@ -95,6 +95,36 @@ public static T request(RestTemplate restTemplate,
String url, HttpMethod me
}
}
+/**
+ * Send an HTTP request by the given rest template.
+ */
+public static T request(RestTemplate restTemplate, String[] urls,
HttpMethod method,
+String param, HttpHeaders header, Class cls) throws Exception {
+ResponseEntity exchange;
+for (int i = 0; i < urls.length; i++) {
Review Comment:
NPE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-9470][Manager] Fix the problem of failed to verify if the namespace exists [inlong]
vernedeng commented on code in PR #9471:
URL: https://github.com/apache/inlong/pull/9471#discussion_r1426215616
##
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/cluster/pulsar/PulsarClusterInfo.java:
##
@@ -55,4 +58,13 @@ public PulsarClusterRequest genRequest() {
return CommonBeanUtils.copyProperties(this, PulsarClusterRequest::new);
}
+public String[] getAdminUrls(String urlSuffix) {
+String adminUrl = this.getAdminUrl();
Review Comment:
NPE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-9470][Manager] Fix the problem of failed to verify if the namespace exists [inlong]
fuweng11 commented on code in PR #9471:
URL: https://github.com/apache/inlong/pull/9471#discussion_r1426218242
##
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/cluster/pulsar/PulsarClusterInfo.java:
##
@@ -55,4 +58,13 @@ public PulsarClusterRequest genRequest() {
return CommonBeanUtils.copyProperties(this, PulsarClusterRequest::new);
}
+public String[] getAdminUrls(String urlSuffix) {
+String adminUrl = this.getAdminUrl();
Review Comment:
Done.
##
inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/util/HttpUtils.java:
##
@@ -95,6 +95,36 @@ public static T request(RestTemplate restTemplate,
String url, HttpMethod me
}
}
+/**
+ * Send an HTTP request by the given rest template.
+ */
+public static T request(RestTemplate restTemplate, String[] urls,
HttpMethod method,
+String param, HttpHeaders header, Class cls) throws Exception {
+ResponseEntity exchange;
+for (int i = 0; i < urls.length; i++) {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [INLONG-9473][Sort] Support transform of embedding for LLM applications [inlong]
aloyszhang opened a new pull request, #9474: URL: https://github.com/apache/inlong/pull/9474 - Fixes #9473 ### Motivation Support transform of embedding for LLM applications ### Modifications - add a new transform UDF named `EmbeddingFunction` - add test in `EmbeddingFunctionTest` ### Verifying this change - [ ] This change added tests and can be verified as follows: Added new test `EmbeddingFunctionTest` ### Documentation No docs needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
