Re: [PR] [INLONG-9806][Agent] Add installer configuration file [inlong]
dockerzhang merged PR #9815: URL: https://github.com/apache/inlong/pull/9815 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-9806][Agent] Add installer configuration file (#9815)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new b08d1612b0 [INLONG-9806][Agent] Add installer configuration file
(#9815)
b08d1612b0 is described below
commit b08d1612b07b218755c2f4b630bc34f67a178067
Author: justinwwhuang
AuthorDate: Thu Mar 14 16:32:08 2024 +0800
[INLONG-9806][Agent] Add installer configuration file (#9815)
---
inlong-agent/agent-installer/assembly.xml | 63
.../agent-installer/conf/installer.properties | 43 +
inlong-agent/agent-installer/conf/log4j2.xml | 175 +
inlong-agent/agent-installer/pom.xml | 38 +
4 files changed, 319 insertions(+)
diff --git a/inlong-agent/agent-installer/assembly.xml
b/inlong-agent/agent-installer/assembly.xml
new file mode 100644
index 00..ad8b0cef4f
--- /dev/null
+++ b/inlong-agent/agent-installer/assembly.xml
@@ -0,0 +1,63 @@
+
+http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0
http://maven.apache.org/xsd/assembly-1.1.0.xsd";>
+bin
+
+dir
+tar.gz
+
+
+false
+
+
+
+../bin
+
+*.*
+
+
+*rock*
+
+0755
+bin
+unix
+
+
+
+
+conf
+
+*.*
+
+conf
+unix
+
+
+
+
+
+false
+true
+lib
+
+
+
diff --git a/inlong-agent/agent-installer/conf/installer.properties
b/inlong-agent/agent-installer/conf/installer.properties
new file mode 100755
index 00..52b4ce071f
--- /dev/null
+++ b/inlong-agent/agent-installer/conf/installer.properties
@@ -0,0 +1,43 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+###
+# common config
+###
+local.ip=127.0.0.1
+agent.enable.oom.exit=false
+
+
+# manager config
+
+manager.addr=http://127.0.0.1:8083
+manager.auth.secretId=
+manager.auth.secretKey=
+
+
+# cluster config for automatically report and register
+
+cluster.tag=default_cluster
+cluster.name=default_agent
+
+
+# audit config
+
+# whether to enable audit
+audit.enable=true
+# audit proxy address
+audit.proxys=127.0.0.1:10081
diff --git a/inlong-agent/agent-installer/conf/log4j2.xml
b/inlong-agent/agent-installer/conf/log4j2.xml
new file mode 100644
index 00..3db2c5dee3
--- /dev/null
+++ b/inlong-agent/agent-installer/conf/log4j2.xml
@@ -0,0 +1,175 @@
+
+
+
+
+${sys:agent.home}/logs
+[%X{trace_id} %X{span_id}] %d{-MM-dd
HH:mm:ss.SSS} -%5p ${PID:-} [%30.30t] %-30.30C{1.}:%L %m%n
+1
+1G
+INFO
+15
+${basePath}/debug.log
+${basePath}/debug-%d{-MM-dd}-%i.log.gz
+10
+${basePath}/info.log
+${basePath}/info-%d{-MM-dd}-%i.log.gz
+10
+${basePath}/warn.log
+${basePath}/warn-%d{-MM-dd}-%i.log.gz
+10
+${basePath}/error.log
+${basePath}/error-%d{-MM-dd}-%i.log.gz
+10
+INFO
+10
+${basePath}/index.log
+${basePath}/index-%d{-MM-dd}-%i.log.gz
+10
+${basePath}/monitors.log
+${basePath}/monitors-%d{-MM-dd}-%i.log.gz
+15d
+
+
+
+
+
+
+
+
+
+
+
+
+]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Re: [PR] [INLONG-9818][Manager] Decode Msg based on the manager's configuration [inlong]
dockerzhang merged PR #9819: URL: https://github.com/apache/inlong/pull/9819 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-9818][Manager] Decode Msg based on the manager's configuration (#9819)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 58b2fc963c [INLONG-9818][Manager] Decode Msg based on the manager's
configuration (#9819)
58b2fc963c is described below
commit 58b2fc963cad0715ab0d4a82de50a9c95fd37159
Author: fuweng11 <76141879+fuwen...@users.noreply.github.com>
AuthorDate: Thu Mar 14 16:45:53 2024 +0800
[INLONG-9818][Manager] Decode Msg based on the manager's configuration
(#9819)
---
.../manager/service/resource/queue/pulsar/PulsarOperator.java | 10 ++
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/queue/pulsar/PulsarOperator.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/queue/pulsar/PulsarOperator.java
index 99653c05c1..ccabb716f1 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/queue/pulsar/PulsarOperator.java
+++
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/queue/pulsar/PulsarOperator.java
@@ -453,10 +453,12 @@ public class PulsarOperator {
messagePosition);
PulsarMessageInfo messageInfo =
PulsarUtils.getMessageFromHttpResponse(httpResponse, topicPartition);
Map headers = messageInfo.getProperties();
-int wrapTypeId =
Integer.parseInt(headers.getOrDefault(InlongConstants.MSG_ENCODE_VER,
-Integer.toString(MessageWrapType.INLONG_MSG_V0.getId(;
-DeserializeOperator deserializeOperator =
deserializeOperatorFactory.getInstance(
-MessageWrapType.valueOf(wrapTypeId));
+MessageWrapType messageWrapType =
MessageWrapType.forType(streamInfo.getWrapType());
+if (headers.get(InlongConstants.MSG_ENCODE_VER) != null) {
+messageWrapType =
+
MessageWrapType.valueOf(Integer.parseInt(headers.get(InlongConstants.MSG_ENCODE_VER)));
+}
+DeserializeOperator deserializeOperator =
deserializeOperatorFactory.getInstance(messageWrapType);
briefMQMessages.addAll(deserializeOperator.decodeMsg(streamInfo,
messageInfo.getBody(),
headers, index));
} catch (Exception e) {
Re: [PR] [INLONG-8676][Manager] Elasticsearch - Modify the calling method from SDK to HTTP [inlong]
fuweng11 commented on code in PR #9057:
URL: https://github.com/apache/inlong/pull/9057#discussion_r1524524352
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/sink/es/ElasticsearchConfig.java:
##
@@ -56,54 +58,84 @@ public class ElasticsearchConfig {
private String password;
/**
- * highLevelClient
+ * Get http rest client.
*
- * @return RestHighLevelClient
+ * @return springframework RestTemplate
*/
-public RestHighLevelClient highLevelClient() {
-if (highLevelClient != null) {
-return highLevelClient;
+public RestTemplate getRestClient() {
+if (restTemplate != null) {
+return restTemplate;
}
try {
-synchronized (RestHighLevelClient.class) {
-if (highLevelClient == null) {
-List hosts = new ArrayList<>();
-String[] hostArrays =
this.hosts.split(InlongConstants.SEMICOLON);
-for (String host : hostArrays) {
-if (StringUtils.isNotBlank(host)) {
-host = host.trim();
-hosts.add(HttpHost.create(host));
-}
-}
-RestClientBuilder clientBuilder =
RestClient.builder(hosts.toArray(new HttpHost[0]));
-this.setEsAuth(clientBuilder);
-highLevelClient = new RestHighLevelClient(clientBuilder);
+synchronized (RestTemplate.class) {
+if (restTemplate == null) {
+restTemplate = new RestTemplate();
}
}
} catch (Exception e) {
logger.error("get es high level client error", e);
}
-return highLevelClient;
+return restTemplate;
}
/**
- * Elasticsearch authentication
+ * Get http hosts.
*
- * @param builder The builder
+ * @return list of http host info
*/
-private void setEsAuth(RestClientBuilder builder) {
+public List getHttpHosts() {
+if (httpHosts != null) {
+return httpHosts;
+}
try {
-logger.info("set es auth of enable={}", authEnable);
-if (authEnable) {
-final CredentialsProvider credentialsProvider = new
BasicCredentialsProvider();
-credentialsProvider.setCredentials(AuthScope.ANY, new
UsernamePasswordCredentials(username, password));
-builder.setHttpClientConfigCallback(
-httpClientBuilder ->
httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider));
-
+synchronized (HttpHost.class) {
+httpHosts = new ArrayList<>();
+String[] hostArrays =
this.hosts.split(InlongConstants.SEMICOLON);
+for (String host : hostArrays) {
+if (StringUtils.isNotBlank(host)) {
+host = host.trim();
+httpHosts.add(HttpHost.create(host));
+}
+}
}
} catch (Exception e) {
-logger.error("set es auth error ", e);
+logger.error("get es http hosts error", e);
+}
+return httpHosts;
+}
+
+/**
+ * Get one http url.
+ *
+ * @return a http url
+ * @throws Exception any exception if occurred
+ */
+public String getOneHttpUrl() throws Exception {
+getHttpHosts();
+if (!httpHosts.isEmpty() && httpHosts.size() > 0) {
+return httpHosts.get(rand.nextInt(httpHosts.size())).toString();
+} else {
+throw new Exception("http hosts is empty! please check hosts!");
}
Review Comment:
```suggestion
getHttpHosts();
if (!httpHosts.isEmpty() && httpHosts.size() > 0) {
return httpHosts.get(rand.nextInt(httpHosts.size())).toString();
}
throw new Exception("http hosts is empty! please check hosts!");
```
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/sink/es/ElasticsearchConfig.java:
##
@@ -56,54 +58,84 @@ public class ElasticsearchConfig {
private String password;
/**
- * highLevelClient
+ * Get http rest client.
*
- * @return RestHighLevelClient
+ * @return springframework RestTemplate
*/
-public RestHighLevelClient highLevelClient() {
-if (highLevelClient != null) {
-return highLevelClient;
+public RestTemplate getRestClient() {
+if (restTemplate != null) {
+return restTemplate;
}
try {
-synchronized (RestHighLevelClient.class) {
-if (highLevelClient == null) {
-List hosts = new ArrayList<>();
-String[] hos
[PR] [INLONG-9820][Dashboard][Manager] Update Pulsar source field for Ingestion [inlong]
haifxu opened a new pull request, #9821: URL: https://github.com/apache/inlong/pull/9821 ### Prepare a Pull Request - Fixes #9820 ### Motivation 1. Dashboard add fields `subscription`, `scanStartupMode`, `subscriptionType`, `resetTime` and `dataTimeZone` for Ingestion. 2. Add fields `subscriptionType` and `resetTime` in Manager. 3. Change `resetTime` field type and `subscriptionPosition` name for better interaction with the manager. ### Modifications  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-9820][Dashboard][Manager] Update Pulsar source field for Ingestion [inlong]
leezng commented on code in PR #9821:
URL: https://github.com/apache/inlong/pull/9821#discussion_r1524635464
##
inlong-dashboard/src/plugins/sources/defaults/Pulsar.ts:
##
@@ -21,6 +21,8 @@ import { DataWithBackend } from '@/plugins/DataWithBackend';
import { RenderRow } from '@/plugins/RenderRow';
import { RenderList } from '@/plugins/RenderList';
import { SourceInfo } from '../common/SourceInfo';
+import dayjs from 'dayjs';
Review Comment:
Is dayjs useless?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-9820][Dashboard][Manager] Update Pulsar source field for Ingestion [inlong]
dockerzhang commented on code in PR #9821:
URL: https://github.com/apache/inlong/pull/9821#discussion_r1524668917
##
inlong-dashboard/src/plugins/sources/defaults/Pulsar.ts:
##
@@ -123,13 +125,87 @@ export default class PulsarSource
@FieldDecorator({
type: 'input',
+tooltip: i18n.t('meta.Sources.Pulsar.SubscriptionHelp'),
props: values => ({
disabled: values?.status === 101,
}),
})
@ColumnDecorator()
- @SyncField()
@IngestionField()
+ @I18n('meta.Sources.Pulsar.Subscription')
+ subscription: string;
+
+ @FieldDecorator({
+type: 'select',
+initialValue: 'Latest',
+props: values => ({
+ disabled: values?.status === 101,
+ options: [
+{
+ label: 'Earliest',
+ value: 'Earliest',
+},
+{
+ label: 'Latest',
+ value: 'Latest',
+},
+ ],
+}),
+ })
+ @ColumnDecorator()
+ @IngestionField()
+ @I18n('meta.Sources.Pulsar.SubscriptionPosition')
+ scanStartupMode: string;
+
+ @FieldDecorator({
+type: 'select',
+initialValue: 'Shared',
+tooltip: i18n.t('meta.Sources.Pulsar.SubscriptionTypeHelp'),
Review Comment:
the subscription type should be configured by the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-9820][Dashboard][Manager] Update Pulsar source field for Ingestion [inlong]
aloyszhang commented on code in PR #9821:
URL: https://github.com/apache/inlong/pull/9821#discussion_r1525662234
##
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/source/pulsar/PulsarSourceRequest.java:
##
@@ -76,6 +76,12 @@ public class PulsarSourceRequest extends SourceRequest {
+ " Available options are earliest, latest, external-subscription,
and specific-offsets.")
private String scanStartupMode = "earliest";
+@ApiModelProperty("Configure the Pulsar subscription mode. Available
options are Exclusive, Shared, and Failover.")
Review Comment:
What about the Key_Shared subscription type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-9820][Dashboard][Manager] Update Pulsar source field for Ingestion [inlong]
aloyszhang commented on code in PR #9821:
URL: https://github.com/apache/inlong/pull/9821#discussion_r1525663726
##
inlong-dashboard/src/plugins/sources/defaults/Pulsar.ts:
##
@@ -123,13 +125,87 @@ export default class PulsarSource
@FieldDecorator({
type: 'input',
+tooltip: i18n.t('meta.Sources.Pulsar.SubscriptionHelp'),
props: values => ({
disabled: values?.status === 101,
}),
})
@ColumnDecorator()
- @SyncField()
@IngestionField()
+ @I18n('meta.Sources.Pulsar.Subscription')
+ subscription: string;
+
+ @FieldDecorator({
+type: 'select',
+initialValue: 'Latest',
+props: values => ({
+ disabled: values?.status === 101,
+ options: [
+{
+ label: 'Earliest',
+ value: 'Earliest',
+},
+{
+ label: 'Latest',
+ value: 'Latest',
+},
+ ],
+}),
+ })
+ @ColumnDecorator()
+ @IngestionField()
+ @I18n('meta.Sources.Pulsar.SubscriptionPosition')
+ scanStartupMode: string;
+
+ @FieldDecorator({
+type: 'select',
+initialValue: 'Shared',
+tooltip: i18n.t('meta.Sources.Pulsar.SubscriptionTypeHelp'),
+props: values => ({
+ disabled: values?.status === 101,
+ options: [
Review Comment:
Why not support the Key_Shared type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-9820][Dashboard][Manager] Update Pulsar source field for Ingestion [inlong]
aloyszhang commented on code in PR #9821:
URL: https://github.com/apache/inlong/pull/9821#discussion_r1525663726
##
inlong-dashboard/src/plugins/sources/defaults/Pulsar.ts:
##
@@ -123,13 +125,87 @@ export default class PulsarSource
@FieldDecorator({
type: 'input',
+tooltip: i18n.t('meta.Sources.Pulsar.SubscriptionHelp'),
props: values => ({
disabled: values?.status === 101,
}),
})
@ColumnDecorator()
- @SyncField()
@IngestionField()
+ @I18n('meta.Sources.Pulsar.Subscription')
+ subscription: string;
+
+ @FieldDecorator({
+type: 'select',
+initialValue: 'Latest',
+props: values => ({
+ disabled: values?.status === 101,
+ options: [
+{
+ label: 'Earliest',
+ value: 'Earliest',
+},
+{
+ label: 'Latest',
+ value: 'Latest',
+},
+ ],
+}),
+ })
+ @ColumnDecorator()
+ @IngestionField()
+ @I18n('meta.Sources.Pulsar.SubscriptionPosition')
+ scanStartupMode: string;
+
+ @FieldDecorator({
+type: 'select',
+initialValue: 'Shared',
+tooltip: i18n.t('meta.Sources.Pulsar.SubscriptionTypeHelp'),
+props: values => ({
+ disabled: values?.status === 101,
+ options: [
Review Comment:
missed the Key_Shared type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [INLONG-9822][Manager] Support flink job runtime execution mode configuration [inlong]
aloyszhang opened a new pull request, #9823: URL: https://github.com/apache/inlong/pull/9823 - Fixes #9822 ### Motivation Support flink job runtime execution mode configuration ### Modifications Make the runtime execution mode of flink job configurable - streaming execution mode for real-time data sync - batch execution mode for offline data sync ### Documentation not need -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-9755][Sort] NPE exception when using default configuration to start inlong-sort-standalone [inlong]
haibo-duan opened a new pull request, #9824: URL: https://github.com/apache/inlong/pull/9824 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #XYZ ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve?* ### Modifications *Describe the modifications you've done.* ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
