Re: [PR] [INLONG-11298][Agent] Fix bug for pulsar source with empty data process and specified time consumption [inlong]
aloyszhang commented on code in PR #11299:
URL: https://github.com/apache/inlong/pull/11299#discussion_r1792973785
##
inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/PulsarSource.java:
##
@@ -68,8 +69,12 @@ protected void initSource(InstanceProfile profile) {
topic = profile.getInstanceId();
serviceUrl = profile.get(TASK_PULSAR_SERVICE_URL);
subscription = profile.get(TASK_PULSAR_SUBSCRIPTION,
PULSAR_SUBSCRIPTION_PREFIX + inlongStreamId);
-subscriptionPosition =
profile.get(TASK_PULSAR_SUBSCRIPTION_POSITION,
-SubscriptionInitialPosition.Latest.name());
+String position = profile.get(TASK_PULSAR_SUBSCRIPTION_POSITION,
SubscriptionInitialPosition.Latest.name());
+if (position.equals(SUBSCRIPTION_CUSTOM)) {
Review Comment:
what does `SUBSCRIPTION_CUSTOM` mean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(inlong) branch master updated: [INLONG-11298][Agent] Fix bug for pulsar source with empty data process and specified time consumption (#11299)
This is an automated email from the ASF dual-hosted git repository.
wenweihuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 3f52c5c821 [INLONG-11298][Agent] Fix bug for pulsar source with empty
data process and specified time consumption (#11299)
3f52c5c821 is described below
commit 3f52c5c821e7fc391cd4d13f12b60325c6f8d127
Author: justinwwhuang
AuthorDate: Wed Oct 9 23:38:19 2024 +0800
[INLONG-11298][Agent] Fix bug for pulsar source with empty data process and
specified time consumption (#11299)
* [INLONG-11298][Agent] Fix bug for pulsar source
* [INLONG-11298][Agent] Fix the issue of inconsistent logic between the
page and backend code
---
.../inlong/agent/constant/CommonConstants.java | 2 +-
.../agent/message/file/ProxyMessageCache.java | 19 -
.../agent/core/instance/InstanceManager.java | 2 +-
.../inlong/agent/plugin/sources/PulsarSource.java | 47 +-
.../agent/plugin/sources/file/AbstractSource.java | 8 ++--
5 files changed, 34 insertions(+), 44 deletions(-)
diff --git
a/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/constant/CommonConstants.java
b/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/constant/CommonConstants.java
index 45320406ef..53a5bd976c 100644
---
a/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/constant/CommonConstants.java
+++
b/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/constant/CommonConstants.java
@@ -59,7 +59,7 @@ public class CommonConstants {
public static final int DEFAULT_PROXY_PACKAGE_MAX_TIMEOUT_MS = 4 * 1000;
public static final String PROXY_BATCH_FLUSH_INTERVAL =
"proxy.batch.flush.interval";
-public static final int DEFAULT_PROXY_BATCH_FLUSH_INTERVAL = 100;
+public static final int DEFAULT_PROXY_BATCH_FLUSH_INTERVAL = 1;
public static final String PROXY_SENDER_MAX_TIMEOUT =
"proxy.sender.maxTimeout";
// max timeout in seconds.
diff --git
a/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/message/file/ProxyMessageCache.java
b/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/message/file/ProxyMessageCache.java
index c7b151a26c..7e2aa28034 100644
---
a/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/message/file/ProxyMessageCache.java
+++
b/inlong-agent/agent-common/src/main/java/org/apache/inlong/agent/message/file/ProxyMessageCache.java
@@ -32,14 +32,11 @@ import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicLong;
import static
org.apache.inlong.agent.constant.CommonConstants.DEFAULT_PROXY_INLONG_STREAM_ID_QUEUE_MAX_NUMBER;
import static
org.apache.inlong.agent.constant.CommonConstants.DEFAULT_PROXY_PACKAGE_MAX_SIZE;
-import static
org.apache.inlong.agent.constant.CommonConstants.DEFAULT_PROXY_PACKAGE_MAX_TIMEOUT_MS;
import static
org.apache.inlong.agent.constant.CommonConstants.PROXY_INLONG_STREAM_ID_QUEUE_MAX_NUMBER;
import static
org.apache.inlong.agent.constant.CommonConstants.PROXY_PACKAGE_MAX_SIZE;
-import static
org.apache.inlong.agent.constant.CommonConstants.PROXY_PACKAGE_MAX_TIMEOUT_MS;
import static
org.apache.inlong.agent.constant.TaskConstants.TASK_AUDIT_VERSION;
import static org.apache.inlong.agent.constant.TaskConstants.TASK_CYCLE_UNIT;
import static org.apache.inlong.common.msg.AttributeConstants.AUDIT_VERSION;
@@ -56,11 +53,8 @@ public class ProxyMessageCache {
private final int maxPackSize;
private final int maxQueueNumber;
private final String groupId;
-// ms
-private final int cacheTimeout;
// streamId -> list of proxyMessage
private final ConcurrentHashMap>
messageQueueMap;
-private final AtomicLong cacheSize = new AtomicLong(0);
private long lastPrintTime = 0;
private long dataTime;
private boolean isRealTime = false;
@@ -76,7 +70,6 @@ public class ProxyMessageCache {
this.maxPackSize = instanceProfile.getInt(PROXY_PACKAGE_MAX_SIZE,
DEFAULT_PROXY_PACKAGE_MAX_SIZE);
this.maxQueueNumber =
instanceProfile.getInt(PROXY_INLONG_STREAM_ID_QUEUE_MAX_NUMBER,
DEFAULT_PROXY_INLONG_STREAM_ID_QUEUE_MAX_NUMBER);
-this.cacheTimeout =
instanceProfile.getInt(PROXY_PACKAGE_MAX_TIMEOUT_MS,
DEFAULT_PROXY_PACKAGE_MAX_TIMEOUT_MS);
messageQueueMap = new ConcurrentHashMap<>();
dataTime = instanceProfile.getSinkDataTime();
extraMap.put(AttributeConstants.MESSAGE_SYNC_SEND, "false");
@@ -109,7 +102,6 @@ public class ProxyMessageCache {
return false;
}
messageQueue.put(message);
-cacheSize.addAndGet(message.getBody().length);
return true;
} catch
Re: [PR] [INLONG-11298][Agent] Fix bug for pulsar source with empty data process and specified time consumption [inlong]
justinwwhuang merged PR #11299: URL: https://github.com/apache/inlong/pull/11299 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11298][Agent] Fix bug for pulsar source with empty data process and specified time consumption [inlong]
justinwwhuang commented on code in PR #11299:
URL: https://github.com/apache/inlong/pull/11299#discussion_r1793001456
##
inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/PulsarSource.java:
##
@@ -68,8 +69,12 @@ protected void initSource(InstanceProfile profile) {
topic = profile.getInstanceId();
serviceUrl = profile.get(TASK_PULSAR_SERVICE_URL);
subscription = profile.get(TASK_PULSAR_SUBSCRIPTION,
PULSAR_SUBSCRIPTION_PREFIX + inlongStreamId);
-subscriptionPosition =
profile.get(TASK_PULSAR_SUBSCRIPTION_POSITION,
-SubscriptionInitialPosition.Latest.name());
+String position = profile.get(TASK_PULSAR_SUBSCRIPTION_POSITION,
SubscriptionInitialPosition.Latest.name());
+if (position.equals(SUBSCRIPTION_CUSTOM)) {
Review Comment:
> what does `SUBSCRIPTION_CUSTOM` mean?
Subscribe to data at a specified time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-11267][Doc] Add signoz doc [inlong-website]
sususama commented on PR #1045: URL: https://github.com/apache/inlong-website/pull/1045#issuecomment-2402223590 > docs should be added in `https://github.com/apache/inlong-website/tree/master/i18n/zh-CN/docusaurus-plugin-content-docs/current ` instead of `i18n/zh-CN/docusaurus-plugin-content-docs/version-1.13.0` > > The English document is also required. done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11300][SDK] Transform SQL supports "JSON_ARRAY_APPEND" function [inlong]
Zkplo commented on code in PR #11310: URL: https://github.com/apache/inlong/pull/11310#discussion_r1793516782 ## inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/JsonArrayAppendFunction.java: ## @@ -0,0 +1,121 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.inlong.sdk.transform.process.function; + +import org.apache.inlong.sdk.transform.decode.SourceData; +import org.apache.inlong.sdk.transform.process.Context; +import org.apache.inlong.sdk.transform.process.operator.OperatorTools; +import org.apache.inlong.sdk.transform.process.parser.ValueParser; + +import com.alibaba.fastjson.JSON; +import com.alibaba.fastjson.JSONArray; +import com.alibaba.fastjson.JSONPath; +import net.sf.jsqlparser.expression.Expression; +import net.sf.jsqlparser.expression.Function; + +import java.util.ArrayList; +import java.util.List; + Review Comment: Thank you, indeed, I have added an explanation to it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11267][Doc] Add signoz doc [inlong-website]
aloyszhang commented on PR #1045: URL: https://github.com/apache/inlong-website/pull/1045#issuecomment-2402103364 docs should be added in `https://github.com/apache/inlong-website/tree/master/i18n/zh-CN/docusaurus-plugin-content-docs/current ` instead of `i18n/zh-CN/docusaurus-plugin-content-docs/version-1.13.0` The English document is required also. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-11315][Audit] Resolve the conflict between the Audit SDK and other components' Protobuf versions [inlong]
doleyzi opened a new pull request, #11316: URL: https://github.com/apache/inlong/pull/11316 - Fixes #11315 ### Motivation Resolve the conflict between the Audit SDK and other components' Protobuf versions ### Modifications By shading the Protobuf that the audit SDK depends on, the conflict between the audit SDK and other component Protobuf versions is resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-1048][Doc] Add doc for integrating a logging system for Sort [inlong-website]
aloyszhang commented on code in PR #1049: URL: https://github.com/apache/inlong-website/pull/1049#discussion_r1793379171 ## i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/sort/log_report.md: ## @@ -0,0 +1,227 @@ +--- +title: 日志上报 +sidebar_position: 6 +--- + + + +## 概览 + +由于`InLong Sor`t会运行在`Apache Flink`的不同`Task Manager`节点上，每个节点独立存储产生的日志，我们需要到每个节点上查看日志，维护效率低下。为此`InLong Sort`提供了基于OpenTelemetry的日志集中管理方案，用户可以高效地集中处理`Flink`日志。 Review Comment: ```suggestion 由于`InLong Sort`会运行在`Apache Flink`的不同`Task Manager`节点上，每个节点独立存储产生的日志，我们需要到每个节点上查看日志，维护效率低下。为此`InLong Sort`提供了基于OpenTelemetry的日志集中管理方案，用户可以高效地集中处理`Flink`日志。 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-11283][SDK]Transform enhance Map-related Collection Functions [inlong]
emptyOVO opened a new pull request, #11317: URL: https://github.com/apache/inlong/pull/11317 Fixes #11283 ### Motivation add function classes referenced in issue, provide unit tests for each class ### Modifications ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [x] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-1048][Doc] Add doc for integrating a logging system for Sort [inlong-website]
aloyszhang commented on code in PR #1049: URL: https://github.com/apache/inlong-website/pull/1049#discussion_r1793391315 ## i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/sort/log_report.md: ## @@ -0,0 +1,227 @@ +--- +title: 日志上报 Review Comment: ```suggestion title: OpenTelemetry 日志上报 ``` ## docs/modules/sort/log_report.md: ## @@ -0,0 +1,229 @@ +--- +title: Log Report Review Comment: ```suggestion title: OpenTelemetry Log Report ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11199][Sort] Integrate Grafana Loki for connectors [inlong]
aloyszhang commented on PR #11212: URL: https://github.com/apache/inlong/pull/11212#issuecomment-2402134585 I found there are some duplicated modifications with #11070. Please fix this problem if both of these two pull requests are needed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11283][SDK]Transform enhance Map-related Collection Functions [inlong]
aloyszhang commented on PR #11317: URL: https://github.com/apache/inlong/pull/11317#issuecomment-2402139896 please add some descriptions for every function -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11283][SDK]Transform enhance Map-related Collection Functions [inlong]
emptyOVO commented on PR #11317: URL: https://github.com/apache/inlong/pull/11317#issuecomment-2402212916 > please add some descriptions for every function thanks, done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11267][Doc] Add signoz doc [inlong-website]
sususama commented on PR #1045: URL: https://github.com/apache/inlong-website/pull/1045#issuecomment-2402214120 > docs should be added in `https://github.com/apache/inlong-website/tree/master/i18n/zh-CN/docusaurus-plugin-content-docs/current ` instead of `i18n/zh-CN/docusaurus-plugin-content-docs/version-1.13.0` > > The English document is also required. done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11286][Audit] Optimize the statistics of daily Audit data [inlong]
aloyszhang merged PR #11312: URL: https://github.com/apache/inlong/pull/11312 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-11286][Audit] Optimize the statistics of daily Audit data (#11312)
This is an automated email from the ASF dual-hosted git repository.
aloyszhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 778cbe6050 [INLONG-11286][Audit] Optimize the statistics of daily
Audit data (#11312)
778cbe6050 is described below
commit 778cbe6050ff4c60b7fd17dbeabfe614289cac7e
Author: doleyzi <43397300+dole...@users.noreply.github.com>
AuthorDate: Wed Oct 9 19:32:17 2024 +0800
[INLONG-11286][Audit] Optimize the statistics of daily Audit data (#11312)
---
.../inlong/audit/config/ConfigConstants.java | 6 +-
.../apache/inlong/audit/config/SqlConstants.java | 12 +-
.../inlong/audit/entities/PartitionEntity.java | 71
.../apache/inlong/audit/entities/SourceConfig.java | 4 +-
.../org/apache/inlong/audit/main/Application.java | 3 +
.../apache/inlong/audit/service/EtlService.java| 153 ++---
.../inlong/audit/service/PartitionManager.java | 186 +
.../org/apache/inlong/audit/sink/AuditSink.java| 24 +++
.../org/apache/inlong/audit/sink/CacheSink.java| 2 +-
.../org/apache/inlong/audit/sink/JdbcSink.java | 152 ++---
.../org/apache/inlong/audit/source/JdbcSource.java | 37 +---
.../org/apache/inlong/audit/utils/JdbcUtils.java | 37
inlong-audit/sql/apache_inlong_audit_mysql.sql | 4 +-
13 files changed, 413 insertions(+), 278 deletions(-)
diff --git
a/inlong-audit/audit-service/src/main/java/org/apache/inlong/audit/config/ConfigConstants.java
b/inlong-audit/audit-service/src/main/java/org/apache/inlong/audit/config/ConfigConstants.java
index d5afb1306e..d74e162f78 100644
---
a/inlong-audit/audit-service/src/main/java/org/apache/inlong/audit/config/ConfigConstants.java
+++
b/inlong-audit/audit-service/src/main/java/org/apache/inlong/audit/config/ConfigConstants.java
@@ -57,9 +57,6 @@ public class ConfigConstants {
public static final int DEFAULT_SOURCE_DB_SINK_BATCH = 1000;
public static final String KEY_CONFIG_UPDATE_INTERVAL_SECONDS =
"config.update.interval.seconds";
public static final int DEFAULT_CONFIG_UPDATE_INTERVAL_SECONDS = 60;
-
-public static final String KEY_ENABLE_MANAGE_PARTITIONS =
"enable.manage.partitions";
-public static final boolean DEFAULT_ENABLE_MANAGE_PARTITIONS = true;
public static final String KEY_CHECK_PARTITION_INTERVAL_HOURS =
"check.partition.interval.hours";
public static final int DEFAULT_CHECK_PARTITION_INTERVAL_HOURS = 6;
@@ -113,4 +110,7 @@ public class ConfigConstants {
public static final int MAX_INIT_COUNT = 2;
public static final int RANDOM_BOUND = 10;
+public static final String KEY_ENABLE_STAT_AUDIT_DAY =
"enable.stat.audit.day";
+public static final boolean DEFAULT_ENABLE_STAT_AUDIT_DAY = true;
+
}
diff --git
a/inlong-audit/audit-service/src/main/java/org/apache/inlong/audit/config/SqlConstants.java
b/inlong-audit/audit-service/src/main/java/org/apache/inlong/audit/config/SqlConstants.java
index b48368b921..04fee349b5 100644
---
a/inlong-audit/audit-service/src/main/java/org/apache/inlong/audit/config/SqlConstants.java
+++
b/inlong-audit/audit-service/src/main/java/org/apache/inlong/audit/config/SqlConstants.java
@@ -170,8 +170,12 @@ public class SqlConstants {
public static final String KEY_AUDIT_DATA_TEMP_DELETE_PARTITION_SQL =
"audit.data.temp.delete.partition.sql";
public static final String DEFAULT_AUDIT_DATA_TEMP_DELETE_PARTITION_SQL =
"ALTER TABLE audit_data_temp DROP PARTITION %s";
-
-public static final String KEY_AUDIT_DATA_TEMP_CHECK_PARTITION_SQL =
"audit.data.temp.check.partition.sql";
-public static final String DEFAULT_AUDIT_DATA_TEMP_CHECK_PARTITION_SQL =
-"SELECT COUNT(*) AS count FROM INFORMATION_SCHEMA.PARTITIONS WHERE
TABLE_NAME = 'audit_data_temp' and PARTITION_NAME = ?";
+public static final String KEY_TABLE_AUDIT_DATA_CHECK_PARTITION_SQL =
"audit.data.check.partition.sql";
+public static final String DEFAULT_TABLE_AUDIT_DATA_CHECK_PARTITION_SQL =
+"SELECT COUNT(*) AS count FROM INFORMATION_SCHEMA.PARTITIONS WHERE
TABLE_NAME = '%s' and PARTITION_NAME = '%s'";
+public static final String KEY_TABLE_AUDIT_DATA_DAY_ADD_PARTITION_SQL =
"audit.data.day.add.partition.sql";
+public static final String DEFAULT_TABLE_AUDIT_DATA_DAY_ADD_PARTITION_SQL =
+"ALTER TABLE audit_data_day ADD PARTITION (PARTITION %s VALUES
LESS THAN (TO_DAYS('%s')))";
+public static final String TABLE_AUDIT_DATA_DAY = "audit_data_day";
+public static final String TABLE_AUDIT_DATA_TEMP = "audit_data_temp";
}
diff --git
a/inlong-audit/audit-service/src/main/java/org/apache/inlong/audit/entities/PartitionEntity.java
b/inlong-audit/audit-service/src/main/java/org/apache/inlong/audit/entities/PartitionEntity.java
new file mode 100644
index 00..4704ea1031
-
Re: [PR] [INLONG-11266][Sort] Add end-to-end test case for sort-connector-pulsar-v1.15 [inlong]
aloyszhang merged PR #11268: URL: https://github.com/apache/inlong/pull/11268 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-11266][Sort] Add end-to-end test case for sort-connector-pulsar-v1.15 (#11268)
This is an automated email from the ASF dual-hosted git repository.
aloyszhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new e97615d56d [INLONG-11266][Sort] Add end-to-end test case for
sort-connector-pulsar-v1.15 (#11268)
e97615d56d is described below
commit e97615d56da6e3e2c2de75e02d72e47324cc6838
Author: PeterZh6
AuthorDate: Wed Oct 9 19:30:35 2024 +0800
[INLONG-11266][Sort] Add end-to-end test case for
sort-connector-pulsar-v1.15 (#11268)
---
.../sort-end-to-end-tests-v1.15/pom.xml| 20 +++
.../inlong/sort/tests/Pulsar2StarRocksTest.java| 149 +
.../src/test/resources/flinkSql/pulsar_test.sql| 39 ++
pom.xml| 6 +
4 files changed, 214 insertions(+)
diff --git
a/inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.15/pom.xml
b/inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.15/pom.xml
index cfaaee3f65..57d8d16d7a 100644
--- a/inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.15/pom.xml
+++ b/inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.15/pom.xml
@@ -51,6 +51,11 @@
org.testcontainers
kafka
+
+org.testcontainers
+pulsar
+test
+
org.testcontainers
mongodb
@@ -157,6 +162,13 @@
jedis
test
+
+
+org.apache.pulsar
+pulsar-client
+${pulsar.version}
+
+
@@ -246,6 +258,14 @@
jar
${project.build.directory}/dependencies
+
+org.apache.inlong
+
sort-connector-pulsar-v1.15
+${project.version}
+
sort-connector-pulsar.jar
+jar
+
${project.build.directory}/dependencies
+
diff --git
a/inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.15/src/test/java/org/apache/inlong/sort/tests/Pulsar2StarRocksTest.java
b/inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.15/src/test/java/org/apache/inlong/sort/tests/Pulsar2StarRocksTest.java
new file mode 100644
index 00..e5252d0b4a
--- /dev/null
+++
b/inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.15/src/test/java/org/apache/inlong/sort/tests/Pulsar2StarRocksTest.java
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.tests;
+
+import org.apache.inlong.sort.tests.utils.FlinkContainerTestEnvJRE8;
+import org.apache.inlong.sort.tests.utils.JdbcProxy;
+import org.apache.inlong.sort.tests.utils.StarRocksContainer;
+import org.apache.inlong.sort.tests.utils.TestUtils;
+
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.PulsarClient;
+import org.apache.pulsar.client.api.Schema;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.Container;
+import org.testcontainers.containers.PulsarContainer;
+import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.utility.DockerImageName;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+
+import static
org.apache.inlong.sort.tests.utils.StarRocksManager.INTER_CONTAINER_STAR_ROCKS_ALIAS;
+import static
org.apache.inlong.sort.tests.utils.StarRocksManager.STAR_ROCKS_LOG;
+import static
org.apache.inlong.sort.tests.utils.StarRocksManager.getNewStarRocksImageName;
+import static
org.apache.inlong.sort.tests.utils.StarRocksManager.initializeStarRocksTable;
+
+public class Pulsar2StarRocksT
Re: [PR] [INLONG-11300][SDK] Transform SQL supports "JSON_ARRAY_APPEND" function [inlong]
aloyszhang commented on code in PR #11310: URL: https://github.com/apache/inlong/pull/11310#discussion_r1793361631 ## inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/JsonArrayAppendFunction.java: ## @@ -0,0 +1,121 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.inlong.sdk.transform.process.function; + +import org.apache.inlong.sdk.transform.decode.SourceData; +import org.apache.inlong.sdk.transform.process.Context; +import org.apache.inlong.sdk.transform.process.operator.OperatorTools; +import org.apache.inlong.sdk.transform.process.parser.ValueParser; + +import com.alibaba.fastjson.JSON; +import com.alibaba.fastjson.JSONArray; +import com.alibaba.fastjson.JSONPath; +import net.sf.jsqlparser.expression.Expression; +import net.sf.jsqlparser.expression.Function; + +import java.util.ArrayList; +import java.util.List; + Review Comment: please add some descriptions for this function -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11067][Sort] Report mysql-connector logs through openTelemetry [inlong]
qy-liuhuo closed pull request #11070: [INLONG-11067][Sort] Report mysql-connector logs through openTelemetry URL: https://github.com/apache/inlong/pull/11070 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-11067][Sort] Report mysql-connector logs through openTelemetry [inlong]
qy-liuhuo opened a new pull request, #11070: URL: https://github.com/apache/inlong/pull/11070 Fixes #11067 ### Motivation Use openTelemetryLogger(https://github.com/apache/inlong/pull/11066) to report mysql-connector logs to the docker container of opentelemetry-collector ### Modifications - Change `docker-compose.yml` to add a opentelemetry-collector container - Add the `docker/docker-compose/otel-config.yaml` file for configure the opentelemetry-collector container - Change the `inlong-sort/sort-flink/sort-flink-v1.15/sort-connectors/mysql-cdc/src/main/java/org/apache/inlong/sort/mysql/source/reader/MySqlSourceReader.java` file to add log reporting function to this connector ### Verifying this change *(Please pick either of the following options)* - [x] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11067][Sort] Report mysql-connector logs through openTelemetry [inlong]
qy-liuhuo commented on PR #11070: URL: https://github.com/apache/inlong/pull/11070#issuecomment-2402044602 > Please resolve the conflict I have resolved it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11178][Dashboard] Added http sink and http node [inlong]
fuweng11 commented on PR #11311: URL: https://github.com/apache/inlong/pull/11311#issuecomment-2402093880 English page displays Chinese content, please fix it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11067][Sort] Report mysql-connector logs through openTelemetry [inlong]
qy-liuhuo closed pull request #11070: [INLONG-11067][Sort] Report mysql-connector logs through openTelemetry URL: https://github.com/apache/inlong/pull/11070 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11199][Sort] Integrate Grafana Loki for connectors [inlong]
qy-liuhuo commented on PR #11212: URL: https://github.com/apache/inlong/pull/11212#issuecomment-2402158506 > I found there are some duplicated modifications with #11070. Please fix this problem if both of these two pull requests are needed Yeah，#11070 is just an example of usage， so i think we can temporarily close it（#11070）. After this issue(#11199 ) is completed, We can raise a new issue to integrate for all connectors -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11199][Sort] Integrate Grafana Loki for connectors [inlong]
qy-liuhuo closed pull request #11212: [INLONG-11199][Sort] Integrate Grafana Loki for connectors URL: https://github.com/apache/inlong/pull/11212 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11199][Sort] Integrate Grafana Loki for connectors [inlong]
qy-liuhuo commented on PR #11212: URL: https://github.com/apache/inlong/pull/11212#issuecomment-2402166836 Due to new changes in #11212, this PR is closed and a new implementation will be provided later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11067][Sort] Report mysql-connector logs through openTelemetry [inlong]
qy-liuhuo commented on PR #11070: URL: https://github.com/apache/inlong/pull/11070#issuecomment-2402168326 Due to new changes in #11212, this PR is closed and a new implementation will be provided later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-10463][SDK] Optimization of ultra-long field processing in InlongSDK (Temp) [inlong]
gosonzhang commented on code in PR #11213:
URL: https://github.com/apache/inlong/pull/11213#discussion_r1794432842
##
inlong-sdk/dataproxy-sdk/src/main/java/org/apache/inlong/sdk/dataproxy/ConfigConstants.java:
##
@@ -73,5 +73,6 @@ public class ConfigConstants {
public static String HTTP = "http://";;
public static String HTTPS = "https://";;
+public static int MAX_MESSAGE_LENGTH = 500 * 1024;
Review Comment:
Manager already supports configuring the maximum packet length
maxPacketLength setting allowed by DataProxy cluster for each {groupId,
streamId} [1].
SDK needs to check the length of the reported message based on the
maxPacketLength provided by Manager.
1. https://github.com/apache/inlong/pull/11314
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(inlong) branch master updated: [INLONG-11313][Manager] Dataproxy cluster increases maximum packet length configuration (#11314)
This is an automated email from the ASF dual-hosted git repository.
gosonzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 47efde7781 [INLONG-11313][Manager] Dataproxy cluster increases maximum
packet length configuration (#11314)
47efde7781 is described below
commit 47efde7781df011499bb6b7de21d67ea793bcc62
Author: fuweng11 <76141879+fuwen...@users.noreply.github.com>
AuthorDate: Thu Oct 10 08:57:45 2024 +0800
[INLONG-11313][Manager] Dataproxy cluster increases maximum packet length
configuration (#11314)
---
.../inlong/common/pojo/dataproxy/DataProxyNodeInfo.java | 6 ++
.../pojo/cluster/dataproxy/DataProxyClusterDTO.java | 4
.../pojo/cluster/dataproxy/DataProxyClusterInfo.java| 3 +++
.../pojo/cluster/dataproxy/DataProxyClusterRequest.java | 3 +++
.../service/cluster/InlongClusterServiceImpl.java | 17 -
5 files changed, 32 insertions(+), 1 deletion(-)
diff --git
a/inlong-common/src/main/java/org/apache/inlong/common/pojo/dataproxy/DataProxyNodeInfo.java
b/inlong-common/src/main/java/org/apache/inlong/common/pojo/dataproxy/DataProxyNodeInfo.java
index 1311a1b573..87d081d79a 100644
---
a/inlong-common/src/main/java/org/apache/inlong/common/pojo/dataproxy/DataProxyNodeInfo.java
+++
b/inlong-common/src/main/java/org/apache/inlong/common/pojo/dataproxy/DataProxyNodeInfo.java
@@ -49,4 +49,10 @@ public class DataProxyNodeInfo {
* Node last report load
*/
private Integer nodeLoad;
+
+/**
+ * Node max packet length
+ */
+private Integer maxPacketLength;
+
}
diff --git
a/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/cluster/dataproxy/DataProxyClusterDTO.java
b/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/cluster/dataproxy/DataProxyClusterDTO.java
index 7143c87a09..e510af919f 100644
---
a/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/cluster/dataproxy/DataProxyClusterDTO.java
+++
b/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/cluster/dataproxy/DataProxyClusterDTO.java
@@ -49,6 +49,9 @@ public class DataProxyClusterDTO {
@ApiModelProperty("Load of the DataProxy cluster, default is 20")
private Integer load = 20;
+@ApiModelProperty("Max packet length of the DataProxy cluster")
+private Integer maxPacketLength;
+
/**
* Get the dto instance from the request
*/
@@ -57,6 +60,7 @@ public class DataProxyClusterDTO {
.isIntranet(request.getIsIntranet())
.isSwitch(request.getIsSwitch())
.load(request.getLoad())
+.maxPacketLength(request.getMaxPacketLength())
.build();
}
diff --git
a/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/cluster/dataproxy/DataProxyClusterInfo.java
b/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/cluster/dataproxy/DataProxyClusterInfo.java
index 5dd7ce0e4d..a38d139c87 100644
---
a/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/cluster/dataproxy/DataProxyClusterInfo.java
+++
b/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/cluster/dataproxy/DataProxyClusterInfo.java
@@ -47,6 +47,9 @@ public class DataProxyClusterInfo extends ClusterInfo {
@ApiModelProperty("Load of the DataProxy cluster, default is 20")
private Integer load = 20;
+@ApiModelProperty("Max packet length of the DataProxy cluster")
+private Integer maxPacketLength;
+
public DataProxyClusterInfo() {
this.setType(ClusterType.DATAPROXY);
}
diff --git
a/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/cluster/dataproxy/DataProxyClusterRequest.java
b/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/cluster/dataproxy/DataProxyClusterRequest.java
index fb84b01aeb..84bba41957 100644
---
a/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/cluster/dataproxy/DataProxyClusterRequest.java
+++
b/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/cluster/dataproxy/DataProxyClusterRequest.java
@@ -46,6 +46,9 @@ public class DataProxyClusterRequest extends ClusterRequest {
@ApiModelProperty("Load of the DataProxy cluster, default is 20")
private Integer load = 20;
+@ApiModelProperty("Max packet length of the DataProxy cluster")
+private Integer maxPacketLength;
+
public DataProxyClusterRequest() {
this.setType(ClusterType.DATAPROXY);
}
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/cluster/InlongClusterServiceImpl.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/cluster/InlongClusterServiceImpl.java
index 7882e29c12..99505b1bfe 10064
Re: [PR] [INLONG-11313][Manager] Dataproxy cluster add maximum packet length configuration [inlong]
gosonzhang merged PR #11314: URL: https://github.com/apache/inlong/pull/11314 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11178][Dashboard] Added http sink and http node [inlong]
wohainilaodou commented on PR #11311: URL: https://github.com/apache/inlong/pull/11311#issuecomment-2403735703 > English page displays Chinese content, please fix it. done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11178][Dashboard] Added http sink and http node [inlong]
wohainilaodou commented on PR #11311: URL: https://github.com/apache/inlong/pull/11311#issuecomment-2403722614 > 英文页面显示中文内容，请修复。 done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-10883][SDK] Transform SQL support InitCap function [inlong]
MOONSakura0614 commented on PR #10892: URL: https://github.com/apache/inlong/pull/10892#issuecomment-2403914270 > You should rebase the master branch and then force push you branch solved -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11267][Doc] Add signoz doc [inlong-website]
aloyszhang commented on code in PR #1045: URL: https://github.com/apache/inlong-website/pull/1045#discussion_r1794722347 ## docs/modules/sort/log_reporting.md: ## @@ -0,0 +1,80 @@ +--- +title: Log reporting +sidebar_position: 6 +--- + +## 概览 Review Comment: please translate to english -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11267][Doc] Add signoz doc [inlong-website]
aloyszhang commented on code in PR #1045: URL: https://github.com/apache/inlong-website/pull/1045#discussion_r1794722887 ## i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/sort/log_reporting.md: ## @@ -0,0 +1,81 @@ +--- +title: 日志上报 Review Comment: ```suggestion title: OpenTelemetry 日志上报 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11267][Doc] Add signoz doc [inlong-website]
aloyszhang commented on code in PR #1045: URL: https://github.com/apache/inlong-website/pull/1045#discussion_r1794723183 ## docs/modules/sort/log_reporting.md: ## @@ -0,0 +1,80 @@ +--- +title: Log reporting Review Comment: ```suggestion title: OpenTelemetry log reporting ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-11029][SDK] Transform SQL support StartsWith & EndsWith functions [inlong]
MOONSakura0614 opened a new pull request, #11318: URL: https://github.com/apache/inlong/pull/11318 Fixes #11029 ### Motivation Add two function classes: StartsWith & EndsWith functions and their unit tests. - STARTSWITH(expr, startExpr): Returns whether expr starts with startExpr. If startExpr is empty, the result is true. - ENDSWITH(expr, endExpr): Returns whether expr ends with endExpr. If endExpr is empty, the result is true. ### Modifications ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [x] This change is already covered by existing tests, such as: *(please describe tests)* - [x] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-11301][SDK] Transform SQL supports "JSON_ARRAY_INSERT" function [inlong]
Zkplo opened a new pull request, #11319: URL: https://github.com/apache/inlong/pull/11319 Fixes #11301 ### Motivation Add the JsonArrayInsertFunction class and enhance the corresponding test class: TestJsonArrayInsertFunction. ### Modifications ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [x] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-10883][SDK] Transform SQL support InitCap function [inlong]
aloyszhang commented on PR #10892: URL: https://github.com/apache/inlong/pull/10892#issuecomment-2403748355 You should rebase the master branch and then force push you branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11315][Audit] Resolve the conflict between the Audit SDK and other components' Protobuf versions [inlong]
doleyzi merged PR #11316: URL: https://github.com/apache/inlong/pull/11316 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-11315][Audit] Resolve the conflict between the Audit SDK and other components' Protobuf versions (#11316)
This is an automated email from the ASF dual-hosted git repository.
doleyzi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new c7f358f4bf [INLONG-11315][Audit] Resolve the conflict between the
Audit SDK and other components' Protobuf versions (#11316)
c7f358f4bf is described below
commit c7f358f4bffcf641c36cbbe235db0ade3e14f56c
Author: doleyzi <43397300+dole...@users.noreply.github.com>
AuthorDate: Thu Oct 10 09:56:07 2024 +0800
[INLONG-11315][Audit] Resolve the conflict between the Audit SDK and other
components' Protobuf versions (#11316)
---
inlong-audit/audit-common/pom.xml | 30 ++
1 file changed, 30 insertions(+)
diff --git a/inlong-audit/audit-common/pom.xml
b/inlong-audit/audit-common/pom.xml
index 3ecbd4b3d7..aaf5048b29 100644
--- a/inlong-audit/audit-common/pom.xml
+++ b/inlong-audit/audit-common/pom.xml
@@ -67,6 +67,36 @@
+
+org.apache.maven.plugins
+maven-shade-plugin
+${plugin.shade.version}
+
+
+shade-audit
+
+shade
+
+package
+
+
+
false
+
+
+org.apache.inlong:*
+com.google.protobuf:*
+
+
+
+
+com.google.protobuf
+
org.apache.inlong.audit.shaded.com.google.protobuf
+
+
+
+
+
+
(inlong) branch master updated: [INLONG-11178][Dashboard] Added http sink and http node (#11311)
This is an automated email from the ASF dual-hosted git repository.
aloyszhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 80125f0f63 [INLONG-11178][Dashboard] Added http sink and http node
(#11311)
80125f0f63 is described below
commit 80125f0f6336d9990ea3dbc0b77cb7330b19648f
Author: kamianlaida <165994047+wohainilao...@users.noreply.github.com>
AuthorDate: Thu Oct 10 09:58:54 2024 +0800
[INLONG-11178][Dashboard] Added http sink and http node (#11311)
---
.../src/plugins/clusters/defaults/SortHttp.ts | 27 +++
.../src/plugins/clusters/defaults/index.ts | 5 +
.../src/plugins/nodes/defaults/Http.ts | 79 +
.../src/plugins/nodes/defaults/index.ts| 5 +
.../src/plugins/sinks/defaults/Http.ts | 186 +
.../src/plugins/sinks/defaults/index.ts| 5 +
inlong-dashboard/src/ui/locales/cn.json| 11 ++
inlong-dashboard/src/ui/locales/en.json| 11 ++
.../src/ui/pages/Clusters/CreateModal.tsx | 3 +-
9 files changed, 331 insertions(+), 1 deletion(-)
diff --git a/inlong-dashboard/src/plugins/clusters/defaults/SortHttp.ts
b/inlong-dashboard/src/plugins/clusters/defaults/SortHttp.ts
new file mode 100644
index 00..473d3dcaff
--- /dev/null
+++ b/inlong-dashboard/src/plugins/clusters/defaults/SortHttp.ts
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { ClusterInfo } from '@/plugins/clusters/common/ClusterInfo';
+import { DataWithBackend } from '@/plugins/DataWithBackend';
+import { RenderRow } from '@/plugins/RenderRow';
+import { RenderList } from '@/plugins/RenderList';
+
+export default class SortHttp
+ extends ClusterInfo
+ implements DataWithBackend, RenderRow, RenderList {}
diff --git a/inlong-dashboard/src/plugins/clusters/defaults/index.ts
b/inlong-dashboard/src/plugins/clusters/defaults/index.ts
index 1c5e49bb49..0c947572b5 100644
--- a/inlong-dashboard/src/plugins/clusters/defaults/index.ts
+++ b/inlong-dashboard/src/plugins/clusters/defaults/index.ts
@@ -71,4 +71,9 @@ export const allDefaultClusters:
MetaExportWithBackendList = [
value: 'SORT_KAFKA',
LoadEntity: () => import('./SortKafka'),
},
+ {
+label: 'Sort Http',
+value: 'SORT_HTTP',
+LoadEntity: () => import('./SortHttp'),
+ },
];
diff --git a/inlong-dashboard/src/plugins/nodes/defaults/Http.ts
b/inlong-dashboard/src/plugins/nodes/defaults/Http.ts
new file mode 100644
index 00..8017394143
--- /dev/null
+++ b/inlong-dashboard/src/plugins/nodes/defaults/Http.ts
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { DataWithBackend } from '@/plugins/DataWithBackend';
+import { RenderRow } from '@/plugins/RenderRow';
+import { RenderList } from '@/plugins/RenderList';
+import { NodeInfo } from '../common/NodeInfo';
+import i18n from 'i18next';
+import { Input } from 'antd';
+
+const { I18n } = DataWithBackend;
+const { FieldDecorator } = RenderRow;
+
+export default class HttpNodeInfo
+ extends NodeInfo
+ implements DataWithBackend, RenderRow, RenderList
+{
+ @FieldDecorator({
+type: 'input',
+rules: [{ required: true }],
+ })
+ @I18n('meta.Nodes.Http.BaseUrl')
+ baseUrl: string;
+
+ @FieldDecorator({
+type: 'radio',
+rules: [{ required: true }],
+
Re: [PR] [INLONG-11178][Dashboard] Added http sink and http node [inlong]
aloyszhang merged PR #11311: URL: https://github.com/apache/inlong/pull/11311 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11300][SDK] Transform SQL supports "JSON_ARRAY_APPEND" function [inlong]
luchunliang merged PR #11310: URL: https://github.com/apache/inlong/pull/11310 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-11300][SDK] Transform SQL supports "JSON_ARRAY_APPEND" function (#11310)
This is an automated email from the ASF dual-hosted git repository.
luchunliang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 58fd7ddd85 [INLONG-11300][SDK] Transform SQL supports
"JSON_ARRAY_APPEND" function (#11310)
58fd7ddd85 is described below
commit 58fd7ddd85467658120f995a82afc68f5583fac6
Author: Zkplo <87751516+zk...@users.noreply.github.com>
AuthorDate: Thu Oct 10 11:34:00 2024 +0800
[INLONG-11300][SDK] Transform SQL supports "JSON_ARRAY_APPEND" function
(#11310)
Co-authored-by: ZKpLo <14148880+zk...@user.noreply.gitee.com>
---
.../process/function/JsonArrayAppendFunction.java | 127 +
.../string/TestJsonArrayAppendFunction.java| 93 +++
2 files changed, 220 insertions(+)
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/JsonArrayAppendFunction.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/JsonArrayAppendFunction.java
new file mode 100644
index 00..58c144c283
--- /dev/null
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/JsonArrayAppendFunction.java
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson.JSONArray;
+import com.alibaba.fastjson.JSONPath;
+import net.sf.jsqlparser.expression.Expression;
+import net.sf.jsqlparser.expression.Function;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * JsonArrayAppendFunction -> JSON_ARRAY_APPEND(json_doc, path, val[,
path, val] ...)
+ * description:
+ * - return NULL if any argument is NULL.
+ * - return the result of appends values to the end of the indicated arrays
within a JSON document.
+ */
+@TransformFunction(names = {"json_array_append"})
+public class JsonArrayAppendFunction implements ValueParser {
+
+private ValueParser jsonDocParser;
+private List pathValuePairsParser;
+
+public JsonArrayAppendFunction(Function expr) {
+List expressions = expr.getParameters().getExpressions();
+jsonDocParser = OperatorTools.buildParser(expressions.get(0));
+pathValuePairsParser = new ArrayList<>();
+for (int i = 1; i < expressions.size(); i++) {
+
pathValuePairsParser.add(OperatorTools.buildParser(expressions.get(i)));
+}
+}
+
+@Override
+public Object parse(SourceData sourceData, int rowIndex, Context context) {
+Object jsonDocObj = jsonDocParser.parse(sourceData, rowIndex, context);
+if (jsonDocObj == null) {
+return null;
+}
+ArrayList pathValuePairs = new ArrayList<>();
+for (ValueParser valueParser : pathValuePairsParser) {
+pathValuePairs.add(valueParser.parse(sourceData, rowIndex,
context));
+}
+return jsonArrayAppend(jsonDocObj.toString(), pathValuePairs);
+}
+
+public static String jsonArrayAppend(String jsonDoc, ArrayList
pathValuePairs) {
+if (jsonDoc == null || pathValuePairs == null || pathValuePairs.size()
% 2 != 0) {
+return null;
+}
+
+Object jsonObject;
+try {
+jsonObject = JSON.parse(jsonDoc);
+} catch (Exception e) {
+throw new IllegalArgumentException("Invalid JSON document", e);
+}
+
+// Process each pair of path and val
+for (int i = 0; i < pathValuePairs.size(); i += 2) {
+String path = (String) pathValuePairs.get(i);
+Object value = pathValuePairs.get(i + 1);
+
+// Attempt to append a value to the array pointed to by the
specified path
+try {
+jsonObject = appendValueToArray(jsonObject, path, value);
+
Re: [PR] [INLONG-11283][SDK]Transform enhance Map-related Collection Functions [inlong]
luchunliang merged PR #11317: URL: https://github.com/apache/inlong/pull/11317 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-11283][SDK]Transform enhance Map-related Collection Functions (#11317)
This is an automated email from the ASF dual-hosted git repository.
luchunliang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 25e786b828 [INLONG-11283][SDK]Transform enhance Map-related Collection
Functions (#11317)
25e786b828 is described below
commit 25e786b82886a682d36cd3d9cd045c1e2d3a9bfe
Author: emptyOVO <118812562+empty...@users.noreply.github.com>
AuthorDate: Thu Oct 10 11:33:44 2024 +0800
[INLONG-11283][SDK]Transform enhance Map-related Collection Functions
(#11317)
* [INLONG-11283][SDK]Transform enhance Map-related Collection Functions
* feat: add description for functions
---
.../process/function/MapEntriesFunction.java | 56 +
.../process/function/MapFromArraysFunction.java| 77 +
.../process/function/MapKeysFunction.java | 59 +
.../process/function/MapUnionFunction.java | 77 +
.../process/function/MapValueFunction.java | 60 ++
.../function/string/TestMapEntriesFunction.java| 90
.../function/string/TestMapFromArraysFunction.java | 90
.../function/string/TestMapKeysFunction.java | 89
.../function/string/TestMapUnionFunction.java | 96 ++
.../function/string/TestMapValuesFunction.java | 90
10 files changed, 784 insertions(+)
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/MapEntriesFunction.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/MapEntriesFunction.java
new file mode 100644
index 00..dde1c82b78
--- /dev/null
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/MapEntriesFunction.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import net.sf.jsqlparser.expression.Function;
+
+import java.util.Arrays;
+import java.util.Map;
+/**
+ * MapEntriesFunction
+ * description: MAP_ENTRIES(map)--Returns an array of all entries in the given
map. No order guaranteed.
+ * for example: map_entries(Map('he',1,'xxd','cloud'))--return [he=1,
xxd=cloud]
+ * map_entries(Map(1,2,'cloud','xxd'))--return [xxd=cloud, 1=2]
+ */
+@TransformFunction(names = {"map_entries"})
+public class MapEntriesFunction implements ValueParser {
+
+private final ValueParser mapParser;
+
+public MapEntriesFunction(Function expr) {
+this.mapParser =
OperatorTools.buildParser(expr.getParameters().getExpressions().get(0));
+}
+
+@Override
+public Object parse(SourceData sourceData, int rowIndex, Context context) {
+Object mapObj = mapParser.parse(sourceData, rowIndex, context);
+if (mapObj == null) {
+return null;
+}
+if (mapObj instanceof Map) {
+Map map = (Map) mapObj;
+return Arrays.toString(map.entrySet().toArray(new Map.Entry[0]));
+}
+return null;
+}
+}
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/MapFromArraysFunction.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/MapFromArraysFunction.java
new file mode 100644
index 00..2e90e7d6e1
--- /dev/null
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/MapFromArraysFunction.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version
[PR] [INLONG-11321][CI] Fix workflow failure when removing unnecessary packages [inlong]
aloyszhang opened a new pull request, #11322: URL: https://github.com/apache/inlong/pull/11322 Fixes #11321 ### Motivation Fix workflow failure when removing unnecessary packages ### Modifications improve package remove logic ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. ### Documentation - Does this pull request introduce a new feature? (no) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug]When the tenant associated with the user is not public, login is not possible. [inlong]
aloyszhang commented on issue #10586: URL: https://github.com/apache/inlong/issues/10586#issuecomment-2403843872 Close this issue since there has been no feedback for a long time, and feel free to re-open it if needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11237][SDK] Transform SQL supports CHAR_LENGTH function [inlong]
luchunliang merged PR #11275: URL: https://github.com/apache/inlong/pull/11275 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11252][SDK] Transform support PB sink data [inlong]
luchunliang merged PR #11280: URL: https://github.com/apache/inlong/pull/11280 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-11237][SDK] Transform SQL supports CHAR_LENGTH function (#11275)
This is an automated email from the ASF dual-hosted git repository.
luchunliang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 4ba289dabd [INLONG-11237][SDK] Transform SQL supports CHAR_LENGTH
function (#11275)
4ba289dabd is described below
commit 4ba289dabdb55c203b634fcc00a8d0be0ff92792
Author: Zkplo <87751516+zk...@users.noreply.github.com>
AuthorDate: Thu Oct 10 11:04:11 2024 +0800
[INLONG-11237][SDK] Transform SQL supports CHAR_LENGTH function (#11275)
Co-authored-by: ZKpLo <14148880+zk...@user.noreply.gitee.com>
---
...LengthFunction.java => CharLengthFunction.java} | 19 ++
.../transform/process/function/LengthFunction.java | 24 ++--
...thFunction.java => TestCharLengthFunction.java} | 43 +
.../function/string/TestCompressFunction.java | 20 +-
.../function/string/TestLengthFunction.java| 44 +-
5 files changed, 104 insertions(+), 46 deletions(-)
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/LengthFunction.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/CharLengthFunction.java
similarity index 73%
copy from
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/LengthFunction.java
copy to
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/CharLengthFunction.java
index 0c1abba8f0..a78d16db8f 100644
---
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/LengthFunction.java
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/CharLengthFunction.java
@@ -22,21 +22,25 @@ import org.apache.inlong.sdk.transform.process.Context;
import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+import net.sf.jsqlparser.expression.Expression;
import net.sf.jsqlparser.expression.Function;
+import java.util.List;
+
/**
* LengthFunction
- * description: length(string)
- * - return the length of the string
+ * description: char_length(string)
+ * - return the character length of the string
* - return NULL if the string is NULL
*/
-@TransformFunction(names = {"length"})
-public class LengthFunction implements ValueParser {
+@TransformFunction(names = {"char_length"})
+public class CharLengthFunction implements ValueParser {
private final ValueParser stringParser;
-public LengthFunction(Function expr) {
-stringParser =
OperatorTools.buildParser(expr.getParameters().getExpressions().get(0));
+public CharLengthFunction(Function expr) {
+List expressions = expr.getParameters().getExpressions();
+stringParser = OperatorTools.buildParser(expressions.get(0));
}
@Override
@@ -45,6 +49,7 @@ public class LengthFunction implements ValueParser {
if (stringObject == null) {
return null;
}
-return OperatorTools.parseString(stringObject).length();
+String str = OperatorTools.parseString(stringObject);
+return str.length();
}
}
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/LengthFunction.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/LengthFunction.java
index 0c1abba8f0..e7c7df2a22 100644
---
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/LengthFunction.java
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/LengthFunction.java
@@ -22,21 +22,31 @@ import org.apache.inlong.sdk.transform.process.Context;
import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+import net.sf.jsqlparser.expression.Expression;
import net.sf.jsqlparser.expression.Function;
+import java.nio.charset.Charset;
+import java.util.List;
+
/**
* LengthFunction
- * description: length(string)
- * - return the length of the string
+ * description: length(string,[charsetName])
+ * - return the byte length of the string
* - return NULL if the string is NULL
*/
@TransformFunction(names = {"length"})
public class LengthFunction implements ValueParser {
private final ValueParser stringParser;
+private ValueParser charSetNameParser;
+private final Charset DEFAULT_CHARSET = Charset.defaultCharset();
public LengthFunction(Function expr) {
-stringParser =
OperatorTools.buildParser(expr.getParameters().getExpressions().get(0));
+List expressions = expr.getParameters().getExpressions();
+stringParser = OperatorTools.buildParser(expressions.get(0));
+if (expres
(inlong) branch master updated (4ba289dabd -> 7845da55c0)
This is an automated email from the ASF dual-hosted git repository. luchunliang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong.git from 4ba289dabd [INLONG-11237][SDK] Transform SQL supports CHAR_LENGTH function (#11275) add 7845da55c0 [INLONG-11233][SDK] Transform SQL supports mid function (#11234) No new revisions were added by this update. Summary of changes: .../process/function/SubstringFunction.java | 2 +- .../function/string/TestSubstringFunction.java | 20 2 files changed, 21 insertions(+), 1 deletion(-)
Re: [PR] [INLONG-11233][SDK] Transform SQL supports mid function [inlong]
luchunliang merged PR #11234: URL: https://github.com/apache/inlong/pull/11234 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11301][SDK] Transform SQL supports "JSON_ARRAY_INSERT" function [inlong]
emptyOVO commented on code in PR #11319:
URL: https://github.com/apache/inlong/pull/11319#discussion_r1794549509
##
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/JsonArrayInsertFunction.java:
##
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson.JSONArray;
+import com.alibaba.fastjson.JSONObject;
+import com.alibaba.fastjson.JSONPath;
+import net.sf.jsqlparser.expression.Expression;
+import net.sf.jsqlparser.expression.Function;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * JsonArrayInsertFunction -> JSON_ARRAY_INSERT(json_doc, path, val[, path,
val] ...)
+ * description:
+ * - return NULL if any argument is NULL;
+ * - return the document inserted into the array.
+ * Note: If multiple groups of parameters are passed in, the parameter
subscripts of the latter groups
+ * need to be based on the document subscripts after the previous group of
parameters are updated.
+ */
+@TransformFunction(names = {"json_array_insert"})
+public class JsonArrayInsertFunction implements ValueParser {
+
+private ValueParser jsonDocParser;
+private List pathValuePairsParser;
+
+public JsonArrayInsertFunction(Function expr) {
+List expressions = expr.getParameters().getExpressions();
+jsonDocParser = OperatorTools.buildParser(expressions.get(0));
+pathValuePairsParser = new ArrayList<>();
+for (int i = 1; i < expressions.size(); i++) {
+
pathValuePairsParser.add(OperatorTools.buildParser(expressions.get(i)));
+}
+}
+
+@Override
+public Object parse(SourceData sourceData, int rowIndex, Context context) {
+Object jsonDocObj = jsonDocParser.parse(sourceData, rowIndex, context);
+if (jsonDocObj == null) {
+return null;
+}
+ArrayList pathValuePairs = new ArrayList<>();
+for (ValueParser valueParser : pathValuePairsParser) {
+pathValuePairs.add(valueParser.parse(sourceData, rowIndex,
context));
+}
+return jsonArrayInsert(jsonDocObj.toString(), pathValuePairs);
+}
+
+public static String jsonArrayInsert(String jsonDoc, ArrayList
pathValuePairs) {
+if (jsonDoc == null || pathValuePairs == null || pathValuePairs.size()
% 2 != 0) {
+return null;
+}
+
+Object jsonObject = JSON.parse(jsonDoc);
+
+for (int i = 0; i < pathValuePairs.size(); i += 2) {
+String path = (String) pathValuePairs.get(i);
+Object value = pathValuePairs.get(i + 1);
+
+if (!path.endsWith("]")) {
+throw new IllegalArgumentException("Path must end with an
array index: " + path);
+}
+
+// Find the parent path
+String parentPath = path.substring(0, path.lastIndexOf('['));
+Object parentNode = JSONPath.eval(jsonObject, parentPath);
+
+if (parentNode instanceof JSONArray) {
+// If the parent path is an array, perform the insert operation
+insertIntoArray((JSONArray) parentNode, path, value);
+} else if (parentNode instanceof JSONObject) {
+// If the parent path is an object, try inserting it into the
array inside the object
+String arrayIndexPart = path.substring(path.lastIndexOf('['),
path.lastIndexOf(']') + 1);
+handleArrayInsertionInObject((JSONObject) parentNode,
arrayIndexPart, value);
+} else {
+throw new IllegalArgumentException("Invalid path or target
node is not an array or object: " + path);
+}
+}
+
+return JSON.toJSONString(jsonObject);
+}
+
+private static void insertIntoArray(JSONArray array, String path, Object
value) {
Review Comment:
Same
--
This is
Re: [PR] [INLONG-11301][SDK] Transform SQL supports "JSON_ARRAY_INSERT" function [inlong]
emptyOVO commented on code in PR #11319:
URL: https://github.com/apache/inlong/pull/11319#discussion_r1794549379
##
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/JsonArrayInsertFunction.java:
##
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson.JSONArray;
+import com.alibaba.fastjson.JSONObject;
+import com.alibaba.fastjson.JSONPath;
+import net.sf.jsqlparser.expression.Expression;
+import net.sf.jsqlparser.expression.Function;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * JsonArrayInsertFunction -> JSON_ARRAY_INSERT(json_doc, path, val[, path,
val] ...)
+ * description:
+ * - return NULL if any argument is NULL;
+ * - return the document inserted into the array.
+ * Note: If multiple groups of parameters are passed in, the parameter
subscripts of the latter groups
+ * need to be based on the document subscripts after the previous group of
parameters are updated.
+ */
+@TransformFunction(names = {"json_array_insert"})
+public class JsonArrayInsertFunction implements ValueParser {
+
+private ValueParser jsonDocParser;
+private List pathValuePairsParser;
+
+public JsonArrayInsertFunction(Function expr) {
+List expressions = expr.getParameters().getExpressions();
+jsonDocParser = OperatorTools.buildParser(expressions.get(0));
+pathValuePairsParser = new ArrayList<>();
+for (int i = 1; i < expressions.size(); i++) {
+
pathValuePairsParser.add(OperatorTools.buildParser(expressions.get(i)));
+}
+}
+
+@Override
+public Object parse(SourceData sourceData, int rowIndex, Context context) {
+Object jsonDocObj = jsonDocParser.parse(sourceData, rowIndex, context);
+if (jsonDocObj == null) {
+return null;
+}
+ArrayList pathValuePairs = new ArrayList<>();
+for (ValueParser valueParser : pathValuePairsParser) {
+pathValuePairs.add(valueParser.parse(sourceData, rowIndex,
context));
+}
+return jsonArrayInsert(jsonDocObj.toString(), pathValuePairs);
+}
+
+public static String jsonArrayInsert(String jsonDoc, ArrayList
pathValuePairs) {
Review Comment:
private? doesn’t need static I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(inlong) branch master updated (7845da55c0 -> 857d4b4499)
This is an automated email from the ASF dual-hosted git repository.
luchunliang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
from 7845da55c0 [INLONG-11233][SDK] Transform SQL supports mid function
(#11234)
add 857d4b4499 [INLONG-11252][SDK] Transform support PB sink data (#11280)
No new revisions were added by this update.
Summary of changes:
.../inlong/sdk/transform/encode/PbSinkEncoder.java | 115 +++
.../sdk/transform/encode/SinkEncoderFactory.java | 6 +
.../pojo/{KvSourceInfo.java => PbSinkInfo.java}| 44 +++---
.../apache/inlong/sdk/transform/pojo/SinkInfo.java | 1 +
.../process/processor/TestAny2PbProcessor.java | 157 +
5 files changed, 305 insertions(+), 18 deletions(-)
create mode 100644
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/encode/PbSinkEncoder.java
copy
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/pojo/{KvSourceInfo.java
=> PbSinkInfo.java} (63%)
create mode 100644
inlong-sdk/transform-sdk/src/test/java/org/apache/inlong/sdk/transform/process/processor/TestAny2PbProcessor.java
Re: [I] [Feature][Sort] Add ElasticSearch connector on Flink 1.15 [inlong]
aloyszhang commented on issue #9730: URL: https://github.com/apache/inlong/issues/9730#issuecomment-2403850332 duplicated with #10740, close this issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11301][SDK] Transform SQL supports "JSON_ARRAY_INSERT" function [inlong]
Zkplo commented on code in PR #11319:
URL: https://github.com/apache/inlong/pull/11319#discussion_r1794557147
##
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/JsonArrayInsertFunction.java:
##
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson.JSONArray;
+import com.alibaba.fastjson.JSONObject;
+import com.alibaba.fastjson.JSONPath;
+import net.sf.jsqlparser.expression.Expression;
+import net.sf.jsqlparser.expression.Function;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * JsonArrayInsertFunction -> JSON_ARRAY_INSERT(json_doc, path, val[, path,
val] ...)
+ * description:
+ * - return NULL if any argument is NULL;
+ * - return the document inserted into the array.
+ * Note: If multiple groups of parameters are passed in, the parameter
subscripts of the latter groups
+ * need to be based on the document subscripts after the previous group of
parameters are updated.
+ */
+@TransformFunction(names = {"json_array_insert"})
+public class JsonArrayInsertFunction implements ValueParser {
+
+private ValueParser jsonDocParser;
+private List pathValuePairsParser;
+
+public JsonArrayInsertFunction(Function expr) {
+List expressions = expr.getParameters().getExpressions();
+jsonDocParser = OperatorTools.buildParser(expressions.get(0));
+pathValuePairsParser = new ArrayList<>();
+for (int i = 1; i < expressions.size(); i++) {
+
pathValuePairsParser.add(OperatorTools.buildParser(expressions.get(i)));
+}
+}
+
+@Override
+public Object parse(SourceData sourceData, int rowIndex, Context context) {
+Object jsonDocObj = jsonDocParser.parse(sourceData, rowIndex, context);
+if (jsonDocObj == null) {
+return null;
+}
+ArrayList pathValuePairs = new ArrayList<>();
+for (ValueParser valueParser : pathValuePairsParser) {
+pathValuePairs.add(valueParser.parse(sourceData, rowIndex,
context));
+}
+return jsonArrayInsert(jsonDocObj.toString(), pathValuePairs);
+}
+
+public static String jsonArrayInsert(String jsonDoc, ArrayList
pathValuePairs) {
Review Comment:
Thk! I think it makes sense and have modified it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] [Feature][SDK] Supporting Data Sharding with GroupBy Semantics [inlong]
luchunliang commented on issue #10119: URL: https://github.com/apache/inlong/issues/10119#issuecomment-2403858290 Close -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11229][SDK] Transform Support ELT Function [inlong]
ying-hua commented on PR #11248: URL: https://github.com/apache/inlong/pull/11248#issuecomment-2403947793 > @ying-hua please rebase from the master branch to fix the failed UTs. done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-11199][Sort] Integrate Grafana Loki for connectors [inlong]
qy-liuhuo commented on PR #11212: URL: https://github.com/apache/inlong/pull/11212#issuecomment-2404033865 This new commit make log reporting related containers optional： When running docker-compose normally, only necessary containers will be started. If you need to enable the opentelemetry log reporting function of the Sort module, you can add the `--profile sort-report` option when starting, for example, `docker compose --profile sort-report up -d` To support the profiles feature, we need to update the version setting of `docker-compose.yml` from `2.4` to `3.0`, and the `docker compose` version needs to be higher than `1.28.0`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
