[GitHub] [inlong] dockerzhang merged pull request #6739: [INLONG-6738][DataProxy] New sink architecture integration
dockerzhang merged PR #6739: URL: https://github.com/apache/inlong/pull/6739 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-6738][DataProxy] New sink architecture integration (#6739)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 5d062bc93 [INLONG-6738][DataProxy] New sink architecture integration
(#6739)
5d062bc93 is described below
commit 5d062bc93ba62a0d4e09268d5b328fa6663dff55
Author: woofyzhao <490467...@qq.com>
AuthorDate: Wed Dec 7 17:27:19 2022 +0800
[INLONG-6738][DataProxy] New sink architecture integration (#6739)
---
.../inlong/common/heartbeat/ComponentHeartbeat.java | 12
.../apache/inlong/common/heartbeat/HeartbeatMsg.java| 7 ++-
inlong-dataproxy/conf/common.properties | 5 +
inlong-dataproxy/conf/dataproxy-pulsar.conf | 15 ++-
.../inlong/dataproxy/config/RemoteConfigManager.java| 11 ++-
.../apache/inlong/dataproxy/consts/ConfigConstants.java | 1 +
.../inlong/dataproxy/heartbeat/HeartbeatManager.java| 1 +
.../dataproxy/sink/mq/MessageQueueZoneProducer.java | 4
.../inlong/dataproxy/sink/mq/pulsar/PulsarHandler.java | 17 +
.../inlong/dataproxy/source/ServerMessageHandler.java | 6 --
.../apache/inlong/dataproxy/utils/ConfStringUtils.java | 3 +++
.../manager/service/heartbeat/HeartbeatManager.java | 2 ++
.../service/repository/DataProxyConfigRepository.java | 3 +++
13 files changed, 66 insertions(+), 21 deletions(-)
diff --git
a/inlong-common/src/main/java/org/apache/inlong/common/heartbeat/ComponentHeartbeat.java
b/inlong-common/src/main/java/org/apache/inlong/common/heartbeat/ComponentHeartbeat.java
index 567992918..96c173672 100644
---
a/inlong-common/src/main/java/org/apache/inlong/common/heartbeat/ComponentHeartbeat.java
+++
b/inlong-common/src/main/java/org/apache/inlong/common/heartbeat/ComponentHeartbeat.java
@@ -31,6 +31,8 @@ public class ComponentHeartbeat {
private String clusterTag;
+private String extTag;
+
private String clusterName;
private String componentType;
@@ -49,11 +51,12 @@ public class ComponentHeartbeat {
public ComponentHeartbeat() {
}
-public ComponentHeartbeat(String clusterTag, String clusterName,
-String componentType, String ip, String port,
-String inCharges, String protocolType) {
+public ComponentHeartbeat(String clusterTag, String extTag,
+String clusterName, String componentType, String ip,
+String port, String inCharges, String protocolType) {
this.nodeSrvStatus = NodeSrvStatus.OK;
this.clusterTag = clusterTag;
+this.extTag = extTag;
this.clusterName = clusterName;
this.componentType = componentType;
this.ip = ip;
@@ -64,11 +67,12 @@ public class ComponentHeartbeat {
}
public ComponentHeartbeat(NodeSrvStatus nodeSrvStatus,
-String clusterTag, String clusterName,
+String clusterTag, String extTag, String clusterName,
String componentType, String ip, String port,
String inCharges, String protocolType, int loadValue) {
this.nodeSrvStatus = nodeSrvStatus;
this.clusterTag = clusterTag;
+this.extTag = extTag;
this.clusterName = clusterName;
this.componentType = componentType;
this.ip = ip;
diff --git
a/inlong-common/src/main/java/org/apache/inlong/common/heartbeat/HeartbeatMsg.java
b/inlong-common/src/main/java/org/apache/inlong/common/heartbeat/HeartbeatMsg.java
index c386c9b60..cbfddf470 100644
---
a/inlong-common/src/main/java/org/apache/inlong/common/heartbeat/HeartbeatMsg.java
+++
b/inlong-common/src/main/java/org/apache/inlong/common/heartbeat/HeartbeatMsg.java
@@ -75,6 +75,11 @@ public class HeartbeatMsg {
*/
private String clusterTag;
+/**
+ * Ext tag of cluster, key=value pairs seperated by &
+ */
+private String extTag;
+
/**
* Name of responsible person, separated by commas(,)
*/
@@ -96,7 +101,7 @@ public class HeartbeatMsg {
private Integer load = 0x;
public ComponentHeartbeat componentHeartbeat() {
-return new ComponentHeartbeat(nodeSrvStatus, clusterTag, clusterName,
+return new ComponentHeartbeat(nodeSrvStatus, clusterTag, extTag,
clusterName,
componentType, ip, port, inCharges, protocolType, load);
}
}
diff --git a/inlong-dataproxy/conf/common.properties
b/inlong-dataproxy/conf/common.properties
index 557a10cb2..a13d63ac6 100644
--- a/inlong-dataproxy/conf/common.properties
+++ b/inlong-dataproxy/conf/common.properties
@@ -24,6 +24,7 @@ manager.auth.secretKey=
# proxy cluster name
proxy.cluster.name=default_dataproxy
proxy.cluster.tag=default_cluster
+proxy.cluster.extTag=default=true
proxy.cluster.inCharges=admin
# check interval of local config (millisecond)
configCheckInterval=1
@@ -38,3 +39,7 @
[GitHub] [inlong] liaorui closed pull request #6741: [INLONG-6740][Sort][Manager]StarRocks connector supports transferring all tables for all schemas in one database
liaorui closed pull request #6741: [INLONG-6740][Sort][Manager]StarRocks connector supports transferring all tables for all schemas in one database URL: https://github.com/apache/inlong/pull/6741 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang opened a new pull request, #6778: [INLONG-6777][CVE] Bump com.h2database.h2 to 2.1.214
dockerzhang opened a new pull request, #6778: URL: https://github.com/apache/inlong/pull/6778 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #6777 ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve?* ### Modifications *Describe the modifications you've done.* ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #6748: [INLONG-6747][Sort] StarRocks connector supports transferring all tables for all schemas in one database
EMsnap commented on code in PR #6748:
URL: https://github.com/apache/inlong/pull/6748#discussion_r1042089775
##
inlong-sort/sort-connectors/starrocks/src/main/java/org/apache/inlong/sort/starrocks/manager/StarRocksStreamLoadVisitor.java:
##
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.starrocks.manager;
+
+import com.starrocks.connector.flink.manager.StarRocksSinkBufferEntity;
+import
com.starrocks.connector.flink.manager.StarRocksStreamLoadFailedException;
+import com.starrocks.connector.flink.row.sink.StarRocksDelimiterParser;
+import com.starrocks.connector.flink.row.sink.StarRocksSinkOP;
+import com.starrocks.connector.flink.table.sink.StarRocksSinkOptions;
+import com.starrocks.shade.com.alibaba.fastjson.JSON;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.HttpURLConnection;
+import java.net.URL;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import org.apache.commons.codec.binary.Base64;
+import org.apache.http.HttpEntity;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.entity.ByteArrayEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.DefaultRedirectStrategy;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.util.EntityUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StarRocksStreamLoadVisitor implements Serializable {
Review Comment:
Please add license
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] Yizhou-Yang opened a new pull request, #6779: [INLONG-6730][Sort] Supports dirty data side-output for sqlserver sink
Yizhou-Yang opened a new pull request, #6779: URL: https://github.com/apache/inlong/pull/6779 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #XYZ ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve?* ### Modifications *Describe the modifications you've done.* ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] chyueyi commented on issue #6723: 官方文档Doris多表写入文档错误
chyueyi commented on issue #6723: URL: https://github.com/apache/inlong/issues/6723#issuecomment-1340976755 > 这里是flink任务运行需要的Flink SQL。多表和整库同步的逻辑可以只关注多表部分，忽略单表的部分，可能会更容易理解。 > 多表的核心是： > 1.mysql按正则读多表 > 'database-name' = 'user_db', > 'table-name' = 'user_id_*' > (这部分文档中写的确实有问题) > 2.doris写设置如下参数： > 'sink.multiple.enable' > 'sink.multiple.format' > 'sink.multiple.database-pattern' > 'sink.multiple.table-pattern' 感谢大佬的回答👍🏻 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] kuansix commented on issue #6723: 官方文档Doris多表写入文档错误
kuansix commented on issue #6723: URL: https://github.com/apache/inlong/issues/6723#issuecomment-1340977158 来信已收到，谢谢！ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] gong commented on pull request #6748: [INLONG-6747][Sort] StarRocks connector supports transferring all tables for all schemas in one database
gong commented on PR #6748: URL: https://github.com/apache/inlong/pull/6748#issuecomment-1340981698 @liaorui Please add packaging config for this connector in `inlong-distribution/src/main/assemblies/sort-connectors.xml` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #6778: [INLONG-6777][CVE] Bump com.h2database.h2 to 2.1.214
dockerzhang merged PR #6778: URL: https://github.com/apache/inlong/pull/6778 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-6777][CVE] Bump com.h2database.h2 to 2.1.214 (#6778)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/inlong.git The following commit(s) were added to refs/heads/master by this push: new 05fa2f347 [INLONG-6777][CVE] Bump com.h2database.h2 to 2.1.214 (#6778) 05fa2f347 is described below commit 05fa2f347d11ef5d701c7756978fe5b9ec1c1a1e Author: Charles Zhang AuthorDate: Wed Dec 7 21:58:04 2022 +0800 [INLONG-6777][CVE] Bump com.h2database.h2 to 2.1.214 (#6778) --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index ab939b419..d9e9dd0b8 100644 --- a/pom.xml +++ b/pom.xml @@ -91,7 +91,7 @@ 1.4.2 5.3.1 -2.1.210 +2.1.214 2.0.0 1.8.0.Final 6.14.6
[GitHub] [inlong] ZuoFengZhang opened a new pull request, #6783: [INLONG-6782][Sort] Introduce apache hudi sort connector
ZuoFengZhang opened a new pull request, #6783: URL: https://github.com/apache/inlong/pull/6783 ### Prepare a Pull Request **Introduce apache hudi sort connector** - Fixes #6782 ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve?* ### Modifications *Describe the modifications you've done.* ### Verifying this change *(Please pick either of the following options)* - [x] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #6774: [INLONG-6773][Manager] Fix the problem of sink cannot jump from the configuration to successful configuration
dockerzhang merged PR #6774: URL: https://github.com/apache/inlong/pull/6774 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-6773][Manager] Fix the problem of sink cannot jump from the configuration to successful configuration (#6774)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new db8a69dd6 [INLONG-6773][Manager] Fix the problem of sink cannot jump
from the configuration to successful configuration (#6774)
db8a69dd6 is described below
commit db8a69dd65859ece7f55e9811453821533c025ae
Author: fuweng11 <76141879+fuwen...@users.noreply.github.com>
AuthorDate: Wed Dec 7 22:28:33 2022 +0800
[INLONG-6773][Manager] Fix the problem of sink cannot jump from the
configuration to successful configuration (#6774)
---
.../apache/inlong/manager/service/sink/StreamSinkServiceImpl.java | 8 ++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/sink/StreamSinkServiceImpl.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/sink/StreamSinkServiceImpl.java
index 9c776748b..bd6332fd2 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/sink/StreamSinkServiceImpl.java
+++
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/sink/StreamSinkServiceImpl.java
@@ -120,7 +120,9 @@ public class StreamSinkServiceImpl implements
StreamSinkService {
int id = sinkOperator.saveOpt(request, operator);
boolean streamSuccess =
StreamStatus.CONFIG_SUCCESSFUL.getCode().equals(streamEntity.getStatus());
if (streamSuccess ||
StreamStatus.CONFIG_FAILED.getCode().equals(streamEntity.getStatus())) {
-SinkStatus nextStatus = SinkStatus.CONFIG_ING;
+boolean enableCreateResource =
InlongConstants.ENABLE_CREATE_RESOURCE.equals(
+request.getEnableCreateResource());
+SinkStatus nextStatus = enableCreateResource ?
SinkStatus.CONFIG_ING : SinkStatus.CONFIG_SUCCESSFUL;
StreamSinkEntity sinkEntity = sinkMapper.selectByPrimaryKey(id);
sinkEntity.setStatus(nextStatus.getCode());
sinkMapper.updateStatus(sinkEntity);
@@ -246,7 +248,9 @@ public class StreamSinkServiceImpl implements
StreamSinkService {
SinkStatus nextStatus = null;
boolean streamSuccess =
StreamStatus.CONFIG_SUCCESSFUL.getCode().equals(streamEntity.getStatus());
if (streamSuccess ||
StreamStatus.CONFIG_FAILED.getCode().equals(streamEntity.getStatus())) {
-nextStatus = SinkStatus.CONFIG_ING;
+boolean enableCreateResource =
InlongConstants.ENABLE_CREATE_RESOURCE.equals(
+request.getEnableCreateResource());
+nextStatus = enableCreateResource ? SinkStatus.CONFIG_ING :
SinkStatus.CONFIG_SUCCESSFUL;
}
StreamSinkOperator sinkOperator =
operatorFactory.getInstance(request.getSinkType());
sinkOperator.updateOpt(request, nextStatus, operator);
[GitHub] [inlong] ZuoFengZhang opened a new pull request, #6789: [INLONG-6784][Sort] Support apache hudi LoadNode
ZuoFengZhang opened a new pull request, #6789: URL: https://github.com/apache/inlong/pull/6789 ### Prepare a Pull Request *[INLONG-6784][Sort] Support apache hudi LoadNode* - Fixes #6784 ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve?* ### Modifications *Describe the modifications you've done.* ### Verifying this change *(Please pick either of the following options)* - [x] This change is a trivial rework/code cleanup without any test coverage. - [x] This change is already covered by existing tests, such as: *(please describe tests)* - [x] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #6677: [INLONG-6676][Manager] Support list all InlongTopicInfo under a given cluster tag
healchow commented on code in PR #6677:
URL: https://github.com/apache/inlong/pull/6677#discussion_r1042287801
##
inlong-manager/manager-web/src/main/java/org/apache/inlong/manager/web/controller/InlongGroupController.java:
##
@@ -95,6 +97,12 @@ public Response getTopic(@PathVariable
String groupId) {
return Response.success(groupService.getTopic(groupId));
}
+@GetMapping(value = "/group/listTopicsByTag/{clusterTag}")
Review Comment:
Are you still working on this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] healchow commented on pull request #6741: [INLONG-6740][Sort][Manager]StarRocks connector supports transferring all tables for all schemas in one database
healchow commented on PR #6741: URL: https://github.com/apache/inlong/pull/6741#issuecomment-1341069854 @liaorui Suggest after merging #6746, commit the changes of the Sort module to this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #6753: [INLONG-6734][Manager] Support non-InlongMessage format of Kafka and Pulsar
healchow commented on code in PR #6753:
URL: https://github.com/apache/inlong/pull/6753#discussion_r1042309796
##
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/source/kafka/KafkaSource.java:
##
@@ -90,6 +90,8 @@ public class KafkaSource extends StreamSource {
@ApiModelProperty(value = "Data field escape symbol")
private String dataEscapeChar;
+@ApiModelProperty("Whether wrap content with InlongMessage")
+private boolean wrapWithInlongMessage = true;
Review Comment:
1. Add one blank line before the new field.
2. Suggest changing the field name to `wrapWithInlongMsg`, because
`InlongMsg` is the same as the `InLongMsgXxx`.
```suggestion
@ApiModelProperty("Whether wrap content with InlongMsg")
private boolean wrapWithInlongMsg = true;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] ZuoFengZhang opened a new pull request, #6790: [INLONG-6785][Manager] Support register and manage hudi sort
ZuoFengZhang opened a new pull request, #6790: URL: https://github.com/apache/inlong/pull/6790 ### Prepare a Pull Request *[Manager] Support register and manage hudi sort* - Fixes #6785 ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve?* ### Modifications *Describe the modifications you've done.* ### Verifying this change *(Please pick either of the following options)* - [x] This change is a trivial rework/code cleanup without any test coverage. - [x] This change is already covered by existing tests, such as: *(please describe tests)* - [x] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] ZuoFengZhang opened a new pull request, #6791: [INLONG-6786][Dashboard] Supoort Apache hudi in frontend
ZuoFengZhang opened a new pull request, #6791: URL: https://github.com/apache/inlong/pull/6791 ### Prepare a Pull Request *[Dashboard] Supoort Apache hudi in frontend* - Fixes #6786 ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve?* ### Modifications *Describe the modifications you've done.* ### Verifying this change *(Please pick either of the following options)* - [x] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] ZuoFengZhang closed pull request #6645: [INLONG-6646][Sort] Support Apache Hudi connector
ZuoFengZhang closed pull request #6645: [INLONG-6646][Sort] Support Apache Hudi connector URL: https://github.com/apache/inlong/pull/6645 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] healchow merged pull request #6753: [INLONG-6734][Manager] Support non-InlongMessage format of Kafka and Pulsar
healchow merged PR #6753: URL: https://github.com/apache/inlong/pull/6753 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated (db8a69dd6 -> 01de2893b)
This is an automated email from the ASF dual-hosted git repository. healchow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong.git from db8a69dd6 [INLONG-6773][Manager] Fix the problem of sink cannot jump from the configuration to successful configuration (#6774) add 01de2893b [INLONG-6734][Manager] Support non-InlongMessage format of Kafka and Pulsar (#6753) No new revisions were added by this update. Summary of changes: .../manager/common/consts/InlongConstants.java | 2 + .../manager/pojo/sort/util/ExtractNodeUtils.java | 109 ++--- .../manager/pojo/source/kafka/KafkaSource.java | 3 + .../manager/pojo/source/pulsar/PulsarSource.java | 4 + .../service/source/kafka/KafkaSourceOperator.java | 7 ++ .../source/pulsar/PulsarSourceOperator.java| 6 ++ 6 files changed, 73 insertions(+), 58 deletions(-)
[GitHub] [inlong] healchow commented on a diff in pull request #6790: [INLONG-6785][Manager] Support register and manage hudi sort
healchow commented on code in PR #6790: URL: https://github.com/apache/inlong/pull/6790#discussion_r1042845015 ## inlong-manager/manager-plugins/src/main/java/org/apache/inlong/manager/plugin/listener/RestartStreamListener.java: ## @@ -38,12 +43,6 @@ import org.apache.inlong.manager.workflow.event.ListenerResult; import org.apache.inlong.manager.workflow.event.task.SortOperateListener; -import java.util.HashMap; Review Comment: Please refer this: https://github.com/apache/inlong/discussions/3082, and reformat those packages' order, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-6765][Sort] Supports dirty data side-output for Iceberg sink (#6766)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 08db5ea09 [INLONG-6765][Sort] Supports dirty data side-output for
Iceberg sink (#6766)
08db5ea09 is described below
commit 08db5ea092f825ab44dace4534a9c67cc748800c
Author: yunqingmoswu <44659300+yunqingmo...@users.noreply.github.com>
AuthorDate: Thu Dec 8 09:54:36 2022 +0800
[INLONG-6765][Sort] Supports dirty data side-output for Iceberg sink (#6766)
---
.../org/apache/inlong/sort/base/Constants.java | 11 ++-
.../apache/inlong/sort/base/dirty/DirtyData.java | 41 +++-
.../apache/inlong/sort/base/dirty/DirtyType.java | 12 +++
.../sort/base/dirty/sink/log/LogDirtySink.java | 27 +++---
.../sort/base/dirty/sink/s3/S3DirtySink.java | 33 ---
.../inlong/sort/base/dirty/utils/FormatUtils.java | 30 ++
.../inlong/sort/base/sink/MultipleSinkOption.java | 1 -
.../inlong/sort/base/dirty/FormatUtilsTest.java| 4 +-
.../sort/iceberg/FlinkDynamicTableFactory.java | 13 ++-
.../inlong/sort/iceberg/IcebergTableSink.java | 16 +++-
.../apache/inlong/sort/iceberg/sink/FlinkSink.java | 35 +--
.../sink/multiple/DynamicSchemaHandleOperator.java | 106 -
.../sink/multiple/IcebergMultipleStreamWriter.java | 19 +++-
.../sink/multiple/IcebergSingleStreamWriter.java | 60 ++--
.../sort/parser/IcebergNodeSqlParserTest.java | 11 ++-
15 files changed, 328 insertions(+), 91 deletions(-)
diff --git
a/inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java
b/inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java
index 983b3c207..23676ec0f 100644
---
a/inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java
+++
b/inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java
@@ -227,11 +227,11 @@ public final class Constants {
.stringType()
.noDefaultValue()
.withDescription(
-"The identifier of dirty data, "
-+ "it will be used for filename generation
of file dirty sink, "
+"The identifier of dirty data, it will be used for
filename generation of file dirty sink, "
+ "topic generation of mq dirty sink,
tablename generation of database, etc."
+ "and it supports variable replace like
'${variable}'."
-+ "There are two system
variables[SYSTEM_TIME|DIRTY_TYPE] are currently supported,"
++ "There are several system
variables[SYSTEM_TIME|DIRTY_TYPE|DIRTY_MESSAGE] "
++ "are currently supported, "
+ "and the support of other variables is
determined by the connector.");
public static final ConfigOption DIRTY_SIDE_OUTPUT_ENABLE =
ConfigOptions.key("dirty.side-output.enable")
@@ -266,7 +266,8 @@ public final class Constants {
.withDescription(
"The labels of dirty side-output, format is
'key1=value1&key2=value2', "
+ "it supports variable replace like
'${variable}',"
-+ "There are two system
variables[SYSTEM_TIME|DIRTY_TYPE] are currently supported,"
++ "There are two system
variables[SYSTEM_TIME|DIRTY_TYPE|DIRTY_MESSAGE] "
++ "are currently supported,"
+ " and the support of other variables is
determined by the connector.");
public static final ConfigOption DIRTY_SIDE_OUTPUT_LOG_TAG =
ConfigOptions.key("dirty.side-output.log-tag")
@@ -274,7 +275,7 @@ public final class Constants {
.defaultValue("DirtyData")
.withDescription(
"The log tag of dirty side-output, it supports
variable replace like '${variable}'."
-+ "There are two system
variables[SYSTEM_TIME|DIRTY_TYPE] are currently supported,"
++ "There are two system
variables[SYSTEM_TIME|DIRTY_TYPE|DIRTY_MESSAGE] are currently supported,"
+ " and the support of other variables is
determined by the connector.");
public static final ConfigOption DIRTY_SIDE_OUTPUT_FIELD_DELIMITER
=
ConfigOptions.key("dirty.side-output.field-delimiter")
diff --git
a/inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/dirty/DirtyData.java
b/inlong-sort/sort
[GitHub] [inlong] dockerzhang merged pull request #6766: [INLONG-6765][Sort] Supports dirty data side-output for Iceberg sink
dockerzhang merged PR #6766: URL: https://github.com/apache/inlong/pull/6766 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang commented on pull request #6741: [INLONG-6740][Sort][Manager] StarRocks connector supports transferring all tables for all schemas in one database
dockerzhang commented on PR #6741: URL: https://github.com/apache/inlong/pull/6741#issuecomment-1341863304 split to #6746 and #6748, close it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang closed pull request #6741: [INLONG-6740][Sort][Manager] StarRocks connector supports transferring all tables for all schemas in one database
dockerzhang closed pull request #6741: [INLONG-6740][Sort][Manager] StarRocks connector supports transferring all tables for all schemas in one database URL: https://github.com/apache/inlong/pull/6741 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] leezng commented on a diff in pull request #6791: [INLONG-6786][Dashboard] Supoort Apache Hudi sink management
leezng commented on code in PR #6791: URL: https://github.com/apache/inlong/pull/6791#discussion_r1042857586 ## inlong-dashboard/src/locales/cn.json: ## @@ -177,6 +177,25 @@ "meta.Sinks.Iceberg.FieldDescription": "字段描述", "meta.Sinks.Iceberg.PartitionStrategy": "分区策略", "meta.Sinks.Iceberg.DataNodeName": "数据节点", + "meta.Sinks.Hudi.DbName": "DB名称", + "meta.Sinks.Hudi.TableName": "表名称", + "meta.Sinks.Hudi.Warehouse": "仓库路径", + "meta.Sinks.Hudi.FileFormat": "⽂件格式", + "meta.Sinks.Hudi.Description": "表描述", + "meta.Sinks.Hudi.ExtList": "属性", + "meta.Sinks.Hudi.DataConsistency": "数据一致性", + "meta.Sinks.Hudi.FieldName": "字段名", + "meta.Sinks.Hudi.FieldNameRule": "以英文字母或下划线开头，只能包含英文字母、数字、下划线", + "meta.Sinks.Hudi.FieldType": "字段类型", + "meta.Sinks.Hudi.FieldDescription": "字段描述", + "meta.Sinks.Hudi.IsPrimaryKey": "主键", Review Comment: Is this key(`IsPrimaryKey`) not used? ## inlong-dashboard/src/locales/cn.json: ## @@ -177,6 +177,25 @@ "meta.Sinks.Iceberg.FieldDescription": "字段描述", "meta.Sinks.Iceberg.PartitionStrategy": "分区策略", "meta.Sinks.Iceberg.DataNodeName": "数据节点", + "meta.Sinks.Hudi.DbName": "DB名称", + "meta.Sinks.Hudi.TableName": "表名称", + "meta.Sinks.Hudi.Warehouse": "仓库路径", + "meta.Sinks.Hudi.FileFormat": "⽂件格式", + "meta.Sinks.Hudi.Description": "表描述", + "meta.Sinks.Hudi.ExtList": "属性", + "meta.Sinks.Hudi.DataConsistency": "数据一致性", + "meta.Sinks.Hudi.FieldName": "字段名", + "meta.Sinks.Hudi.FieldNameRule": "以英文字母或下划线开头，只能包含英文字母、数字、下划线", + "meta.Sinks.Hudi.FieldType": "字段类型", + "meta.Sinks.Hudi.FieldDescription": "字段描述", + "meta.Sinks.Hudi.IsPrimaryKey": "主键", + "meta.Sinks.Hudi.PrimaryKey": "主键", + "meta.Sinks.Hudi.IsPartitionKey": "分区字段", Review Comment: Is this key(`IsPartitionKey`) not used? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] leosanqing commented on issue #6771: [Improve][Manager] Add login failure limit
leosanqing commented on issue #6771: URL: https://github.com/apache/inlong/issues/6771#issuecomment-1341943519 I'll try to do, please assign it to me -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] yunqingmoswu opened a new pull request, #6793: [INLONG-6792][Sort] Supports dirty data side-output for hbase sink
yunqingmoswu opened a new pull request, #6793: URL: https://github.com/apache/inlong/pull/6793 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title: [INLONG-6792][Sort] Supports dirty data side-output for hbase sink *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #6792 ### Motivation Supports dirty data side-output for hbase sink. In this part: 1. Load 'DirtySinkFactory' and create 'DirtySink' by the config 2. It needs to determine whether it is dirty data in the connector. 3. Side output dirty data by the 'DirtySink' dependents on the configured, the built-in side-out of dirty data has 'LogDirtySink'(#6618) and 'S3DirtySink'(#6655). ### Modifications 1. Create a dirty sink and inject it into hbase sink 2. Add dirty handle for 'HBaseSinkFunction' 3. Add a unit test for this ### Verifying this change *(Please pick either of the following options)* - [x] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] leosanqing commented on a diff in pull request #6790: [INLONG-6785][Manager] Support register and manage the resource of Apache Hudi
leosanqing commented on code in PR #6790: URL: https://github.com/apache/inlong/pull/6790#discussion_r1042918024 ## inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sink/hudi/HudiSink.java: ## @@ -0,0 +1,91 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.inlong.manager.pojo.sink.hudi; + +import io.swagger.annotations.ApiModel; +import io.swagger.annotations.ApiModelProperty; +import java.util.HashMap; +import java.util.List; +import lombok.AllArgsConstructor; +import lombok.Builder; +import lombok.Data; +import lombok.EqualsAndHashCode; +import lombok.ToString; +import lombok.experimental.SuperBuilder; +import org.apache.inlong.manager.common.consts.SinkType; +import org.apache.inlong.manager.common.util.CommonBeanUtils; +import org.apache.inlong.manager.common.util.JsonTypeDefine; +import org.apache.inlong.manager.pojo.sink.SinkRequest; +import org.apache.inlong.manager.pojo.sink.StreamSink; + +/** + * Iceberg sink info Review Comment: Hudi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] ZuoFengZhang commented on a diff in pull request #6790: [INLONG-6785][Manager] Support register and manage the resource of Apache Hudi
ZuoFengZhang commented on code in PR #6790: URL: https://github.com/apache/inlong/pull/6790#discussion_r1042946640 ## inlong-manager/manager-plugins/src/main/java/org/apache/inlong/manager/plugin/listener/RestartStreamListener.java: ## @@ -38,12 +43,6 @@ import org.apache.inlong.manager.workflow.event.ListenerResult; import org.apache.inlong.manager.workflow.event.task.SortOperateListener; -import java.util.HashMap; Review Comment: Ok, I have a question why the spotless script doesn't work -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang commented on a diff in pull request #6790: [INLONG-6785][Manager] Support register and manage the resource of Apache Hudi
dockerzhang commented on code in PR #6790: URL: https://github.com/apache/inlong/pull/6790#discussion_r1042960165 ## inlong-manager/manager-plugins/src/main/java/org/apache/inlong/manager/plugin/listener/RestartStreamListener.java: ## @@ -38,12 +43,6 @@ import org.apache.inlong.manager.workflow.event.ListenerResult; import org.apache.inlong.manager.workflow.event.task.SortOperateListener; -import java.util.HashMap; Review Comment: > Please refer this: #3082, and reformat those packages' order, thanks. spotless support defines the order of the import packages. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] allendang001 commented on issue #627: [Feature][Doc] Translate http report guide
allendang001 commented on issue #627: URL: https://github.com/apache/inlong-website/issues/627#issuecomment-1342114366 please assign to me? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] gong commented on pull request #6748: [INLONG-6747][Sort] StarRocks connector supports transferring all tables for all schemas in one database
gong commented on PR #6748: URL: https://github.com/apache/inlong/pull/6748#issuecomment-1342115939 @liaorui Please add base shade in starrocks.pom, ref to iceberg pom. Because It can avoid modifying base lead classloader problem. ``` org.apache.inlong.sort.base org.apache.inlong.sort.starraocks.shaded.org.apache.inlong.sort.base ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #6793: [INLONG-6792][Sort] Supports dirty data side-output for hbase sink
EMsnap commented on code in PR #6793:
URL: https://github.com/apache/inlong/pull/6793#discussion_r1042971675
##
inlong-sort/sort-core/src/test/java/org/apache/inlong/sort/parser/HbaseLoadFlinkSqlParseTest.java:
##
@@ -71,14 +72,21 @@ private MySqlExtractNode buildMySQLExtractNode() {
* @return hbase load node
*/
private HbaseLoadNode buildHbaseLoadNode() {
+Map properties = new LinkedHashMap<>();
+properties.put("dirty.side-output.connector", "log");
+properties.put("dirty.ignore", "true");
+properties.put("dirty.side-output.enable", "true");
+properties.put("dirty.side-output.format", "csv");
+properties.put("dirty.side-output.labels",
+
"SYSTEM_TIME=${SYSTEM_TIME}&DIRTY_TYPE=${DIRTY_TYPE}&database=inlong&table=inlong_iceberg");
Review Comment:
table=inlong_hbase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong-website] allendang001 opened a new pull request, #641: [INLONG-627] [Doc] Translate http report guide (#627)
allendang001 opened a new pull request, #641: URL: https://github.com/apache/inlong-website/pull/641 [[INLONG-627] [Doc] Translate http report guide] translate http report guide -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] allendang001 commented on pull request #641: [INLONG-627] [Doc] Translate http report guide (#627)
allendang001 commented on PR #641: URL: https://github.com/apache/inlong-website/pull/641#issuecomment-1342138438 cc @dockerzhang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] leezng opened a new pull request, #6794: [INLONG-6758][Dashboard] Unify the log function components of Group and Stream
leezng opened a new pull request, #6794: URL: https://github.com/apache/inlong/pull/6794 - Fixes #6758 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] dockerzhang commented on a diff in pull request #641: [INLONG-627] [Doc] Translate http report guide (#627)
dockerzhang commented on code in PR #641: URL: https://github.com/apache/inlong-website/pull/641#discussion_r1043007916 ## docs/sdk/dataproxy-sdk/http.md: ## @@ -1,34 +1,34 @@ --- -title: HTTP 上报 +title: HTTP Report sidebar_position: 3 --- -## 新建实时同步任务 -在 Dashboard 或者通过命令行工具创建任务，数据源类型使用 `Auto Push` (自主推送)。 +## Create real-time synchronization task +Create a task on the Dashboard or through the command line, and use `Auto Push` (autonomous push) as the data source type. -## 方式一：调用 URL 上报 +## Method 1: Call the interface to report ```bash curl -X POST -d 'groupId=give_your_group_id&streamId=give_your_stream_id&dt=data_time&body=give_your_data_body&cnt=1' http://dataproxy_url:46802/dataproxy/message ``` -- 参数说明： +- Parameter Description： -| 参数 | 含义 | 备注 | -|--|--|-| -| groupId | 数据流组 id | | -| streamId | 数据流 ID | | -| body | 推送的数据内容 | | -| dt | 推送的数据时间 | | -| cnt | 推送条数 | | +| parameter | meaning | Remark | +|---|---|-| +| groupId | data stream group id | | Review Comment: data->Data -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] dockerzhang commented on a diff in pull request #641: [INLONG-627] [Doc] Translate http report guide (#627)
dockerzhang commented on code in PR #641: URL: https://github.com/apache/inlong-website/pull/641#discussion_r1043008313 ## docs/sdk/dataproxy-sdk/http.md: ## @@ -1,34 +1,34 @@ --- -title: HTTP 上报 +title: HTTP Report sidebar_position: 3 --- -## 新建实时同步任务 -在 Dashboard 或者通过命令行工具创建任务，数据源类型使用 `Auto Push` (自主推送)。 +## Create real-time synchronization task +Create a task on the Dashboard or through the command line, and use `Auto Push` (autonomous push) as the data source type. -## 方式一：调用 URL 上报 +## Method 1: Call the interface to report ```bash curl -X POST -d 'groupId=give_your_group_id&streamId=give_your_stream_id&dt=data_time&body=give_your_data_body&cnt=1' http://dataproxy_url:46802/dataproxy/message ``` -- 参数说明： +- Parameter Description： -| 参数 | 含义 | 备注 | -|--|--|-| -| groupId | 数据流组 id | | -| streamId | 数据流 ID | | -| body | 推送的数据内容 | | -| dt | 推送的数据时间 | | -| cnt | 推送条数 | | +| parameter | meaning | Remark | +|---|---|-| +| groupId | data stream group id | | +| streamId | data stream ID| | Review Comment: ditto -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] allendang001 commented on a diff in pull request #641: [INLONG-627] [Doc] Translate http report guide (#627)
allendang001 commented on code in PR #641: URL: https://github.com/apache/inlong-website/pull/641#discussion_r1043009756 ## docs/sdk/dataproxy-sdk/http.md: ## @@ -1,34 +1,34 @@ --- -title: HTTP 上报 +title: HTTP Report sidebar_position: 3 --- -## 新建实时同步任务 -在 Dashboard 或者通过命令行工具创建任务，数据源类型使用 `Auto Push` (自主推送)。 +## Create real-time synchronization task +Create a task on the Dashboard or through the command line, and use `Auto Push` (autonomous push) as the data source type. -## 方式一：调用 URL 上报 +## Method 1: Call the interface to report ```bash curl -X POST -d 'groupId=give_your_group_id&streamId=give_your_stream_id&dt=data_time&body=give_your_data_body&cnt=1' http://dataproxy_url:46802/dataproxy/message ``` -- 参数说明： +- Parameter Description： -| 参数 | 含义 | 备注 | -|--|--|-| -| groupId | 数据流组 id | | -| streamId | 数据流 ID | | -| body | 推送的数据内容 | | -| dt | 推送的数据时间 | | -| cnt | 推送条数 | | +| parameter | meaning | Remark | +|---|---|-| +| groupId | data stream group id | | Review Comment: ok，l wiil change it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] yunqingmoswu opened a new pull request, #6796: [INLONG-6795][Sort] Supports dirty data side-output for hive sink
yunqingmoswu opened a new pull request, #6796: URL: https://github.com/apache/inlong/pull/6796 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title: [INLONG-6795][Sort] Supports dirty data side-output for hive sink *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #6795 ### Motivation Supports dirty data side-output for hive sink. In this part: 1. Load 'DirtySinkFactory' and create 'DirtySink' by the config 2. It needs to determine whether it is dirty data in the connector. 3. Side output dirty data by the 'DirtySink' dependents on the configured, the built-in side-out of dirty data has 'LogDirtySink'(#6618) and 'S3DirtySink'(#6655). ### Modifications 1. Create a dirty sink and inject it into hive sink 2. Add dirty handle for 'AbstractStreamingWriter' 3. Add a unit test for this ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] bluewang opened a new pull request, #6798: [INLONG-6772][Dashboard] Support Clickhouse node management
bluewang opened a new pull request, #6798: URL: https://github.com/apache/inlong/pull/6798 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes https://github.com/apache/inlong/issues/6772 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] dockerzhang merged pull request #641: [INLONG-627] [Doc] Translate http report guide (#627)
dockerzhang merged PR #641: URL: https://github.com/apache/inlong-website/pull/641 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] dockerzhang closed issue #627: [Feature][Doc] Translate http report guide
dockerzhang closed issue #627: [Feature][Doc] Translate http report guide URL: https://github.com/apache/inlong-website/issues/627 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong-website] branch master updated (b7937278aa -> 908c99b574)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong-website.git from b7937278aa [INLONG-625] [translation] Translate C++ SDK report guide (#625) (#639) add 908c99b574 [INLONG-627] [Doc] Translate http report guide (#627) (#641) No new revisions were added by this update. Summary of changes: docs/sdk/dataproxy-sdk/http.md | 38 +++--- 1 file changed, 19 insertions(+), 19 deletions(-)
[GitHub] [inlong-website] dockerzhang merged pull request #640: [INLONG-638][sort] Add support for elasticsearch-5
dockerzhang merged PR #640: URL: https://github.com/apache/inlong-website/pull/640 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong-website] branch master updated: [INLONG-638] [sort] Add support for elasticsearch-5 (#640)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/inlong-website.git The following commit(s) were added to refs/heads/master by this push: new 5cbabbacb3 [INLONG-638] [sort] Add support for elasticsearch-5 (#640) 5cbabbacb3 is described below commit 5cbabbacb34168b39e2b4c5074e06990c16f0bda Author: jiancheng Lv <635617...@qq.com> AuthorDate: Thu Dec 8 15:47:01 2022 +0800 [INLONG-638] [sort] Add support for elasticsearch-5 (#640) Co-authored-by: jianchenglv <6> --- docs/data_node/load_node/elasticsearch.md | 6 +++--- .../current/data_node/load_node/elasticsearch.md| 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/data_node/load_node/elasticsearch.md b/docs/data_node/load_node/elasticsearch.md index 2178a2060a..e49c326b5e 100644 --- a/docs/data_node/load_node/elasticsearch.md +++ b/docs/data_node/load_node/elasticsearch.md @@ -17,7 +17,7 @@ If no primary key is defined on the DDL, the Load Node can only operate in appen | Load Node | Version | |-|| -| [elasticsearch](./elasticsearch.md) | [Elasticsearch](https://www.elastic.co/): 6.x, 7.x | +| [elasticsearch](./elasticsearch.md) | [Elasticsearch](https://www.elastic.co/): 5.x, 6.x, 7.x | ### Dependencies @@ -92,7 +92,7 @@ TODO: It will be supported in the future. String Specify what connector to use, valid values are: - elasticsearch-6-inlong: connect to Elasticsearch 6.x cluster. + elasticsearch-6-inlong: connect to Elasticsearch 5.x and 6.x cluster. elasticsearch-7-inlong: connect to Elasticsearch 7.x and later versions cluster. @@ -114,7 +114,7 @@ TODO: It will be supported in the future. document-type - required in 6.x + required in 5.x and 6.x (none) String Elasticsearch document type. Not necessary anymore in elasticsearch-7. diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/load_node/elasticsearch.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/load_node/elasticsearch.md index c967f1d609..d10a6ceb85 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/load_node/elasticsearch.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/load_node/elasticsearch.md @@ -17,7 +17,7 @@ Elasticsearch Load 节点允许将数据写入到 Elasticsearch 引擎的索引 | Load 节点 | 版本 | |-|| -| [elasticsearch](./elasticsearch.md) | [Elasticsearch](https://www.elastic.co/): 6.x, 7.x | +| [elasticsearch](./elasticsearch.md) | [Elasticsearch](https://www.elastic.co/): 5.x, 6.x, 7.x | ### 依赖 @@ -92,7 +92,7 @@ TODO: 将在未来支持这个特性。 String 指定要使用的连接器，有效值为： - elasticsearch-6：连接到 Elasticsearch 6.x 的集群。 + elasticsearch-6：连接到 Elasticsearch 5.x and 6.x 的集群。 elasticsearch-7：连接到 Elasticsearch 7.x 及更高版本的集群。
[GitHub] [inlong-website] dockerzhang closed issue #638: [Feature][Doc] Elasticsearch-6-inlong Connector is compatible with elasticsearch versions 5 and 6
dockerzhang closed issue #638: [Feature][Doc] Elasticsearch-6-inlong Connector is compatible with elasticsearch versions 5 and 6 URL: https://github.com/apache/inlong-website/issues/638 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] dockerzhang commented on a diff in pull request #637: [INLONG-636][Doc] Postgresql cdc suggest setting replica identity to FULL
dockerzhang commented on code in PR #637: URL: https://github.com/apache/inlong-website/pull/637#discussion_r1043027819 ## docs/data_node/extract_node/postgresql-cdc.md: ## @@ -69,6 +69,28 @@ shared_preload_libraries = 'decoderbufs' wal_level = logical ``` +### replica identity + +`REPLICA IDENTITY` is a PostgreSQL-specific table-level setting that determines the amount of information that is available to the logical decoding plug-in for UPDATE and DELETE events. See [more](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-replica-identity). + +Please keep the `replica indentity` level of source tables to `FULL`. You can check and change this value by SQL: + +```aidl +-- show replica identity +SELECT CASE relreplident + WHEN 'd' THEN 'default' + WHEN 'n' THEN 'nothing' + WHEN 'f' THEN 'full' + WHEN 'i' THEN 'index' + END AS replica_identity +FROM pg_class +WHERE oid = 'mytablename'::regclass; + +-- change replica identity +ALTER TABLE mytablename REPLICA IDENTITY FULL; +``` + Review Comment: remove the extra line -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] yunqingmoswu opened a new pull request, #6799: [INLONG-6797][Sort] Supports dirty data side-output for filesystem sink
yunqingmoswu opened a new pull request, #6799: URL: https://github.com/apache/inlong/pull/6799 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title: [INLONG-6797][Sort] Supports dirty data side-output for filesystem sink *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #6797 ### Motivation Supports dirty data side-output for filesystem sink. In this part: 1. Load 'DirtySinkFactory' and create 'DirtySink' by the config 2. It needs to determine whether it is dirty data in the connector. 3. Side output dirty data by the 'DirtySink' dependents on the configured, the built-in side-out of dirty data has 'LogDirtySink'(#6618) and 'S3DirtySink'(#6655). ### Modifications 1. Create a dirty sink and inject it into filesystem sink 2. Add dirty handle for 'AbstractStreamingWriter' 3. Add a unit test for this ### Verifying this change *(Please pick either of the following options)* - [x] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] dockerzhang commented on pull request #637: [INLONG-636][Doc] Postgresql cdc suggest setting replica identity to FULL
dockerzhang commented on PR #637: URL: https://github.com/apache/inlong-website/pull/637#issuecomment-1342220953 @liaorui do we need a Chinese guide doc at the same time for this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #6793: [INLONG-6792][Sort] Supports dirty data side-output for hbase sink
EMsnap commented on code in PR #6793:
URL: https://github.com/apache/inlong/pull/6793#discussion_r1043031591
##
inlong-sort/sort-connectors/hbase/src/main/java/org/apache/inlong/sort/hbase/sink/HBaseSinkFunction.java:
##
@@ -133,11 +144,13 @@ public HBaseSinkFunction(
this.bufferFlushIntervalMillis = bufferFlushIntervalMillis;
this.inlongMetric = inlongMetric;
this.inlongAudit = inlongAudit;
+this.dirtyOptions = dirtyOptions;
+this.dirtySink = dirtySink;
}
@Override
public void open(Configuration parameters) throws Exception {
-LOG.info("start open ...");
+LOGGER.info("start open ...");
Review Comment:
the info log should be more specific
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
