svn commit: r58023 - /dev/inlong/1.4.0-RC0/ /release/inlong/1.4.0/
Author: dockerzhang Date: Tue Nov 15 08:13:24 2022 New Revision: 58023 Log: Release 1.4.0 Added: release/inlong/1.4.0/ - copied from r58022, dev/inlong/1.4.0-RC0/ Removed: dev/inlong/1.4.0-RC0/
svn commit: r58024 - /release/inlong/1.3.0/
Author: dockerzhang Date: Tue Nov 15 08:14:58 2022 New Revision: 58024 Log: Delete 1.3.0 Removed: release/inlong/1.3.0/
svn commit: r58025 - /dev/inlong/1.3.0-RC0/
Author: dockerzhang Date: Tue Nov 15 08:16:00 2022 New Revision: 58025 Log: Delete 1.3.0 Removed: dev/inlong/1.3.0-RC0/
[GitHub] [inlong] dockerzhang commented on issue #6436: [Release] Publish all 1.4.0 images to docker hub
dockerzhang commented on issue #6436: URL: https://github.com/apache/inlong/issues/6436#issuecomment-1314951153 the GitHub workflow has pushed all 1.4.0 images to [the docker hub](https://hub.docker.com/r/inlong/dashboard/tags). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] yunqingmoswu opened a new pull request, #6549: [INLONG-6548][Sort] Optimize metadata field naming for format of canal-json
yunqingmoswu opened a new pull request, #6549: URL: https://github.com/apache/inlong/pull/6549 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title: [INLONG-6548][Sort] Optimize metadata field naming for format of canal-json *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #6548 ### Motivation Optimize metadata field naming for format of canal-json ### Modifications Update the name of metadata field ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [x] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] pocozh opened a new pull request, #6551: [INLONG-6550][SDK] Fix decode heartbeat response error
pocozh opened a new pull request, #6551: URL: https://github.com/apache/inlong/pull/6551 ### Prepare a Pull Request *(Change the title refer to the following example)* - [INLONG-6550][SDK] Fix decode heartbeat response error *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #6550 ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve?* ### Modifications *Describe the modifications you've done.* ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] 01/02: [INLONG-6525][Manager] Update the status of stream and source after the InlongGroup is completed (#6527)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch branch-1.4
in repository https://gitbox.apache.org/repos/asf/inlong.git
commit 4fb36f3299e681052c9057a54d39e2522c901165
Author: fuweng11 <76141879+fuwen...@users.noreply.github.com>
AuthorDate: Mon Nov 14 15:14:29 2022 +0800
[INLONG-6525][Manager] Update the status of stream and source after the
InlongGroup is completed (#6527)
---
.../listener/group/InitGroupCompleteListener.java | 24 ++
1 file changed, 16 insertions(+), 8 deletions(-)
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/group/InitGroupCompleteListener.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/group/InitGroupCompleteListener.java
index ca5a375fb..a16562fd8 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/group/InitGroupCompleteListener.java
+++
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/group/InitGroupCompleteListener.java
@@ -18,26 +18,27 @@
package org.apache.inlong.manager.service.listener.group;
import lombok.extern.slf4j.Slf4j;
+import org.apache.inlong.manager.common.consts.InlongConstants;
import org.apache.inlong.manager.common.enums.GroupStatus;
import org.apache.inlong.manager.common.enums.ProcessEvent;
+import org.apache.inlong.manager.common.enums.SourceStatus;
+import org.apache.inlong.manager.common.enums.StreamStatus;
import org.apache.inlong.manager.common.exceptions.WorkflowListenerException;
import org.apache.inlong.manager.dao.entity.InlongGroupEntity;
import org.apache.inlong.manager.dao.mapper.InlongGroupEntityMapper;
import org.apache.inlong.manager.pojo.group.InlongGroupInfo;
import org.apache.inlong.manager.pojo.group.InlongGroupRequest;
import org.apache.inlong.manager.pojo.group.InlongGroupUtils;
-import org.apache.inlong.manager.pojo.stream.InlongStreamInfo;
import
org.apache.inlong.manager.pojo.workflow.form.process.GroupResourceProcessForm;
import org.apache.inlong.manager.service.group.InlongGroupService;
-import org.apache.inlong.manager.service.stream.InlongStreamProcessService;
+import org.apache.inlong.manager.service.source.StreamSourceService;
+import org.apache.inlong.manager.service.stream.InlongStreamService;
import org.apache.inlong.manager.workflow.WorkflowContext;
import org.apache.inlong.manager.workflow.event.ListenerResult;
import org.apache.inlong.manager.workflow.event.process.ProcessEventListener;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
-import java.util.List;
-
/**
* The listener of InlongGroup when created resources successfully.
*/
@@ -50,7 +51,9 @@ public class InitGroupCompleteListener implements
ProcessEventListener {
@Autowired
private InlongGroupEntityMapper groupMapper;
@Autowired
-private InlongStreamProcessService streamProcessService;
+private InlongStreamService streamService;
+@Autowired
+private StreamSourceService sourceService;
@Override
public ProcessEvent event() {
@@ -81,9 +84,14 @@ public class InitGroupCompleteListener implements
ProcessEventListener {
updateGroupRequest.setVersion(existGroup.getVersion());
groupService.update(updateGroupRequest, operator);
-List streamList = form.getStreamInfos();
-for (InlongStreamInfo streamInfo : streamList) {
-streamProcessService.startProcess(groupId,
streamInfo.getInlongStreamId(), operator, false);
+// update status of other related configs
+if
(InlongConstants.DISABLE_CREATE_RESOURCE.equals(groupInfo.getEnableCreateResource()))
{
+streamService.updateStatus(groupId, null,
StreamStatus.CONFIG_SUCCESSFUL.getCode(), operator);
+if
(InlongConstants.LIGHTWEIGHT_MODE.equals(groupInfo.getLightweight())) {
+sourceService.updateStatus(groupId, null,
SourceStatus.SOURCE_NORMAL.getCode(), operator);
+} else {
+sourceService.updateStatus(groupId, null,
SourceStatus.TO_BE_ISSUED_ADD.getCode(), operator);
+}
}
log.info("success to execute InitGroupCompleteListener for
groupId={}", groupId);
[inlong] branch branch-1.4 updated (b22bbab4b -> 0db3d054f)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch branch-1.4 in repository https://gitbox.apache.org/repos/asf/inlong.git from b22bbab4b [INLONG-6529] Fix microtime time zone error in mysql connector (#6530) new 4fb36f329 [INLONG-6525][Manager] Update the status of stream and source after the InlongGroup is completed (#6527) new 0db3d054f [INLONG-6533][Manager] Fix the problem of not distributing the task according to the cluster name (#6534) The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../dao/mapper/StreamSourceEntityMapper.java | 6 ++-- .../resources/mappers/StreamSourceEntityMapper.xml | 15 + .../service/core/impl/AgentServiceImpl.java| 36 -- .../listener/group/InitGroupCompleteListener.java | 24 ++- 4 files changed, 46 insertions(+), 35 deletions(-)
[inlong] 02/02: [INLONG-6533][Manager] Fix the problem of not distributing the task according to the cluster name (#6534)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch branch-1.4
in repository https://gitbox.apache.org/repos/asf/inlong.git
commit 0db3d054f79277fcdd089826d9791119a4d31cd9
Author: fuweng11 <76141879+fuwen...@users.noreply.github.com>
AuthorDate: Tue Nov 15 12:38:26 2022 +0800
[INLONG-6533][Manager] Fix the problem of not distributing the task
according to the cluster name (#6534)
* Fix the problem of not distributing the task according to the cluster name
* Extract a constant to improve the code
Co-authored-by: healchow
---
.../dao/mapper/StreamSourceEntityMapper.java | 6 ++--
.../resources/mappers/StreamSourceEntityMapper.xml | 15 +
.../service/core/impl/AgentServiceImpl.java| 36 --
3 files changed, 30 insertions(+), 27 deletions(-)
diff --git
a/inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/mapper/StreamSourceEntityMapper.java
b/inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/mapper/StreamSourceEntityMapper.java
index aee466d88..caa80bcc9 100644
---
a/inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/mapper/StreamSourceEntityMapper.java
+++
b/inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/mapper/StreamSourceEntityMapper.java
@@ -81,12 +81,12 @@ public interface StreamSourceEntityMapper {
@Param("clusterName") String clusterName);
/**
- * Query the sources with status 20x by the given agent IP and agent UUID.
+ * Query the sources by the given status and Agent cluster info.
*
* @apiNote Sources with is_deleted > 0 should also be returned to agents
to clear their local tasks.
*/
-List selectByStatusAndIp(@Param("statusList")
List statusList,
-@Param("agentIp") String agentIp, @Param("uuid") String uuid);
+List selectByStatusAndCluster(@Param("statusList")
List statusList,
+@Param("clusterName") String clusterName, @Param("agentIp") String
agentIp, @Param("uuid") String uuid);
/**
* Select all sources by groupIds
diff --git
a/inlong-manager/manager-dao/src/main/resources/mappers/StreamSourceEntityMapper.xml
b/inlong-manager/manager-dao/src/main/resources/mappers/StreamSourceEntityMapper.xml
index e1e7f56a8..97ef663fe 100644
---
a/inlong-manager/manager-dao/src/main/resources/mappers/StreamSourceEntityMapper.xml
+++
b/inlong-manager/manager-dao/src/main/resources/mappers/StreamSourceEntityMapper.xml
@@ -224,18 +224,17 @@
and (agent_ip = #{agentIp, jdbcType=VARCHAR} or
inlong_cluster_name = #{clusterName, jdbcType=VARCHAR})
-
+
select
from stream_source
-agent_ip = #{agentIp, jdbcType=VARCHAR}
-
-and status in
-
-#{item}
-
-
+inlong_cluster_name = #{clusterName, jdbcType=VARCHAR}
+and agent_ip = #{agentIp, jdbcType=VARCHAR}
+and `status` in
+
+#{item}
+
and uuid = #{uuid, jdbcType=VARCHAR}
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/AgentServiceImpl.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/AgentServiceImpl.java
index d091bdf7e..29f0dcbff 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/AgentServiceImpl.java
+++
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/AgentServiceImpl.java
@@ -71,6 +71,18 @@ public class AgentServiceImpl implements AgentService {
private static final int MODULUS_100 = 100;
private static final int TASK_FETCH_SIZE = 2;
+/**
+ * Need issued status list, not included status with TO_BE_ISSUED_ADD and
TO_BE_ISSUED_ACTIVE
+ */
+private static final List NEED_ISSUED_STATUS = Arrays.asList(
+SourceStatus.TO_BE_ISSUED_DELETE.getCode(),
+SourceStatus.TO_BE_ISSUED_RETRY.getCode(),
+SourceStatus.TO_BE_ISSUED_BACKTRACK.getCode(),
+SourceStatus.TO_BE_ISSUED_FROZEN.getCode(),
+SourceStatus.TO_BE_ISSUED_CHECK.getCode(),
+SourceStatus.TO_BE_ISSUED_REDO_METRIC.getCode(),
+SourceStatus.TO_BE_ISSUED_MAKEUP.getCode());
+
@Autowired
private StreamSourceEntityMapper sourceMapper;
@Autowired
@@ -153,20 +165,17 @@ public class AgentServiceImpl implements AgentService {
@Transactional(rollbackFor = Throwable.class, isolation =
Isolation.READ_COMMITTED,
propagation = Propagation.REQUIRES_NEW)
public TaskResult getTaskResult(TaskRequest request) {
-if (request == null || StringUtils.isBlank(request.getAgentIp())) {
+
[GitHub] [inlong] healchow commented on a diff in pull request #6546: [INLONG-6542][Dashboard] Optimize stream execution workflow and execution log
healchow commented on code in PR #6546:
URL: https://github.com/apache/inlong/pull/6546#discussion_r1022665585
##
inlong-dashboard/src/pages/GroupDetail/DataStream/index.tsx:
##
@@ -178,11 +184,14 @@ const Comp = ({ inlongGroupId, readonly, mqType }: Props,
ref) => {
onDelete(record)}>
{t('basic.Delete')}
-{record?.status && (record?.status === 120 || record?.status ===
130) && (
- onWorkflow(record)}>
-{t('meta.Stream.ExecuteWorkflow')}
-
-)}
+{record?.status &&
+ ((groupStatus === 130 && record?.status === 100) ||
+groupStatus === 120 ||
+groupStatus === 130) && (
Review Comment:
Could you please move those two lines into one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] gosonzhang merged pull request #6551: [INLONG-6550][SDK] Fix the decode heartbeat response error
gosonzhang merged PR #6551: URL: https://github.com/apache/inlong/pull/6551 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-6550][SDK] Fix the decode heartbeat response error (#6551)
This is an automated email from the ASF dual-hosted git repository.
gosonzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new c3807e6fd [INLONG-6550][SDK] Fix the decode heartbeat response error
(#6551)
c3807e6fd is described below
commit c3807e6fdcf9245d59f4ed5451cb09f16593d2cf
Author: xueyingzhang <86780714+poc...@users.noreply.github.com>
AuthorDate: Tue Nov 15 19:24:19 2022 +0800
[INLONG-6550][SDK] Fix the decode heartbeat response error (#6551)
---
.../inlong/sdk/dataproxy/codec/EncodeObject.java | 6 +-
.../inlong/sdk/dataproxy/codec/ProtocolDecoder.java| 18 ++
2 files changed, 15 insertions(+), 9 deletions(-)
diff --git
a/inlong-sdk/dataproxy-sdk/src/main/java/org/apache/inlong/sdk/dataproxy/codec/EncodeObject.java
b/inlong-sdk/dataproxy-sdk/src/main/java/org/apache/inlong/sdk/dataproxy/codec/EncodeObject.java
index 8634621cf..fd8ab8c8c 100644
---
a/inlong-sdk/dataproxy-sdk/src/main/java/org/apache/inlong/sdk/dataproxy/codec/EncodeObject.java
+++
b/inlong-sdk/dataproxy-sdk/src/main/java/org/apache/inlong/sdk/dataproxy/codec/EncodeObject.java
@@ -67,11 +67,7 @@ public class EncodeObject {
private String errMsg;
private String dpIp;
-/* Used by de_serialization. msgtype=8*/
-public EncodeObject() {
-}
-
-/* Used by de_serialization. msgtype=7*/
+/* Used by de_serialization. msgtype=7/8 */
public EncodeObject(String attributes) {
handleAttr(attributes);
}
diff --git
a/inlong-sdk/dataproxy-sdk/src/main/java/org/apache/inlong/sdk/dataproxy/codec/ProtocolDecoder.java
b/inlong-sdk/dataproxy-sdk/src/main/java/org/apache/inlong/sdk/dataproxy/codec/ProtocolDecoder.java
index 7bf9bd883..af829f8ef 100644
---
a/inlong-sdk/dataproxy-sdk/src/main/java/org/apache/inlong/sdk/dataproxy/codec/ProtocolDecoder.java
+++
b/inlong-sdk/dataproxy-sdk/src/main/java/org/apache/inlong/sdk/dataproxy/codec/ProtocolDecoder.java
@@ -93,11 +93,21 @@ public class ProtocolDecoder extends
MessageToMessageDecoder {
out.add(object);
} else if (msgType == 8) {
-int attrlen = buffer.getShort(4 + 1 + 4 + 1 + 4 + 2);
-buffer.skipBytes(13 + attrlen + 2);
-EncodeObject object = new EncodeObject();
+// dataTime(4) + body_ver(1) + body_len(4) + body + attr_len(2) +
attr + magic(2)
+buffer.skipBytes(4 + 1 + 4); // skip datatime, body_ver and
body_len
+final short load = buffer.readShort(); // read from body
+int attrLen = buffer.readShort();
+byte[] attrBytes = null;
+if (attrLen > 0) {
+attrBytes = new byte[attrLen];
+buffer.readBytes(attrBytes);
+}
+buffer.skipBytes(2); // skip magic
+
+String attrs = (attrBytes == null ? "" : new String(attrBytes,
StandardCharsets.UTF_8));
+EncodeObject object = new EncodeObject(attrs);
object.setMsgtype(8);
-object.setLoad(buffer.getShort(4 + 1 + 4 + 1 + 4));
+object.setLoad(load);
out.add(object);
}
}
[GitHub] [inlong] gosonzhang merged pull request #6520: [INLONG-6515][Manager] NPE is thrown in the cache configuration of DataProxy
gosonzhang merged PR #6520: URL: https://github.com/apache/inlong/pull/6520 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-6515][Manager] NPE is thrown in the cache configuration of DataProxy (#6520)
This is an automated email from the ASF dual-hosted git repository.
gosonzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new ae984c1c8 [INLONG-6515][Manager] NPE is thrown in the cache
configuration of DataProxy (#6520)
ae984c1c8 is described below
commit ae984c1c8b57580d3f885a8644eead6a1aec7839
Author: 卢春亮 <946240...@qq.com>
AuthorDate: Tue Nov 15 19:31:46 2022 +0800
[INLONG-6515][Manager] NPE is thrown in the cache configuration of
DataProxy (#6520)
Co-authored-by: healchow
---
.../repository/DataProxyConfigRepository.java | 64 --
.../repository/DataProxyConfigRepositoryTest.java | 4 +-
2 files changed, 38 insertions(+), 30 deletions(-)
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/repository/DataProxyConfigRepository.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/repository/DataProxyConfigRepository.java
index 3e93612cb..adaa7c1f3 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/repository/DataProxyConfigRepository.java
+++
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/repository/DataProxyConfigRepository.java
@@ -21,7 +21,6 @@ import com.google.common.base.Splitter;
import com.google.gson.Gson;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
-
import org.apache.commons.beanutils.BeanUtils;
import org.apache.commons.codec.digest.DigestUtils;
import org.apache.commons.lang3.StringUtils;
@@ -61,6 +60,7 @@ import org.springframework.context.annotation.Lazy;
import org.springframework.stereotype.Repository;
import org.springframework.transaction.annotation.Transactional;
+import javax.annotation.PostConstruct;
import java.lang.reflect.InvocationTargetException;
import java.util.ArrayList;
import java.util.Date;
@@ -68,14 +68,11 @@ import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
-import java.util.Objects;
import java.util.Optional;
import java.util.Timer;
import java.util.TimerTask;
import java.util.concurrent.ConcurrentHashMap;
-import javax.annotation.PostConstruct;
-
/**
* DataProxyConfigRepository
*/
@@ -90,14 +87,14 @@ public class DataProxyConfigRepository implements
IRepository {
public static final String KEY_BACKUP_TOPIC = "backup_topic";
public static final String KEY_SORT_TASK_NAME = "defaultSortTaskName";
public static final String KEY_DATA_NODE_NAME = "defaultDataNodeName";
-public static final String KEY_SORT_CONSUEMER_GROUP =
"defaultSortConsumerGroup";
+public static final String KEY_SORT_CONSUMER_GROUP =
"defaultSortConsumerGroup";
public static final String KEY_SINK_NAME = "defaultSinkName";
public static final Splitter.MapSplitter MAP_SPLITTER =
Splitter.on(SEPARATOR).trimResults()
.withKeyValueSeparator(KEY_VALUE_SEPARATOR);
public static final String CACHE_CLUSTER_PRODUCER_TAG = "producer";
public static final String CACHE_CLUSTER_CONSUMER_TAG = "consumer";
-private static final Gson gson = new Gson();
+private static final Gson GSON = new Gson();
// key: proxyClusterName, value: jsonString
private Map proxyConfigJson = new ConcurrentHashMap<>();
@@ -214,7 +211,7 @@ public class DataProxyConfigRepository implements
IRepository {
if (proxyClusterMap.size() == 0) {
return;
}
-// reoload cache cluster
+// reload cache cluster
this.reloadCacheCluster(proxyClusterMap);
// reload inlong group id and inlong stream id
this.reloadInlongId(proxyClusterMap);
@@ -284,7 +281,7 @@ public class DataProxyConfigRepository implements
IRepository {
CacheClusterObject obj = new CacheClusterObject();
obj.setName(cacheCluster.getClusterName());
obj.setZone(cacheCluster.getExtTag());
-
obj.setParams(fromJson(cacheCluster.getExtParams()));
+
obj.setParams(fromJsonToMap(cacheCluster.getExtParams()));
cacheClusters.add(obj);
}
}
@@ -294,16 +291,15 @@ public class DataProxyConfigRepository implements
IRepository {
}
/**
- * fromJson
+ * Parse Json string to Java Map
*/
-private Map fromJson(String jsonString) {
+private Map fromJsonToMap(String jsonString) {
Map mapObj = new HashMap<>();
+if (StringUtils.isBlank(jsonString)) {
+return mapObj;
+}
try {
-JsonObject obj = gson.fromJson(jsonString, JsonObject.class);
-// when jsonString is null or empty str, The return value
parameter will become null
-
[GitHub] [inlong] gosonzhang opened a new pull request, #6553: [INLONG-6531][Manager] Add load value in the response of getIpList method (addendum)
gosonzhang opened a new pull request, #6553: URL: https://github.com/apache/inlong/pull/6553 - Fixes #6531 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] fuweng11 commented on a diff in pull request #6521: [INLONG-6496][Manager] Support Elasticsearch DataNode
fuweng11 commented on code in PR #6521:
URL: https://github.com/apache/inlong/pull/6521#discussion_r1022697975
##
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/node/es/ElasticsearchDataNodeDTO.java:
##
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.pojo.node.es;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import lombok.AllArgsConstructor;
+import lombok.Builder;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+import org.apache.inlong.manager.common.enums.ErrorCodeEnum;
+import org.apache.inlong.manager.common.exceptions.BusinessException;
+import org.apache.inlong.manager.common.util.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.validation.constraints.NotNull;
+
+/**
+ * Elasticsearch data node info
+ */
+@Data
+@Builder
+@NoArgsConstructor
+@AllArgsConstructor
+@ApiModel("Elasticsearch data node info")
+public class ElasticsearchDataNodeDTO {
+
+private static final Logger LOGGER =
LoggerFactory.getLogger(ElasticsearchDataNodeDTO.class);
+
+@ApiModelProperty("Bulk action, default is 4000")
+private Integer bulkAction;
+
+@ApiModelProperty("Bulk size in MB, default is 4000")
+private Integer bulkSizeMb;
+
+@ApiModelProperty("Flush interval, default is 60")
+private Integer flushInterval;
+
+@ApiModelProperty("Concurrent requests, default is 60")
+private Integer concurrentRequests;
+
+@ApiModelProperty("Max connection, default is 10")
+private Integer maxConnect;
+
+@ApiModelProperty("Max keyword length, default is 32767")
+private Integer keywordMaxLength;
+
+@ApiModelProperty("Is use index id, default is false")
+private Boolean isUseIndexId;
+
+@ApiModelProperty("Max threads, default is 2")
+private Integer maxThreads;
+
+@ApiModelProperty("ES username")
+private String username;
+
+@ApiModelProperty("ES password")
+private String password;
+
+@ApiModelProperty("Host of ES server, ip:port format")
+private String httpHosts;
+
+@ApiModelProperty("audit set name")
+private String auditSetName;
+
+/**
+ * Get the dto instance from the request
+ */
+public static ElasticsearchDataNodeDTO
getFromRequest(ElasticsearchDataNodeRequest request) {
+return ElasticsearchDataNodeDTO.builder()
Review Comment:
Recommended use `CommonBeanUtils.copyProperties(request,
ElasticsearchDataNodeDTO::new, true);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] fuweng11 commented on a diff in pull request #6473: [INLONG-6462][Manager] Support CLS datanode
fuweng11 commented on code in PR #6473:
URL: https://github.com/apache/inlong/pull/6473#discussion_r1022703144
##
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/node/cls/ClsDataNodeDTO.java:
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.pojo.node.cls;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import lombok.AllArgsConstructor;
+import lombok.Builder;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+import org.apache.inlong.manager.common.enums.ErrorCodeEnum;
+import org.apache.inlong.manager.common.exceptions.BusinessException;
+import org.apache.inlong.manager.common.util.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.validation.constraints.NotNull;
+
+/**
+ * Cls data node info
+ */
+@Data
+@Builder
+@NoArgsConstructor
+@AllArgsConstructor
+@ApiModel("Cls data node info")
+public class ClsDataNodeDTO {
+private static final Logger LOGGER =
LoggerFactory.getLogger(ClsDataNodeDTO.class);
+
+@ApiModelProperty("Max send thread count, default is 50")
+@Builder.Default
+private Integer maxSendThreadCount = 50;
+
+@ApiModelProperty("Max block time, default is 0 seconds")
+@Builder.Default
+private Integer maxBlockSec = 0;
+
+@ApiModelProperty("Max message batch count, default is 4096")
+@Builder.Default
+private Integer maxBatchCount = 4096;
+
+@ApiModelProperty("Linger, default is 2000ms")
+@Builder.Default
+private Integer lingerMs = 2000;
+
+@ApiModelProperty("Retry times, default is 10")
+@Builder.Default
+private Integer retries = 10;
+
+@ApiModelProperty("Max reserve attempts, default is 11")
+@Builder.Default
+private Integer maxReservedAttempts = 11;
+
+@ApiModelProperty("Base retry backoff time, default is 100ms")
+@Builder.Default
+private Integer baseRetryBackoffMs = 100;
+
+@ApiModelProperty("Max retry backoff time, default is 50ms")
+@Builder.Default
+private Integer maxRetryBackoffMs = 50;
+
+/**
+ * Get the dto instance from the request
+ */
+public static ClsDataNodeDTO getFromRequest(ClsDataNodeRequest request) {
+return ClsDataNodeDTO.builder()
+.baseRetryBackoffMs(request.getBaseRetryBackoffMs())
Review Comment:
Recommended use CommonBeanUtils.copyProperties(request, ClsDataNodeDTO::new,
true);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] e-mhui opened a new pull request, #6554: [INLONG-6552][Sort] Fix MicroTimestamp error in mysql connector
e-mhui opened a new pull request, #6554: URL: https://github.com/apache/inlong/pull/6554 ### Fix MicroTimestamp error in mysql connector - Fixes #6552 ### Motivation The debezium read oracle's timestamp type as a number of microseconds, we should convert it to a string format of date-time. ### Modifications If field schema is timestamp type, we need convert to a string format of date-time from a number of microseconds. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] healchow merged pull request #6547: [INLONG-6544][Manager] Should update the source status after the inlong stream is completed
healchow merged PR #6547: URL: https://github.com/apache/inlong/pull/6547 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-6544][Manager] Should update the source status after the inlong stream is completed (#6547)
This is an automated email from the ASF dual-hosted git repository.
healchow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new e64cbd96c [INLONG-6544][Manager] Should update the source status after
the inlong stream is completed (#6547)
e64cbd96c is described below
commit e64cbd96cf5a8771c3f617531230e26f33508e10
Author: healchow
AuthorDate: Tue Nov 15 20:20:48 2022 +0800
[INLONG-6544][Manager] Should update the source status after the inlong
stream is completed (#6547)
---
.../listener/group/InitGroupCompleteListener.java | 46 ++--
.../stream/InitStreamCompleteListener.java | 21 +++---
.../resource/sort/DefaultSortConfigOperator.java | 21 --
.../service/stream/InlongStreamService.java| 11 ++-
.../service/stream/InlongStreamServiceImpl.java| 87 +-
5 files changed, 67 insertions(+), 119 deletions(-)
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/group/InitGroupCompleteListener.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/group/InitGroupCompleteListener.java
index a16562fd8..bd848f996 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/group/InitGroupCompleteListener.java
+++
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/group/InitGroupCompleteListener.java
@@ -72,30 +72,34 @@ public class InitGroupCompleteListener implements
ProcessEventListener {
String groupId = form.getInlongGroupId();
log.info("begin to execute InitGroupCompleteListener for groupId={}",
groupId);
-// update inlong group status and other info
-InlongGroupInfo groupInfo = form.getGroupInfo();
-String operator = context.getOperator();
-groupService.updateStatus(groupId,
GroupStatus.CONFIG_SUCCESSFUL.getCode(), operator);
-if (InlongGroupUtils.isBatchTask(form.getGroupInfo())) {
-groupService.updateStatus(groupId, GroupStatus.FINISH.getCode(),
operator);
-}
-InlongGroupEntity existGroup = groupMapper.selectByGroupId(groupId);
-InlongGroupRequest updateGroupRequest = groupInfo.genRequest();
-updateGroupRequest.setVersion(existGroup.getVersion());
-groupService.update(updateGroupRequest, operator);
+try {
+// update inlong group status and other info
+InlongGroupInfo groupInfo = form.getGroupInfo();
+String operator = context.getOperator();
+Integer nextStatus =
InlongGroupUtils.isBatchTask(form.getGroupInfo())
+? GroupStatus.FINISH.getCode() :
GroupStatus.CONFIG_SUCCESSFUL.getCode();
+groupService.updateStatus(groupId, nextStatus, operator);
+
+InlongGroupEntity existGroup =
groupMapper.selectByGroupId(groupId);
+InlongGroupRequest updateGroupRequest = groupInfo.genRequest();
+updateGroupRequest.setVersion(existGroup.getVersion());
+groupService.update(updateGroupRequest, operator);
-// update status of other related configs
-if
(InlongConstants.DISABLE_CREATE_RESOURCE.equals(groupInfo.getEnableCreateResource()))
{
-streamService.updateStatus(groupId, null,
StreamStatus.CONFIG_SUCCESSFUL.getCode(), operator);
-if
(InlongConstants.LIGHTWEIGHT_MODE.equals(groupInfo.getLightweight())) {
-sourceService.updateStatus(groupId, null,
SourceStatus.SOURCE_NORMAL.getCode(), operator);
-} else {
-sourceService.updateStatus(groupId, null,
SourceStatus.TO_BE_ISSUED_ADD.getCode(), operator);
+// update status of other related configs
+if
(InlongConstants.DISABLE_CREATE_RESOURCE.equals(groupInfo.getEnableCreateResource()))
{
+streamService.updateStatus(groupId, null,
StreamStatus.CONFIG_SUCCESSFUL.getCode(), operator);
+if
(InlongConstants.LIGHTWEIGHT_MODE.equals(groupInfo.getLightweight())) {
+sourceService.updateStatus(groupId, null,
SourceStatus.SOURCE_NORMAL.getCode(), operator);
+} else {
+sourceService.updateStatus(groupId, null,
SourceStatus.TO_BE_ISSUED_ADD.getCode(), operator);
+}
}
-}
-log.info("success to execute InitGroupCompleteListener for
groupId={}", groupId);
-return ListenerResult.success();
+log.info("success to execute InitGroupCompleteListener for
groupId={}", groupId);
+return ListenerResult.success();
+} catch (Exception e) {
+throw new WorkflowListenerException(e);
+}
}
}
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/
[GitHub] [inlong] gosonzhang merged pull request #6553: [INLONG-6531][Manager] Add load value in the response of getIpList method (addendum)
gosonzhang merged PR #6553: URL: https://github.com/apache/inlong/pull/6553 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-6531][Manager] Add load value in the response of getIpList method (addendum) (#6553)
This is an automated email from the ASF dual-hosted git repository.
gosonzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 8ff76f498 [INLONG-6531][Manager] Add load value in the response of
getIpList method (addendum) (#6553)
8ff76f498 is described below
commit 8ff76f49896bff9f9e018e7528491b2cace6af81
Author: Goson Zhang <4675...@qq.com>
AuthorDate: Tue Nov 15 20:30:56 2022 +0800
[INLONG-6531][Manager] Add load value in the response of getIpList method
(addendum) (#6553)
---
.../apache/inlong/manager/service/heartbeat/HeartbeatManager.java | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/heartbeat/HeartbeatManager.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/heartbeat/HeartbeatManager.java
index fa5e17a7b..8401c76a3 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/heartbeat/HeartbeatManager.java
+++
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/heartbeat/HeartbeatManager.java
@@ -129,7 +129,7 @@ public class HeartbeatManager implements
AbstractHeartbeatManager {
if (clusterNode == null) {
handlerNum += insertClusterNode(clusterInfo, heartbeatMsg,
clusterInfo.getCreator());
} else {
-handlerNum += updateClusterNode(clusterNode);
+handlerNum += updateClusterNode(clusterNode, heartbeatMsg);
}
}
}
@@ -192,7 +192,6 @@ public class HeartbeatManager implements
AbstractHeartbeatManager {
nodeRequest.setIp(heartbeat.getIp());
nodeRequest.setPort(Integer.valueOf(heartbeat.getPort()));
nodeRequest.setProtocolType(heartbeat.getProtocolType());
-nodeRequest.setNodeLoad(heartbeat.getLoad());
return clusterNodeMapper.selectByUniqueKey(nodeRequest);
}
@@ -211,8 +210,9 @@ public class HeartbeatManager implements
AbstractHeartbeatManager {
return clusterNodeMapper.insertOnDuplicateKeyUpdate(clusterNode);
}
-private int updateClusterNode(InlongClusterNodeEntity clusterNode) {
+private int updateClusterNode(InlongClusterNodeEntity clusterNode,
HeartbeatMsg heartbeat) {
clusterNode.setStatus(ClusterStatus.NORMAL.getStatus());
+clusterNode.setNodeLoad(heartbeat.getLoad());
return clusterNodeMapper.updateById(clusterNode);
}
[GitHub] [inlong-website] dockerzhang opened a new issue, #600: [Improve] Update the concept and introduction guide
dockerzhang opened a new issue, #600: URL: https://github.com/apache/inlong-website/issues/600 ### Description Update the concept and introduction guide. ### Are you willing to submit PR? - [X] Yes, I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong-website] 01/01: [INLONG-600][Doc] Update the concept and introduction guide
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a commit to branch fix-600 in repository https://gitbox.apache.org/repos/asf/inlong-website.git commit aa1f59dcb3cc1fb979ea4bcf123f79f553faf710 Author: dockerzhang AuthorDate: Tue Nov 15 22:15:15 2022 +0800 [INLONG-600][Doc] Update the concept and introduction guide --- docs/design_and_concept/basic_concept.md | 28 -- docs/introduction.md | 6 ++--- .../current/design_and_concept/basic_concept.md| 28 -- .../current/introduction.md| 12 +- 4 files changed, 39 insertions(+), 35 deletions(-) diff --git a/docs/design_and_concept/basic_concept.md b/docs/design_and_concept/basic_concept.md index 823e96dee4..cf41f67583 100644 --- a/docs/design_and_concept/basic_concept.md +++ b/docs/design_and_concept/basic_concept.md @@ -3,16 +3,18 @@ title: Basic Concept sidebar_position: 1 --- -| Name | Description | Other | -|--||| -| Standard Architecture| Contains all InLong components such as InLong Agent/Manager/MQ/Sort/Dashboard | Suitable for massive data and large-scale production environments | -| Lightweight Architecture | Contains only one component of InLong Sort, which can be used with Manager/Dashboard | The lightweight architecture is simple and flexible, suitable for small-scale data | -| Group| Data Streams Group, it contains multiple data streams, and one Group represents one data ingestion.| Group has attributes such as ID and Name. | -| Stream | Data Stream, a stream has a specific flow direction. | Stream has attributes such as ID, Name, and data fields. | -| Node | Data Node, including `Extract Node` and `Load Node`, stands for the data source and sink types separately.| | -| InLongMsg| InLong data format, if you consume message directly from the message queue, you need to perform `InLongMsg` parsing first. | | -| Agent| The standard architecture uses Agent for data collection, and Agent represents different types of collection capabilities. | It contains File Agent, SQL Agent, Binlog Agent, etc. | -| DataProxy| Forward received data to different message queues. | Supports data transmission blocking, placing retransmission. | -| Sort | Data stream sorting. | Sort-flink based on Flink, sort-standalone for local sorting. | -| TubeMQ | InLong's self-developed message queuing service | It can also be called Tube, with low-cost, high-performance features. | -| Pulsar | [Apache Pulsar](https://pulsar.apache.org/), a high-performance, high-consistency message queue service| | \ No newline at end of file +| Name| Description | Other | +|-||| +| Standard| Standard Architecture, contains all InLong components such as InLong Agent, Manager, MQ, Sort, Dashboard | Suitable for massive data and large-scale production environments | +| Lightweight | Lightweight Architecture, contains only one component of InLong Sort, which also can be used with Manager, D
[inlong-website] branch fix-600 created (now aa1f59dcb3)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch fix-600 in repository https://gitbox.apache.org/repos/asf/inlong-website.git at aa1f59dcb3 [INLONG-600][Doc] Update the concept and introduction guide This branch includes the following new commits: new aa1f59dcb3 [INLONG-600][Doc] Update the concept and introduction guide The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[GitHub] [inlong-website] dockerzhang opened a new pull request, #601: [INLONG-600][Doc] Update the concept and introduction guide
dockerzhang opened a new pull request, #601: URL: https://github.com/apache/inlong-website/pull/601 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #600 ### Motivation Make some expressions be more precise. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch fix-6555 created (now a17152fc8)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch fix-6555 in repository https://gitbox.apache.org/repos/asf/inlong.git at a17152fc8 [INLONG-6555][CI] Update for set-output command is deprecated This branch includes the following new commits: new a17152fc8 [INLONG-6555][CI] Update for set-output command is deprecated The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[inlong] 01/01: [INLONG-6555][CI] Update for set-output command is deprecated
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch fix-6555
in repository https://gitbox.apache.org/repos/asf/inlong.git
commit a17152fc8cb70136d2a2a858421828318126d9bb
Author: dockerzhang
AuthorDate: Tue Nov 15 23:15:56 2022 +0800
[INLONG-6555][CI] Update for set-output command is deprecated
---
.github/workflows/ci_docker.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/ci_docker.yml b/.github/workflows/ci_docker.yml
index 98af35529..18e253581 100644
--- a/.github/workflows/ci_docker.yml
+++ b/.github/workflows/ci_docker.yml
@@ -94,9 +94,9 @@ jobs:
&& github.repository_owner == 'apache'
run: |
if [[ ${{ github.ref_name }} == ${{
github.event.repository.default_branch }} ]]; then
-echo "::set-output name=match_master::true"
+echo "name=match_master::true" >> $GITHUB_OUTPUT
elif [[ ${{ github.ref_name }} =~ ^branch-[0-9]+\.[0-9]+$ ]]; then
-echo "::set-output name=match_release::true"
+echo "name=match_release::true" >> $GITHUB_OUTPUT
fi
# Publish x86 Docker images when the changes are being pushed to the
master branch or a release branch like 'branch-1.4'.
[GitHub] [inlong] dockerzhang opened a new pull request, #6556: [INLONG-6555][CI] Update for set-output command is deprecated
dockerzhang opened a new pull request, #6556: URL: https://github.com/apache/inlong/pull/6556 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #6555 ### Motivation https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ https://user-images.githubusercontent.com/18047329/201955916-968e9d8f-e931-4505-b6c3-afd04ba3b190.png";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] healchow merged pull request #6556: [INLONG-6555][CI] Update for set-output command is deprecated
healchow merged PR #6556: URL: https://github.com/apache/inlong/pull/6556 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated (8ff76f498 -> cceca481c)
This is an automated email from the ASF dual-hosted git repository. healchow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong.git from 8ff76f498 [INLONG-6531][Manager] Add load value in the response of getIpList method (addendum) (#6553) add cceca481c [INLONG-6555][CI] Update for set-output command is deprecated (#6556) No new revisions were added by this update. Summary of changes: .github/workflows/ci_docker.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[inlong] branch master updated (cceca481c -> 531464d0f)
This is an automated email from the ASF dual-hosted git repository. zirui pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong.git from cceca481c [INLONG-6555][CI] Update for set-output command is deprecated (#6556) add 531464d0f [INLONG-6552][Sort] Fix MicroTimestamp error in oracle connector (#6554) No new revisions were added by this update. Summary of changes: .../cdc/oracle/debezium/table/RowDataDebeziumDeserializeSchema.java| 3 +++ 1 file changed, 3 insertions(+)
[GitHub] [inlong] EMsnap merged pull request #6554: [INLONG-6552][Sort] Fix MicroTimestamp error in oracle connector
EMsnap merged PR #6554: URL: https://github.com/apache/inlong/pull/6554 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] healchow merged pull request #601: [INLONG-600][Doc] Update the concept and introduction guide
healchow merged PR #601: URL: https://github.com/apache/inlong-website/pull/601 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] healchow closed issue #600: [Improve] Update the concept and introduction guide
healchow closed issue #600: [Improve] Update the concept and introduction guide URL: https://github.com/apache/inlong-website/issues/600 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong-website] branch master updated: [INLONG-600][Doc] Update the concept and introduction guide (#601)
This is an automated email from the ASF dual-hosted git repository. healchow pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/inlong-website.git The following commit(s) were added to refs/heads/master by this push: new 4ac5ec641e [INLONG-600][Doc] Update the concept and introduction guide (#601) 4ac5ec641e is described below commit 4ac5ec641e6e168218b07adc5a03b9abc6bec865 Author: Charles Zhang AuthorDate: Wed Nov 16 10:13:41 2022 +0800 [INLONG-600][Doc] Update the concept and introduction guide (#601) --- docs/design_and_concept/basic_concept.md | 28 -- docs/introduction.md | 6 ++--- .../current/design_and_concept/basic_concept.md| 28 -- .../current/introduction.md| 12 +- 4 files changed, 39 insertions(+), 35 deletions(-) diff --git a/docs/design_and_concept/basic_concept.md b/docs/design_and_concept/basic_concept.md index 823e96dee4..cf41f67583 100644 --- a/docs/design_and_concept/basic_concept.md +++ b/docs/design_and_concept/basic_concept.md @@ -3,16 +3,18 @@ title: Basic Concept sidebar_position: 1 --- -| Name | Description | Other | -|--||| -| Standard Architecture| Contains all InLong components such as InLong Agent/Manager/MQ/Sort/Dashboard | Suitable for massive data and large-scale production environments | -| Lightweight Architecture | Contains only one component of InLong Sort, which can be used with Manager/Dashboard | The lightweight architecture is simple and flexible, suitable for small-scale data | -| Group| Data Streams Group, it contains multiple data streams, and one Group represents one data ingestion.| Group has attributes such as ID and Name. | -| Stream | Data Stream, a stream has a specific flow direction. | Stream has attributes such as ID, Name, and data fields. | -| Node | Data Node, including `Extract Node` and `Load Node`, stands for the data source and sink types separately.| | -| InLongMsg| InLong data format, if you consume message directly from the message queue, you need to perform `InLongMsg` parsing first. | | -| Agent| The standard architecture uses Agent for data collection, and Agent represents different types of collection capabilities. | It contains File Agent, SQL Agent, Binlog Agent, etc. | -| DataProxy| Forward received data to different message queues. | Supports data transmission blocking, placing retransmission. | -| Sort | Data stream sorting. | Sort-flink based on Flink, sort-standalone for local sorting. | -| TubeMQ | InLong's self-developed message queuing service | It can also be called Tube, with low-cost, high-performance features. | -| Pulsar | [Apache Pulsar](https://pulsar.apache.org/), a high-performance, high-consistency message queue service| | \ No newline at end of file +| Name| Description | Other | +|-||| +| Standard| Standard Architecture, contains all InLong components such as InLong Agent, Manager, MQ, Sort, Dashboard | Suitable for mas
[GitHub] [inlong] bluewang commented on a diff in pull request #6546: [INLONG-6542][Dashboard] Optimize stream execution workflow and execution log
bluewang commented on code in PR #6546:
URL: https://github.com/apache/inlong/pull/6546#discussion_r1023437534
##
inlong-dashboard/src/pages/GroupDetail/DataStream/index.tsx:
##
@@ -178,11 +184,14 @@ const Comp = ({ inlongGroupId, readonly, mqType }: Props,
ref) => {
onDelete(record)}>
{t('basic.Delete')}
-{record?.status && (record?.status === 120 || record?.status ===
130) && (
- onWorkflow(record)}>
-{t('meta.Stream.ExecuteWorkflow')}
-
-)}
+{record?.status &&
+ ((groupStatus === 130 && record?.status === 100) ||
+groupStatus === 120 ||
+groupStatus === 130) && (
Review Comment:
An error occurs when merging：
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #6546: [INLONG-6542][Dashboard] Optimize stream execution workflow and execution log
healchow commented on code in PR #6546:
URL: https://github.com/apache/inlong/pull/6546#discussion_r1023438153
##
inlong-dashboard/src/pages/GroupDetail/DataStream/index.tsx:
##
@@ -178,11 +184,14 @@ const Comp = ({ inlongGroupId, readonly, mqType }: Props,
ref) => {
onDelete(record)}>
{t('basic.Delete')}
-{record?.status && (record?.status === 120 || record?.status ===
130) && (
- onWorkflow(record)}>
-{t('meta.Stream.ExecuteWorkflow')}
-
-)}
+{record?.status &&
+ ((groupStatus === 130 && record?.status === 100) ||
+groupStatus === 120 ||
+groupStatus === 130) && (
Review Comment:
Okay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong-website] bluewang opened a new issue, #602: [Improve] Optimize the team page to better display the team
bluewang opened a new issue, #602: URL: https://github.com/apache/inlong-website/issues/602 ### Description _No response_ ### Are you willing to submit PR? - [X] Yes, I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #6541: [INLONG-6495][Sort] Support inlong metrics for single table doris and support metric restore for it
EMsnap commented on code in PR #6541:
URL: https://github.com/apache/inlong/pull/6541#discussion_r1023439792
##
inlong-sort/sort-connectors/doris/src/main/java/org/apache/inlong/sort/doris/table/DorisSingleTableFormat.java:
##
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.doris.table;
+
+import org.apache.doris.flink.cfg.DorisExecutionOptions;
+import org.apache.doris.flink.cfg.DorisOptions;
+import org.apache.doris.flink.cfg.DorisReadOptions;
+import org.apache.doris.flink.table.DorisDynamicOutputFormat;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeHint;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.inlong.sort.base.metric.MetricOption;
+import org.apache.inlong.sort.base.metric.MetricState;
+import org.apache.inlong.sort.base.metric.SinkMetricData;
+import org.apache.inlong.sort.base.util.MetricStateUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+
+import static org.apache.inlong.sort.base.Constants.INLONG_METRIC_STATE_NAME;
+import static org.apache.inlong.sort.base.Constants.NUM_BYTES_OUT;
+import static org.apache.inlong.sort.base.Constants.NUM_RECORDS_OUT;
+
+/**
+ * DorisSingleTableFormat extends
{org.apache.doris.flink.table.DorisDynamicOutputFormat}
+ * It is used for single table doris jobs, enhanced the original class to
support metric and
+ * snapshot
+ */
+public class DorisSingleTableFormat extends DorisDynamicOutputFormat
implements DorisOutputFormat {
+
+private static final long serialVersionUID = 1L;
+private static final Logger LOG =
LoggerFactory.getLogger(DorisSingleTableFormat.class);
+private final String inlongMetric;
+private final String auditHostAndPorts;
+private transient SinkMetricData metricData;
+private transient ListState metricStateListState;
+private transient MetricState metricState;
+private volatile long datasize;
+
+public DorisSingleTableFormat(DorisOptions option,
+DorisReadOptions readOptions,
+DorisExecutionOptions executionOptions,
+LogicalType[] logicalTypes,
+String[] fieldNames,
+String inlongMetric,
+String auditHostAndPorts) {
+super(option, readOptions, executionOptions, logicalTypes, fieldNames);
+this.inlongMetric = inlongMetric;
+this.auditHostAndPorts = auditHostAndPorts;
+}
+
+public static SingleTableFormatBuilder builder() {
+return new SingleTableFormatBuilder();
+}
+
+@Override
+public void open(int taskNumber, int numTasks) throws IOException {
+super.open(taskNumber, numTasks);
+MetricOption metricOption;
+metricOption = MetricOption.builder()
+.withInlongLabels(inlongMetric)
+.withInlongAudit(auditHostAndPorts)
+.withInitRecords(metricState != null ?
metricState.getMetricValue(NUM_RECORDS_OUT) : 0L)
+.withInitBytes(metricState != null ?
metricState.getMetricValue(NUM_BYTES_OUT) : 0L)
+.withRegisterMetric(MetricOption.RegisteredMetric.ALL)
+.build();
+
+if (metricOption != null) {
+metricData = new SinkMetricData(metricOption,
getRuntimeContext().getMetricGroup());
+}
+}
+
+@Override
+public synchronized void writeRecord(T row) throws IOException {
+super.writeRecord(row);
+try {
+if (metricData != null) {
+metricData.invokeWithEstimate(row);
+datasize++;
Review Comment:
please use camel case thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
[GitHub] [inlong] leezng commented on a diff in pull request #6546: [INLONG-6542][Dashboard] Optimize stream execution workflow and execution log
leezng commented on code in PR #6546:
URL: https://github.com/apache/inlong/pull/6546#discussion_r1023438219
##
inlong-dashboard/src/pages/GroupDetail/DataStream/index.tsx:
##
@@ -72,6 +74,10 @@ const Comp = ({ inlongGroupId, readonly, mqType }: Props,
ref) => {
},
);
+ const groupInfo = useRequest(`/group/get/${inlongGroupId}`, {
Review Comment:
remove `const groupInfo =` ?
##
inlong-dashboard/src/pages/GroupDetail/DataStream/index.tsx:
##
@@ -178,11 +184,14 @@ const Comp = ({ inlongGroupId, readonly, mqType }: Props,
ref) => {
onDelete(record)}>
{t('basic.Delete')}
-{record?.status && (record?.status === 120 || record?.status ===
130) && (
- onWorkflow(record)}>
-{t('meta.Stream.ExecuteWorkflow')}
-
-)}
+{record?.status &&
+ ((groupStatus === 130 && record?.status === 100) ||
Review Comment:
Are the two conditions `groupStatus === 130 && record?.status === 100` and
`groupStatus === 130` repeated, and only need `groupStatus === 130`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] Yizhou-Yang commented on pull request #6541: [INLONG-6495][Sort] Support inlong metrics for single table doris and support metric restore for it
Yizhou-Yang commented on PR #6541: URL: https://github.com/apache/inlong/pull/6541#issuecomment-1316218744 whole db already added by kuanliu last week -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] Yizhou-Yang commented on pull request #6541: [INLONG-6495][Sort] Support inlong metrics for single table doris and support metric restore for it
Yizhou-Yang commented on PR #6541: URL: https://github.com/apache/inlong/pull/6541#issuecomment-1316219103 > Can you also add metrics for whole migrate ? thx > whole db already added by kuanliu last week -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] yunqingmoswu merged pull request #6549: [INLONG-6548][Sort] Optimize metadata field naming for format of canal-json
yunqingmoswu merged PR #6549: URL: https://github.com/apache/inlong/pull/6549 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-6548][Sort] Optimize metadata field naming for format of canal-json (#6549)
This is an automated email from the ASF dual-hosted git repository.
yunqing pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 489529262 [INLONG-6548][Sort] Optimize metadata field naming for
format of canal-json (#6549)
489529262 is described below
commit 4895292623c22a1d268bc8ce25366143bcd18005
Author: yunqingmoswu <44659300+yunqingmo...@users.noreply.github.com>
AuthorDate: Wed Nov 16 11:08:21 2022 +0800
[INLONG-6548][Sort] Optimize metadata field naming for format of canal-json
(#6549)
---
.../protocol/node/extract/KafkaExtractNode.java| 2 +-
.../sort/protocol/node/load/KafkaLoadNode.java | 8 +-
.../node/extract/KafkaExtractNodeTest.java | 3 +-
.../sort/protocol/node/load/KafkaLoadNodeTest.java | 2 +-
.../canal/CanalJsonEnhancedDecodingFormat.java | 19 ++
.../CanalJsonEnhancedDeserializationSchema.java| 260 -
.../canal/CanalJsonEnhancedEncodingFormat.java | 19 ++
.../CanalJsonEnhancedSerializationSchema.java | 52 ++---
8 files changed, 217 insertions(+), 148 deletions(-)
diff --git
a/inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/extract/KafkaExtractNode.java
b/inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/extract/KafkaExtractNode.java
index 02da2bef3..fb6693548 100644
---
a/inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/extract/KafkaExtractNode.java
+++
b/inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/extract/KafkaExtractNode.java
@@ -215,7 +215,7 @@ public class KafkaExtractNode extends ExtractNode
implements InlongMetric, Metad
metadataKey = "value.event-timestamp";
break;
case OP_TYPE:
-metadataKey = "value.op-type";
+metadataKey = "value.type";
break;
case IS_DDL:
metadataKey = "value.is-ddl";
diff --git
a/inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/KafkaLoadNode.java
b/inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/KafkaLoadNode.java
index be326b478..26fa424ca 100644
---
a/inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/KafkaLoadNode.java
+++
b/inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/KafkaLoadNode.java
@@ -224,7 +224,7 @@ public class KafkaLoadNode extends LoadNode implements
InlongMetric, Metadata, S
metadataKey = "value.event-timestamp";
break;
case OP_TYPE:
-metadataKey = "value.op-type";
+metadataKey = "value.type";
break;
case DATA:
case DATA_CANAL:
@@ -257,8 +257,8 @@ public class KafkaLoadNode extends LoadNode implements
InlongMetric, Metadata, S
@Override
public Set supportedMetaFields() {
return EnumSet.of(MetaField.PROCESS_TIME, MetaField.TABLE_NAME,
MetaField.OP_TYPE,
-MetaField.DATABASE_NAME, MetaField.SQL_TYPE, MetaField.PK_NAMES,
MetaField.TS,
-MetaField.OP_TS, MetaField.IS_DDL, MetaField.MYSQL_TYPE,
MetaField.BATCH_ID,
-MetaField.UPDATE_BEFORE, MetaField.DATA_CANAL, MetaField.DATA);
+MetaField.DATABASE_NAME, MetaField.SQL_TYPE,
MetaField.PK_NAMES, MetaField.TS,
+MetaField.OP_TS, MetaField.IS_DDL, MetaField.MYSQL_TYPE,
MetaField.BATCH_ID,
+MetaField.UPDATE_BEFORE, MetaField.DATA_CANAL, MetaField.DATA);
}
}
diff --git
a/inlong-sort/sort-common/src/test/java/org/apache/inlong/sort/protocol/node/extract/KafkaExtractNodeTest.java
b/inlong-sort/sort-common/src/test/java/org/apache/inlong/sort/protocol/node/extract/KafkaExtractNodeTest.java
index 661d6f547..1d41c96fd 100644
---
a/inlong-sort/sort-common/src/test/java/org/apache/inlong/sort/protocol/node/extract/KafkaExtractNodeTest.java
+++
b/inlong-sort/sort-common/src/test/java/org/apache/inlong/sort/protocol/node/extract/KafkaExtractNodeTest.java
@@ -34,7 +34,6 @@ import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-
import static org.junit.Assert.assertEquals;
/**
@@ -86,7 +85,7 @@ public class KafkaExtractNodeTest extends
SerializeBaseTest {
formatMap.put(MetaField.PROCESS_TIME, "AS PROCTIME()");
formatMap.put(MetaField.TABLE_NAME, "STRING METADATA FROM
'value.table'");
formatMap.put(MetaField.DATABASE_NAME, "STRING METADATA FROM
'value.database'");
-formatMap.put(MetaField.OP_TYPE, "STRING METADATA FROM
'value.op-type'");
+formatMap.put(MetaField.OP_TYPE, "STRING METADATA FROM 'value.type'");
formatMap.put(MetaField.OP_TS, "TIMESTAMP_LTZ(3) METADATA FROM
'value.event-timestamp'");
formatMap.pu
[GitHub] [inlong] gong commented on pull request #6541: [INLONG-6495][Sort] Support inlong metrics for single table doris and support metric restore for it
gong commented on PR #6541: URL: https://github.com/apache/inlong/pull/6541#issuecomment-1316242976 I suggest you can add metric computing Doris `Dynamic Output Format#flush`. Because we need to count data after data is written -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #6523: [INLONG-6522][Docker] Add notes for building arm images
healchow commented on code in PR #6523:
URL: https://github.com/apache/inlong/pull/6523#discussion_r1023462361
##
docker/build-docker-images.sh:
##
@@ -143,6 +146,11 @@ if [ "$BUILD_ARCH" = "$ARCH_X86" ]; then
docker ${USE_BUILDX} build ${USE_PLATFORM} ${TYPE} -t
inlong/tubemq-build:${tag} inlong-tubemq/tubemq-docker/tubemq-build/
fi
+echo "=== Build images result ==="
+docker images | grep inlong
+
Review Comment:
Do we need to add "End to build" info?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #6523: [INLONG-6522][Docker] Add notes for building arm images
healchow commented on code in PR #6523:
URL: https://github.com/apache/inlong/pull/6523#discussion_r1023462361
##
docker/build-docker-images.sh:
##
@@ -143,6 +146,11 @@ if [ "$BUILD_ARCH" = "$ARCH_X86" ]; then
docker ${USE_BUILDX} build ${USE_PLATFORM} ${TYPE} -t
inlong/tubemq-build:${tag} inlong-tubemq/tubemq-docker/tubemq-build/
fi
+echo "=== Build images result ==="
+docker images | grep inlong
+
Review Comment:
Do we need to add "End to build" info?
I see you add the "End to tag" after the tag result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[inlong] branch branch-1.4 updated: [INLONG-6552][Sort] Fix MicroTimestamp error in oracle connector (#6554)
This is an automated email from the ASF dual-hosted git repository.
zirui pushed a commit to branch branch-1.4
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/branch-1.4 by this push:
new 543a4b7df [INLONG-6552][Sort] Fix MicroTimestamp error in oracle
connector (#6554)
543a4b7df is described below
commit 543a4b7df5959b4fcd1be9d4e7220d6caa056ca5
Author: emhui <111486498+e-m...@users.noreply.github.com>
AuthorDate: Wed Nov 16 10:12:26 2022 +0800
[INLONG-6552][Sort] Fix MicroTimestamp error in oracle connector (#6554)
---
.../cdc/oracle/debezium/table/RowDataDebeziumDeserializeSchema.java| 3 +++
1 file changed, 3 insertions(+)
diff --git
a/inlong-sort/sort-connectors/oracle-cdc/src/main/java/org/apache/inlong/sort/cdc/oracle/debezium/table/RowDataDebeziumDeserializeSchema.java
b/inlong-sort/sort-connectors/oracle-cdc/src/main/java/org/apache/inlong/sort/cdc/oracle/debezium/table/RowDataDebeziumDeserializeSchema.java
index 530b96d6d..2569d4154 100644
---
a/inlong-sort/sort-connectors/oracle-cdc/src/main/java/org/apache/inlong/sort/cdc/oracle/debezium/table/RowDataDebeziumDeserializeSchema.java
+++
b/inlong-sort/sort-connectors/oracle-cdc/src/main/java/org/apache/inlong/sort/cdc/oracle/debezium/table/RowDataDebeziumDeserializeSchema.java
@@ -773,6 +773,9 @@ public final class RowDataDebeziumDeserializeSchema
} else if (Timestamp.SCHEMA_NAME.equals(schemaName)) {
Instant instantTime = Instant.ofEpochMilli((Long) fieldValue);
fieldValue = LocalDateTime.ofInstant(instantTime,
ZONE_UTC).toString();
+} else if (MicroTimestamp.SCHEMA_NAME.equals(schemaName)) {
+Instant instantTime = Instant.ofEpochMilli((Long) fieldValue /
1000);
+fieldValue = LocalDateTime.ofInstant(instantTime,
ZONE_UTC).toString();
}
return fieldValue;
}
[inlong] branch master updated: [INLONG-6542][Dashboard] Optimize stream execution workflow and execution log (#6546)
This is an automated email from the ASF dual-hosted git repository.
leezng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new cf12ea389 [INLONG-6542][Dashboard] Optimize stream execution workflow
and execution log (#6546)
cf12ea389 is described below
commit cf12ea389cddc955a8487cf83ab544be25e649c6
Author: Lizhen <88174078+bluew...@users.noreply.github.com>
AuthorDate: Wed Nov 16 11:29:31 2022 +0800
[INLONG-6542][Dashboard] Optimize stream execution workflow and execution
log (#6546)
---
inlong-dashboard/src/pages/GroupDashboard/ExecutionLogModal.tsx | 2 +-
.../src/pages/GroupDetail/DataStream/ExecutionLogModal.tsx| 2 +-
inlong-dashboard/src/pages/GroupDetail/DataStream/index.tsx | 8 +++-
3 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/inlong-dashboard/src/pages/GroupDashboard/ExecutionLogModal.tsx
b/inlong-dashboard/src/pages/GroupDashboard/ExecutionLogModal.tsx
index 70009fb56..d26f2ad50 100644
--- a/inlong-dashboard/src/pages/GroupDashboard/ExecutionLogModal.tsx
+++ b/inlong-dashboard/src/pages/GroupDashboard/ExecutionLogModal.tsx
@@ -47,7 +47,7 @@ const Comp: React.FC = ({ inlongGroupId, ...modalProps
}) => {
params: {
...options,
inlongGroupId: inlongGroupId,
-processNames: 'CREATE_GROUP_RESOURCE,CREATE_STREAM_RESOURCE',
+processNames: 'CREATE_GROUP_RESOURCE',
taskType: 'ServiceTask',
},
},
diff --git
a/inlong-dashboard/src/pages/GroupDetail/DataStream/ExecutionLogModal.tsx
b/inlong-dashboard/src/pages/GroupDetail/DataStream/ExecutionLogModal.tsx
index 6445135c8..0c195c9c6 100644
--- a/inlong-dashboard/src/pages/GroupDetail/DataStream/ExecutionLogModal.tsx
+++ b/inlong-dashboard/src/pages/GroupDetail/DataStream/ExecutionLogModal.tsx
@@ -49,7 +49,7 @@ const Comp: React.FC = ({ inlongGroupId,
inlongStreamId, ...modalProps })
...options,
inlongGroupId: inlongGroupId,
inlongStreamId: inlongStreamId,
-processNames: 'CREATE_GROUP_RESOURCE,CREATE_STREAM_RESOURCE',
+processNames: 'CREATE_STREAM_RESOURCE',
taskType: 'ServiceTask',
},
},
diff --git a/inlong-dashboard/src/pages/GroupDetail/DataStream/index.tsx
b/inlong-dashboard/src/pages/GroupDetail/DataStream/index.tsx
index 065710d64..b6f499594 100644
--- a/inlong-dashboard/src/pages/GroupDetail/DataStream/index.tsx
+++ b/inlong-dashboard/src/pages/GroupDetail/DataStream/index.tsx
@@ -54,6 +54,8 @@ const Comp = ({ inlongGroupId, readonly, mqType }: Props,
ref) => {
inlongStreamId: '',
});
+ const [groupStatus, setGroupStatus] = useState();
+
const {
data,
loading,
@@ -72,6 +74,10 @@ const Comp = ({ inlongGroupId, readonly, mqType }: Props,
ref) => {
},
);
+ useRequest(`/group/get/${inlongGroupId}`, {
+onSuccess: result => setGroupStatus(result.status),
+ });
+
const onOk = () => {
return Promise.resolve();
};
@@ -178,7 +184,7 @@ const Comp = ({ inlongGroupId, readonly, mqType }: Props,
ref) => {
onDelete(record)}>
{t('basic.Delete')}
-{record?.status && (record?.status === 120 || record?.status ===
130) && (
+{record?.status && (groupStatus === 120 || groupStatus === 130) &&
(
onWorkflow(record)}>
{t('meta.Stream.ExecuteWorkflow')}
[GitHub] [inlong] leezng merged pull request #6546: [INLONG-6542][Dashboard] Optimize stream execution workflow and execution log
leezng merged PR #6546: URL: https://github.com/apache/inlong/pull/6546 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] 01/01: [INLONG-6555][CI] Update for set-output command is deprecated
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch fix-6555
in repository https://gitbox.apache.org/repos/asf/inlong.git
commit 1f2fb2329a8d38f9ef1585010d990a0949327001
Author: dockerzhang
AuthorDate: Wed Nov 16 14:55:59 2022 +0800
[INLONG-6555][CI] Update for set-output command is deprecated
---
.github/workflows/ci_chart_test.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/ci_chart_test.yml
b/.github/workflows/ci_chart_test.yml
index dabf18b2a..fa92911a7 100644
--- a/.github/workflows/ci_chart_test.yml
+++ b/.github/workflows/ci_chart_test.yml
@@ -76,7 +76,7 @@ jobs:
run: |
changed=$(ct list-changed --config ${{ env.CT_CONFIG_PATH }}
--target-branch ${{ github.event.repository.default_branch }})
if [[ -n "$changed" ]]; then
-echo "::set-output name=changed::true"
+echo "name=changed::true" >> $GITHUB_OUTPUT
echo "Changed charts: $changed"
else
echo "Charts are not changed, they will not be linted, validated,
installed and tested."
[inlong] branch fix-6555 created (now 1f2fb2329)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch fix-6555 in repository https://gitbox.apache.org/repos/asf/inlong.git at 1f2fb2329 [INLONG-6555][CI] Update for set-output command is deprecated This branch includes the following new commits: new 1f2fb2329 [INLONG-6555][CI] Update for set-output command is deprecated The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[inlong] 01/01: [INLONG-6555][CI] Update for set-output command is deprecated
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch fix-6555
in repository https://gitbox.apache.org/repos/asf/inlong.git
commit 8925fa6728b01a2b07392d21e6b284a59523d592
Author: dockerzhang
AuthorDate: Wed Nov 16 14:55:59 2022 +0800
[INLONG-6555][CI] Update for set-output command is deprecated
---
.github/workflows/ci_chart_test.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/ci_chart_test.yml
b/.github/workflows/ci_chart_test.yml
index dabf18b2a..fa92911a7 100644
--- a/.github/workflows/ci_chart_test.yml
+++ b/.github/workflows/ci_chart_test.yml
@@ -76,7 +76,7 @@ jobs:
run: |
changed=$(ct list-changed --config ${{ env.CT_CONFIG_PATH }}
--target-branch ${{ github.event.repository.default_branch }})
if [[ -n "$changed" ]]; then
-echo "::set-output name=changed::true"
+echo "name=changed::true" >> $GITHUB_OUTPUT
echo "Changed charts: $changed"
else
echo "Charts are not changed, they will not be linted, validated,
installed and tested."
[inlong] branch fix-6555 updated (1f2fb2329 -> 8925fa672)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch fix-6555 in repository https://gitbox.apache.org/repos/asf/inlong.git discard 1f2fb2329 [INLONG-6555][CI] Update for set-output command is deprecated omit a17152fc8 [INLONG-6555][CI] Update for set-output command is deprecated add cceca481c [INLONG-6555][CI] Update for set-output command is deprecated (#6556) add 531464d0f [INLONG-6552][Sort] Fix MicroTimestamp error in oracle connector (#6554) add 489529262 [INLONG-6548][Sort] Optimize metadata field naming for format of canal-json (#6549) add cf12ea389 [INLONG-6542][Dashboard] Optimize stream execution workflow and execution log (#6546) new 8925fa672 [INLONG-6555][CI] Update for set-output command is deprecated This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (1f2fb2329) \ N -- N -- N refs/heads/fix-6555 (8925fa672) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../src/pages/GroupDashboard/ExecutionLogModal.tsx | 2 +- .../GroupDetail/DataStream/ExecutionLogModal.tsx | 2 +- .../src/pages/GroupDetail/DataStream/index.tsx | 8 +- .../protocol/node/extract/KafkaExtractNode.java| 2 +- .../sort/protocol/node/load/KafkaLoadNode.java | 8 +- .../node/extract/KafkaExtractNodeTest.java | 3 +- .../sort/protocol/node/load/KafkaLoadNodeTest.java | 2 +- .../table/RowDataDebeziumDeserializeSchema.java| 3 + .../canal/CanalJsonEnhancedDecodingFormat.java | 19 ++ .../CanalJsonEnhancedDeserializationSchema.java| 260 - .../canal/CanalJsonEnhancedEncodingFormat.java | 19 ++ .../CanalJsonEnhancedSerializationSchema.java | 52 ++--- 12 files changed, 229 insertions(+), 151 deletions(-)
[GitHub] [inlong] dockerzhang opened a new pull request, #6557: [INLONG-6555][CI] Update for set-output command is deprecated
dockerzhang opened a new pull request, #6557: URL: https://github.com/apache/inlong/pull/6557 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #6555 ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve?* ### Modifications *Describe the modifications you've done.* ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] 02/02: [INLONG-6542][Dashboard] Optimize stream execution workflow and execution log (#6546)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch branch-1.4
in repository https://gitbox.apache.org/repos/asf/inlong.git
commit a0af8571f3e67d80cb8f47a8d977a3c090017286
Author: Lizhen <88174078+bluew...@users.noreply.github.com>
AuthorDate: Wed Nov 16 11:29:31 2022 +0800
[INLONG-6542][Dashboard] Optimize stream execution workflow and execution
log (#6546)
---
inlong-dashboard/src/pages/GroupDashboard/ExecutionLogModal.tsx | 2 +-
.../src/pages/GroupDetail/DataStream/ExecutionLogModal.tsx| 2 +-
inlong-dashboard/src/pages/GroupDetail/DataStream/index.tsx | 8 +++-
3 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/inlong-dashboard/src/pages/GroupDashboard/ExecutionLogModal.tsx
b/inlong-dashboard/src/pages/GroupDashboard/ExecutionLogModal.tsx
index 70009fb56..d26f2ad50 100644
--- a/inlong-dashboard/src/pages/GroupDashboard/ExecutionLogModal.tsx
+++ b/inlong-dashboard/src/pages/GroupDashboard/ExecutionLogModal.tsx
@@ -47,7 +47,7 @@ const Comp: React.FC = ({ inlongGroupId, ...modalProps
}) => {
params: {
...options,
inlongGroupId: inlongGroupId,
-processNames: 'CREATE_GROUP_RESOURCE,CREATE_STREAM_RESOURCE',
+processNames: 'CREATE_GROUP_RESOURCE',
taskType: 'ServiceTask',
},
},
diff --git
a/inlong-dashboard/src/pages/GroupDetail/DataStream/ExecutionLogModal.tsx
b/inlong-dashboard/src/pages/GroupDetail/DataStream/ExecutionLogModal.tsx
index 6445135c8..0c195c9c6 100644
--- a/inlong-dashboard/src/pages/GroupDetail/DataStream/ExecutionLogModal.tsx
+++ b/inlong-dashboard/src/pages/GroupDetail/DataStream/ExecutionLogModal.tsx
@@ -49,7 +49,7 @@ const Comp: React.FC = ({ inlongGroupId,
inlongStreamId, ...modalProps })
...options,
inlongGroupId: inlongGroupId,
inlongStreamId: inlongStreamId,
-processNames: 'CREATE_GROUP_RESOURCE,CREATE_STREAM_RESOURCE',
+processNames: 'CREATE_STREAM_RESOURCE',
taskType: 'ServiceTask',
},
},
diff --git a/inlong-dashboard/src/pages/GroupDetail/DataStream/index.tsx
b/inlong-dashboard/src/pages/GroupDetail/DataStream/index.tsx
index 065710d64..b6f499594 100644
--- a/inlong-dashboard/src/pages/GroupDetail/DataStream/index.tsx
+++ b/inlong-dashboard/src/pages/GroupDetail/DataStream/index.tsx
@@ -54,6 +54,8 @@ const Comp = ({ inlongGroupId, readonly, mqType }: Props,
ref) => {
inlongStreamId: '',
});
+ const [groupStatus, setGroupStatus] = useState();
+
const {
data,
loading,
@@ -72,6 +74,10 @@ const Comp = ({ inlongGroupId, readonly, mqType }: Props,
ref) => {
},
);
+ useRequest(`/group/get/${inlongGroupId}`, {
+onSuccess: result => setGroupStatus(result.status),
+ });
+
const onOk = () => {
return Promise.resolve();
};
@@ -178,7 +184,7 @@ const Comp = ({ inlongGroupId, readonly, mqType }: Props,
ref) => {
onDelete(record)}>
{t('basic.Delete')}
-{record?.status && (record?.status === 120 || record?.status ===
130) && (
+{record?.status && (groupStatus === 120 || groupStatus === 130) &&
(
onWorkflow(record)}>
{t('meta.Stream.ExecuteWorkflow')}
[inlong] 01/02: [INLONG-6555][CI] Update for set-output command is deprecated (#6556)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch branch-1.4
in repository https://gitbox.apache.org/repos/asf/inlong.git
commit 8d6ae09d712417e057a8a2ea9b5e3ac3fd978b0e
Author: Charles Zhang
AuthorDate: Wed Nov 16 10:12:01 2022 +0800
[INLONG-6555][CI] Update for set-output command is deprecated (#6556)
---
.github/workflows/ci_docker.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/ci_docker.yml b/.github/workflows/ci_docker.yml
index 98af35529..18e253581 100644
--- a/.github/workflows/ci_docker.yml
+++ b/.github/workflows/ci_docker.yml
@@ -94,9 +94,9 @@ jobs:
&& github.repository_owner == 'apache'
run: |
if [[ ${{ github.ref_name }} == ${{
github.event.repository.default_branch }} ]]; then
-echo "::set-output name=match_master::true"
+echo "name=match_master::true" >> $GITHUB_OUTPUT
elif [[ ${{ github.ref_name }} =~ ^branch-[0-9]+\.[0-9]+$ ]]; then
-echo "::set-output name=match_release::true"
+echo "name=match_release::true" >> $GITHUB_OUTPUT
fi
# Publish x86 Docker images when the changes are being pushed to the
master branch or a release branch like 'branch-1.4'.
[inlong] branch branch-1.4 updated (543a4b7df -> a0af8571f)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch branch-1.4 in repository https://gitbox.apache.org/repos/asf/inlong.git from 543a4b7df [INLONG-6552][Sort] Fix MicroTimestamp error in oracle connector (#6554) new 8d6ae09d7 [INLONG-6555][CI] Update for set-output command is deprecated (#6556) new a0af8571f [INLONG-6542][Dashboard] Optimize stream execution workflow and execution log (#6546) The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/workflows/ci_docker.yml | 4 ++-- inlong-dashboard/src/pages/GroupDashboard/ExecutionLogModal.tsx | 2 +- .../src/pages/GroupDetail/DataStream/ExecutionLogModal.tsx| 2 +- inlong-dashboard/src/pages/GroupDetail/DataStream/index.tsx | 8 +++- 4 files changed, 11 insertions(+), 5 deletions(-)
[GitHub] [inlong-website] EMsnap commented on a diff in pull request #592: [INLONG-591][Doc] Add document for multiple sink of iceberg
EMsnap commented on code in PR #592:
URL: https://github.com/apache/inlong-website/pull/592#discussion_r1023578305
##
i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/load_node/iceberg.md:
##
@@ -148,6 +148,98 @@ TODO
### InLong Manager Client 用法
TODO
+## 特征
+### 多表写入
+目前 Iceberg 支持多表同时写入，需要在 FLINK SQL 的建表参数上添加 `'sink.multiple.enable' = 'true'`
并且目标表的schema
+只能定义成 `BINARY` 或者 `STRING` ，以下是一个建表语句举例：
+```
+CREATE TABLE `table_2`(
+`data` STRING)
+WITH (
+'connector'='iceberg-inlong',
+'catalog-name'='hive_prod',
+'uri'='thrift://localhost:9083',
+'warehouse'='hdfs://localhost:8020/hive/warehouse',
+'sink.multiple.enable' = 'true',
+'sink.multiple.format' = 'canal-json',
+'sink.multiple.add-column.policy' = 'TRY_IT_BEST',
+'sink.multiple.database-pattern' = '${database}',
+'sink.multiple.table-pattern' = 'test_${table}'
+);
+```
+要支持多表写入同时需要设置上游数据的序列化格式(通过选项 'sink.multiple.format'
+来设置, 目前仅支持 [canal-json|debezium-json])。
+
+### 动态表名映射
+Iceberg 在多表写入的时可以自定义映射的数据库名和表名的规则，可以填充占位符然后添加前后缀来修改映射的目标表名称。
+Iceberg Load Node 会解析 `'sink.multiple.database-pattern'` 作为目的端的 数据库名, 解析
`'sink.multiple.table-pattern'`
+作为目的端的表名，占位符是从数据中解析出来的，变量是严格通过 '${VARIABLE_NAME}' 来表示, 变量的取值来自于数据本身,
+即可以是通过 `'sink.multiple.format'` 指定的某种 Format 的元数据字段, 也可以是数据中的物理字段。
+关于 'topic-parttern' 的例子如下:
+- 'sink.multiple.format' 为 'canal-json':
+
+上游数据为:
+```
+{
+ "data": [
+{
+ "id": "111",
+ "name": "scooter",
+ "description": "Big 2-wheel scooter",
+ "weight": "5.18"
+}
+ ],
+ "database": "inventory",
+ "es": 158937356,
+ "id": 9,
+ "isDdl": false,
+ "mysqlType": {
+"id": "INTEGER",
+"name": "VARCHAR(255)",
+"description": "VARCHAR(512)",
+"weight": "FLOAT"
+ },
+ "old": [
+{
+ "weight": "5.15"
+}
+ ],
+ "pkNames": [
+"id"
+ ],
+ "sql": "",
+ "sqlType": {
+"id": 4,
+"name": 12,
+"description": 12,
+"weight": 7
+ },
+ "table": "products",
+ "ts": 1589373560798,
+ "type": "UPDATE"
+}
+```
+'topic-pattern' 为 '{database}_${table}', 提取后的 Topic 为 'inventory_products'
('database', 'table' 为元数据字段,
+'id' 为物理字段)
+
+'topic-pattern' 为 '{database}_${table}_${id}', 提取后的 Topic 为
'inventory_products_4' ('database', 'table'
Review Comment:
111 -> 4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong-website] bluewang opened a new pull request, #603: [INLONG-602] Optimize the team page to better display the team
bluewang opened a new pull request, #603: URL: https://github.com/apache/inlong-website/pull/603 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes https://github.com/apache/inlong-website/issues/602 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong-website] branch fix-588 created (now 9571d21877)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch fix-588 in repository https://gitbox.apache.org/repos/asf/inlong-website.git at 9571d21877 [INLONG-588][Release] Modify download & release notes for 1.4.0 This branch includes the following new commits: new 9571d21877 [INLONG-588][Release] Modify download & release notes for 1.4.0 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[GitHub] [inlong-website] dockerzhang opened a new pull request, #604: [INLONG-588][Release] Modify download & release notes for 1.4.0
dockerzhang opened a new pull request, #604: URL: https://github.com/apache/inlong-website/pull/604 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #588 ### Motivation Modify download & release notes for 1.4.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
