[GitHub] [inlong] dockerzhang merged pull request #6084: [INLONG-6068][Manager] Optimize the usage of ObjectMapper
dockerzhang merged PR #6084: URL: https://github.com/apache/inlong/pull/6084 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang closed pull request #6075: [INLONG-6071][Manager] Support Apache Doris sink
dockerzhang closed pull request #6075: [INLONG-6071][Manager] Support Apache Doris sink URL: https://github.com/apache/inlong/pull/6075 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang commented on pull request #6075: [INLONG-6071][Manager] Support Apache Doris sink
dockerzhang commented on PR #6075: URL: https://github.com/apache/inlong/pull/6075#issuecomment-1271229553 duplicated with https://github.com/apache/inlong/pull/6075, close it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] healchow commented on issue #6071: [Feature][Manager] Support Apache Doris sink
healchow commented on issue #6071: URL: https://github.com/apache/inlong/issues/6071#issuecomment-1272194799 Duplicated with #5043. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #6067: [INLONG-6061][Sort] Support TDSQL-Kafka connector
healchow commented on code in PR #6067: URL: https://github.com/apache/inlong/pull/6067#discussion_r990572691 ## inlong-common/src/main/java/org/apache/inlong/common/enums/TaskTypeEnum.java: ## @@ -31,6 +31,9 @@ public enum TaskTypeEnum { MONGODB(9), TUBEMQ(10), REDIS(11), +CKAFKA(12), +TDSQL_KAFKA(14), Review Comment: Why no `13` here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #6067: [INLONG-6061][Sort] Support TDSQL-Kafka connector
healchow commented on code in PR #6067: URL: https://github.com/apache/inlong/pull/6067#discussion_r990572823 ## inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/constant/TDSQLKafkaConstant.java: ## @@ -0,0 +1,40 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.inlong.sort.protocol.constant; + +/** + * tdsql kafka option constant Review Comment: ```suggestion * TDSQL Kafka option constant ``` ## inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/extract/TDSQLKafkaExtractNode.java: ## @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.inlong.sort.protocol.node.extract; + +import java.io.Serializable; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import javax.annotation.Nonnull; +import javax.annotation.Nullable; +import lombok.EqualsAndHashCode; +import org.apache.commons.lang3.StringUtils; +import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonProperty; +import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonTypeName; +import org.apache.inlong.sort.protocol.FieldInfo; +import org.apache.inlong.sort.protocol.InlongMetric; +import org.apache.inlong.sort.protocol.Metadata; +import org.apache.inlong.sort.protocol.constant.TDSQLKafkaConstant; +import org.apache.inlong.sort.protocol.enums.KafkaScanStartupMode; +import org.apache.inlong.sort.protocol.node.format.Format; +import org.apache.inlong.sort.protocol.node.format.ProtobufFormat; +import org.apache.inlong.sort.protocol.transformation.WatermarkField; + +/** + * Kafka extract node for extract data from kafka Review Comment: ```suggestion * TDSQL Kafka extract node info ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] vernedeng opened a new pull request, #6087: [INLONG-6086][Manager] support base stream sink operator
vernedeng opened a new pull request, #6087: URL: https://github.com/apache/inlong/pull/6087 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #6086 ### Motivation support base stream sink operator support delete stream sink by related id ### Modifications *Describe the modifications you've done.* ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch release-1.3.0 updated: [INLONG-6076][Audit] Bugs in the ClickHouseService.processOutput() (#6077)
This is an automated email from the ASF dual-hosted git repository. gosonzhang pushed a commit to branch release-1.3.0 in repository https://gitbox.apache.org/repos/asf/inlong.git The following commit(s) were added to refs/heads/release-1.3.0 by this push: new 31c96a0f3 [INLONG-6076][Audit] Bugs in the ClickHouseService.processOutput() (#6077) 31c96a0f3 is described below commit 31c96a0f3217ec2c32aecebbc1803feea126139f Author: Goson Zhang <4675...@qq.com> AuthorDate: Fri Sep 30 19:51:14 2022 +0800 [INLONG-6076][Audit] Bugs in the ClickHouseService.processOutput() (#6077) --- .../inlong/audit/service/ClickHouseService.java| 45 +++--- 1 file changed, 22 insertions(+), 23 deletions(-) diff --git a/inlong-audit/audit-store/src/main/java/org/apache/inlong/audit/service/ClickHouseService.java b/inlong-audit/audit-store/src/main/java/org/apache/inlong/audit/service/ClickHouseService.java index 24042d5aa..6d98973a2 100644 --- a/inlong-audit/audit-store/src/main/java/org/apache/inlong/audit/service/ClickHouseService.java +++ b/inlong-audit/audit-store/src/main/java/org/apache/inlong/audit/service/ClickHouseService.java @@ -34,6 +34,7 @@ import java.util.concurrent.ScheduledExecutorService; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; /** * ClickHouseService @@ -54,7 +55,7 @@ public class ClickHouseService implements InsertData, AutoCloseable { private LinkedBlockingQueue batchQueue; private AtomicBoolean needBatchOutput = new AtomicBoolean(false); private AtomicInteger batchCounter = new AtomicInteger(0); - +private AtomicLong lastCheckTime = new AtomicLong(System.currentTimeMillis()); private Connection conn; /** @@ -77,17 +78,11 @@ public class ClickHouseService implements InsertData, AutoCloseable { Class.forName(chConfig.getDriver()); this.reconnect(); } catch (Exception e) { -LOG.error(e.getMessage(), e); +LOG.error("ClickHouseService start failure!", e); } -// timer -long currentTime = System.currentTimeMillis(); -// batch output interval -timerService.scheduleWithFixedDelay(() -> needBatchOutput.compareAndSet(false, true), -currentTime + chConfig.getBatchIntervalMs(), -chConfig.getBatchIntervalMs(), TimeUnit.MILLISECONDS); -// batch output process -timerService.scheduleWithFixedDelay(() -> processOutput(), -currentTime + chConfig.getProcessIntervalMs(), +// start timer +timerService.scheduleWithFixedDelay(this::processOutput, +chConfig.getProcessIntervalMs(), chConfig.getProcessIntervalMs(), TimeUnit.MILLISECONDS); } @@ -95,14 +90,15 @@ public class ClickHouseService implements InsertData, AutoCloseable { * processOutput */ private void processOutput() { -if (!this.needBatchOutput.get()) { +if (!this.needBatchOutput.get() +&& (System.currentTimeMillis() - lastCheckTime.get() < chConfig.getBatchIntervalMs())) { return; } // output try (PreparedStatement pstat = this.conn.prepareStatement(INSERT_SQL)) { -// insert data -ClickHouseDataPo data = this.batchQueue.poll(); int counter = 0; +// output data to clickhouse +ClickHouseDataPo data = this.batchQueue.poll(); while (data != null) { pstat.setString(1, data.getIp()); pstat.setString(2, data.getDockerId()); @@ -124,20 +120,23 @@ public class ClickHouseService implements InsertData, AutoCloseable { this.conn.commit(); counter = 0; } +data = this.batchQueue.poll(); } -this.batchCounter.set(0); -pstat.executeBatch(); -this.conn.commit(); -} catch (Exception e) { -LOG.error(e.getMessage(), e); +if (counter > 0) { +pstat.executeBatch(); +this.conn.commit(); +} +} catch (Exception e1) { +LOG.error("Execute output to clickhouse failure!", e1); +// re-connect clickhouse try { this.reconnect(); -} catch (SQLException e1) { -LOG.error(e1.getMessage(), e1); +} catch (SQLException e2) { +LOG.error("Re-connect clickhouse failure!", e2); } } - -// recover +// recover flag +lastCheckTime.set(System.currentTimeMillis()); this.needBatchOutput.compareAndSet(true, false); }
[GitHub] [inlong] dockerzhang merged pull request #6083: [INLONG-6082][CI] Fix the error in the greeting action
dockerzhang merged PR #6083: URL: https://github.com/apache/inlong/pull/6083 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated (e4e34ebdc -> 6b46efc04)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong.git from e4e34ebdc [INLONG-6068][Manager] Optimize the usage of ObjectMapper (#6084) add 6b46efc04 [INLONG-6082][CI] Fix the error in the greeting action (#6083) No new revisions were added by this update. Summary of changes: .github/workflows/ci_greeting.yml | 17 + 1 file changed, 13 insertions(+), 4 deletions(-)
[GitHub] [inlong] dockerzhang commented on pull request #6074: [INLONG-6073][Kubernetes] Add audit.config.store.mode Configurable to k8s
dockerzhang commented on PR #6074: URL: https://github.com/apache/inlong/pull/6074#issuecomment-1272214955 how to configure the address for different store types? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] kipshi closed pull request #5716: [INLONG-5714][Manager] Support compatibility for Manager Web Api
kipshi closed pull request #5716: [INLONG-5714][Manager] Support compatibility for Manager Web Api URL: https://github.com/apache/inlong/pull/5716 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] kipshi commented on pull request #5716: [INLONG-5714][Manager] Support compatibility for Manager Web Api
kipshi commented on PR #5716: URL: https://github.com/apache/inlong/pull/5716#issuecomment-1272216753 not necessary at all in master branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang commented on pull request #6081: [INLONG-6080][Manager] Change the DataNodeResponse to DataNodeInfo
dockerzhang commented on PR #6081: URL: https://github.com/apache/inlong/pull/6081#issuecomment-1272217232 @woofyzhao PTAL, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] healchow closed pull request #6067: [INLONG-6061][Sort] Support TDSQL-Kafka connector
healchow closed pull request #6067: [INLONG-6061][Sort] Support TDSQL-Kafka connector URL: https://github.com/apache/inlong/pull/6067 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #6087: [INLONG-6086][Manager] support base stream sink operator
healchow commented on code in PR #6087: URL: https://github.com/apache/inlong/pull/6087#discussion_r990587931 ## inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/consts/SinkType.java: ## @@ -36,5 +36,6 @@ public class SinkType { public static final String ORACLE = "ORACLE"; public static final String TDSQLPOSTGRESQL = "TDSQLPOSTGRESQL"; public static final String DLCICEBERG = "DLCICEBERG"; +public static final String BASE = "BASE"; Review Comment: Excuse me, what is the `BASE` type? I know `MYSQL`, `ORACLE`, `HIVE`, etc., but I haven't heard of the `BASE` type. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] vernedeng commented on a diff in pull request #6087: [INLONG-6086][Manager] support base stream sink operator
vernedeng commented on code in PR #6087: URL: https://github.com/apache/inlong/pull/6087#discussion_r990588344 ## inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/consts/SinkType.java: ## @@ -36,5 +36,6 @@ public class SinkType { public static final String ORACLE = "ORACLE"; public static final String TDSQLPOSTGRESQL = "TDSQLPOSTGRESQL"; public static final String DLCICEBERG = "DLCICEBERG"; +public static final String BASE = "BASE"; Review Comment: To differ from sink type in Sort-flink. Base is just a prefix, the real type is base_kafka, base_pulsar, etc. The Base_XXX type means all fields are directly inserted into the very table, instead of complex checking and multi-table operation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #5843: [INLONG-5842][Manager] Support maintenance of message queue cluster
healchow commented on code in PR #5843: URL: https://github.com/apache/inlong/pull/5843#discussion_r990592577 ## inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/cluster/queue/MessageQueueClearTopicRequest.java: ## @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.inlong.manager.pojo.cluster.queue; + +import org.apache.inlong.manager.common.validation.UpdateValidation; + +import javax.validation.constraints.NotBlank; + +import io.swagger.annotations.ApiModel; +import io.swagger.annotations.ApiModelProperty; +import lombok.AllArgsConstructor; +import lombok.Builder; +import lombok.Data; +import lombok.EqualsAndHashCode; +import lombok.NoArgsConstructor; + +/** + * Clear all topic from a message queue cluster + */ +@Data +@EqualsAndHashCode(callSuper = false) +@Builder +@NoArgsConstructor +@AllArgsConstructor +@ApiModel("Clear all topic from a message queue cluster") +public class MessageQueueClearTopicRequest { + +@NotBlank(groups = UpdateValidation.class, message = "miss message queue cluster name.") Review Comment: Suggested changing to `@NotBlank(message = "cluster name cannot be blank")`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] luchunliang closed pull request #5866: [INLONG-5865][Manager] Support maintenance of inlong group
luchunliang closed pull request #5866: [INLONG-5865][Manager] Support maintenance of inlong group URL: https://github.com/apache/inlong/pull/5866 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] luchunliang closed pull request #5858: [INLONG-5857][Manager] Support maintenance of cluster tag
luchunliang closed pull request #5858: [INLONG-5857][Manager] Support maintenance of cluster tag URL: https://github.com/apache/inlong/pull/5858 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] luchunliang closed pull request #5843: [INLONG-5842][Manager] Support maintenance of message queue cluster
luchunliang closed pull request #5843: [INLONG-5842][Manager] Support maintenance of message queue cluster URL: https://github.com/apache/inlong/pull/5843 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] luchunliang closed pull request #5709: [INLONG-5708][DataProxy] Provide single flume channel to reduce synchronized operation
luchunliang closed pull request #5709: [INLONG-5708][DataProxy] Provide single flume channel to reduce synchronized operation URL: https://github.com/apache/inlong/pull/5709 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] gosonzhang commented on pull request #5630: [INLONG-5628][DataProxy] Add buffer limit of sink dispatch queue
gosonzhang commented on PR #5630: URL: https://github.com/apache/inlong/pull/5630#issuecomment-1272244844 @luchunliang, the PR has conflicts that must be resolved, and I see that LinkedBlockingQueue is used in some sinks, but there is no corresponding replacement for BufferQueue. Can it be replaced uniformly? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang opened a new pull request, #6091: [INLONG-6090][IDEA] Show InLong logo in IDEA
dockerzhang opened a new pull request, #6091: URL: https://github.com/apache/inlong/pull/6091 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #6090 ### Motivation after  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org