[GitHub] [incubator-seatunnel] MonsterChenzhuo commented on issue #3746: [Feature][connector][kafka] Support read `canal` format message from kafka
MonsterChenzhuo commented on issue #3746: URL: https://github.com/apache/incubator-seatunnel/issues/3746#issuecomment-1371913317 @hailin0 Thanks, no need for now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] ediconss commented on pull request #3867: [feature][cdc][postgres] add PostgreSQL CDC
ediconss commented on PR #3867: URL: https://github.com/apache/incubator-seatunnel/pull/3867#issuecomment-1371914553 这个可以现在使用吗，支持同步DDL吗 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] MonsterChenzhuo opened a new issue, #3872: [Feature][connector][kafka] Supplement the test case in json format
MonsterChenzhuo opened a new issue, #3872: URL: https://github.com/apache/incubator-seatunnel/issues/3872 ### Search before asking - [X] I had searched in the [feature](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description Supplement the test case in json format ### Usage Scenario _No response_ ### Related issues _No response_ ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] MonsterChenzhuo opened a new issue, #3873: [Bug] [connector][kafka] json deserialize exception log no content
MonsterChenzhuo opened a new issue, #3873: URL: https://github.com/apache/incubator-seatunnel/issues/3873 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened https://user-images.githubusercontent.com/60029759/210742017-33eea3b5-5b51-4598-ad10-5d8d77ebf7fc.png";> ![Uploading 图片.png…]() ### SeaTunnel Version 2.3.0 ### SeaTunnel Config ```conf null ``` ### Running Command ```shell null ``` ### Error Exception ```log null ``` ### Flink or Spark Version _No response_ ### Java or Scala Version _No response_ ### Screenshots _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] MonsterChenzhuo opened a new pull request, #3874: [Bug] [connector][kafka] json deserialize exception log no content #3873
MonsterChenzhuo opened a new pull request, #3874: URL: https://github.com/apache/incubator-seatunnel/pull/3874 ## Purpose of this pull request close #3873 ## Check list * [ ] Code changed are covered with tests, or it does not need tests for reason: * [ ] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md) * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs * [ ] If you are contributing the connector code, please check that the following files are updated: 1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/incubator-seatunnel/tree/dev/docs/en/connector-v2) 2. Update [plugin-mapping.properties](https://github.com/apache/incubator-seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it 3. Update the pom file of [seatunnel-dist](https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-dist/pom.xml) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] MonsterChenzhuo commented on pull request #3874: [Bug] [connector][kafka] json deserialize exception log no content #3873
MonsterChenzhuo commented on PR #3874: URL: https://github.com/apache/incubator-seatunnel/pull/3874#issuecomment-1371963032 https://user-images.githubusercontent.com/60029759/210744144-0223d53f-2154-4baa-b293-90b79282a9b6.png";> https://user-images.githubusercontent.com/60029759/210744122-d59b94af-76fc-4640-93e1-e6d7d3a5f445.png";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] lhyundeadsoul commented on a diff in pull request #2832: [Feature][Connector-V2] add tdengine source
lhyundeadsoul commented on code in PR #2832:
URL:
https://github.com/apache/incubator-seatunnel/pull/2832#discussion_r1062260170
##
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-tdengine-e2e/src/test/java/org/apache/seatunnel/e2e/connector/tdengine/TDengineIT.java:
##
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.e2e.connector.tdengine;
+
+import static org.awaitility.Awaitility.given;
+
+import org.apache.seatunnel.e2e.common.TestResource;
+import org.apache.seatunnel.e2e.common.TestSuiteBase;
+import org.apache.seatunnel.e2e.common.container.TestContainer;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.TestTemplate;
+import org.testcontainers.containers.Container;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.containers.wait.strategy.HostPortWaitStrategy;
+import org.testcontainers.lifecycle.Startables;
+import org.testcontainers.utility.DockerLoggerFactory;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Stream;
+
+@Slf4j
+public class TDengineIT extends TestSuiteBase implements TestResource {
+private static final String DOCKER_IMAGE = "tdengine/tdengine:3.0.2.1";
+private static final String NETWORK_ALIASES1 = "flink_e2e_tdengine_src";
+private static final String NETWORK_ALIASES2 = "flink_e2e_tdengine_sink";
+private static final int PORT = 6041;
+
+private GenericContainer tdengineServer1;
+private GenericContainer tdengineServer2;
+private Connection connection1;
+private Connection connection2;
+private int testDataCount;
+
+@BeforeAll
+@Override
+public void startUp() throws Exception {
+tdengineServer1 = new GenericContainer<>(DOCKER_IMAGE)
+.withNetwork(NETWORK)
+.withNetworkAliases(NETWORK_ALIASES1)
+.withExposedPorts(PORT)
+.withLogConsumer(new
Slf4jLogConsumer(DockerLoggerFactory.getLogger(DOCKER_IMAGE)))
+.waitingFor(new HostPortWaitStrategy()
+.withStartupTimeout(Duration.ofMinutes(2)));
+tdengineServer2 = new GenericContainer<>(DOCKER_IMAGE)
Review Comment:
> Why need to start two nodes? Is it possible to start only one node
Sorry, A container can only create one database cuz `supportVnodes `, a
TDengine parameter, limits.
Is there any problem with starting two nodes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X opened a new pull request, #3875: [Improve] [Checkpoint] Fix CheckpointIDCounter Thread Not Safe Problem
Hisoka-X opened a new pull request, #3875: URL: https://github.com/apache/incubator-seatunnel/pull/3875 ## Purpose of this pull request ## Check list * [ ] Code changed are covered with tests, or it does not need tests for reason: * [ ] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md) * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs * [ ] If you are contributing the connector code, please check that the following files are updated: 1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/incubator-seatunnel/tree/dev/docs/en/connector-v2) 2. Update [plugin-mapping.properties](https://github.com/apache/incubator-seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it 3. Update the pom file of [seatunnel-dist](https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-dist/pom.xml) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #3819: [Improve][Connector-V2][MongoDB] Refactor MongoDB connector e2e test …
TyrantLucifer commented on PR #3819: URL: https://github.com/apache/incubator-seatunnel/pull/3819#issuecomment-1371992436 cc @ashulin @EricJoy2048 @hailin0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on pull request #3818: [Improve][seatunnel-engine-common] move serializer to common
CalvinKirs commented on PR #3818: URL: https://github.com/apache/incubator-seatunnel/pull/3818#issuecomment-1372015784  should delete this comment -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on pull request #3818: [Improve][seatunnel-engine-common] move serializer to common
CalvinKirs commented on PR #3818: URL: https://github.com/apache/incubator-seatunnel/pull/3818#issuecomment-1372020240 Btw, A better way is to serialize as a separate module, and we may support other ways in the future, such as protobuf, json and more -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] sunnyzhu92 commented on issue #3869: The maxcompute source does not support the conversion of date type
sunnyzhu92 commented on issue #3869: URL: https://github.com/apache/incubator-seatunnel/issues/3869#issuecomment-1372022487 can you assign it to me? thanks a lot -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] rhh777 opened a new pull request, #3877: [Improve][Connector-V2][Redis] RedisSource support scan operation
rhh777 opened a new pull request, #3877: URL: https://github.com/apache/incubator-seatunnel/pull/3877 Scan command allow for incremental iteration, returning only a small number of elements per call,. KEYS command may block the server for a long time when called against a lot of keys. ## Purpose of this pull request ## Check list * [ ] Code changed are covered with tests, or it does not need tests for reason: * [ ] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md) * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs * [ ] If you are contributing the connector code, please check that the following files are updated: 1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/incubator-seatunnel/tree/dev/docs/en/connector-v2) 2. Update [plugin-mapping.properties](https://github.com/apache/incubator-seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it 3. Update the pom file of [seatunnel-dist](https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-dist/pom.xml) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] TaoZex closed pull request #3848: [Feature][Connector V2] Add GoogleSheets Sink
TaoZex closed pull request #3848: [Feature][Connector V2] Add GoogleSheets Sink URL: https://github.com/apache/incubator-seatunnel/pull/3848 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #3877: [Improve][Connector-V2][Redis] RedisSource support scan operation
TyrantLucifer commented on PR #3877: URL: https://github.com/apache/incubator-seatunnel/pull/3877#issuecomment-1372135075 It looks like a exciting feature. But it seems that has some problems in e2e test case. BTW, if you write a e2e test case to verify this feature is better. A tested PR will join the host branch more quickly. Thank you again for your contribution to SeaTunnel. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] rhh777 commented on pull request #3877: [Improve][Connector-V2][Redis] RedisSource support scan operation
rhh777 commented on PR #3877: URL: https://github.com/apache/incubator-seatunnel/pull/3877#issuecomment-1372162889 > It looks like a exciting feature. But it seems that has some problems in e2e test case. BTW, if you write a e2e test case to verify this feature is better. A tested PR will join the host branch more quickly. Thank you again for your contribution to SeaTunnel. I unit tested it locally, but I'm still learning e2e, so I'll fill in later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] lightzhao commented on pull request #3836: [Bug][KafkaSource]KafkaConsumer is not close.
lightzhao commented on PR #3836: URL: https://github.com/apache/incubator-seatunnel/pull/3836#issuecomment-1372216049 @TyrantLucifer please merge. thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] TaoZex closed pull request #3848: [Feature][Connector V2] Add GoogleSheets Sink
TaoZex closed pull request #3848: [Feature][Connector V2] Add GoogleSheets Sink URL: https://github.com/apache/incubator-seatunnel/pull/3848 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] TaoZex commented on pull request #3866: [Improve][Connector-V2][DataHub] Refactor DataHub connector e2e test
TaoZex commented on PR #3866: URL: https://github.com/apache/incubator-seatunnel/pull/3866#issuecomment-1372279371 @TyrantLucifer @Hisoka-X PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] ashulin commented on pull request #3867: [feature][cdc][postgres] add PostgreSQL CDC
ashulin commented on PR #3867: URL: https://github.com/apache/incubator-seatunnel/pull/3867#issuecomment-1372296179 > 这个可以现在使用吗，支持同步DDL吗 @ediconss Currently, DDL synchronization is not supported. PostgreSQL CDC is still in `beta` status and needs more testing. Feedback on your test results is welcome. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] Sachin-2206 commented on issue #3857: [Feature]Rename seatunnel-engine module
Sachin-2206 commented on issue #3857: URL: https://github.com/apache/incubator-seatunnel/issues/3857#issuecomment-1372357688 Sir I updated maven module names. Sir should I also update the class package names ?( seatunnel engine to seatunnel zeta ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on a diff in pull request #3877: [Improve][Connector-V2][Redis] RedisSource support scan operation
EricJoy2048 commented on code in PR #3877:
URL:
https://github.com/apache/incubator-seatunnel/pull/3877#discussion_r1062600816
##
seatunnel-connectors-v2/connector-redis/src/main/java/org/apache/seatunnel/connectors/seatunnel/redis/config/RedisConfig.java:
##
@@ -60,7 +61,19 @@ public enum HashKeyParseMode {
Options.key("keys")
.stringType()
.noDefaultValue()
-.withDescription("keys pattern, redis source connector support
fuzzy key matching, user needs to ensure that the matched keys are the same
type");
+.withDescription("keys pattern, redis source connector support
fuzzy key matching, if keys_type_check is false, user needs to ensure that the
matched keys are the same type");
+
+public static final Option KEY_TYPE_CHECK =
Review Comment:
Please update the document and add change log about this pr.
##
seatunnel-connectors-v2/connector-redis/src/main/java/org/apache/seatunnel/connectors/seatunnel/redis/config/RedisConfig.java:
##
@@ -60,7 +61,19 @@ public enum HashKeyParseMode {
Options.key("keys")
.stringType()
.noDefaultValue()
-.withDescription("keys pattern, redis source connector support
fuzzy key matching, user needs to ensure that the matched keys are the same
type");
+.withDescription("keys pattern, redis source connector support
fuzzy key matching, if keys_type_check is false, user needs to ensure that the
matched keys are the same type");
+
+public static final Option KEY_TYPE_CHECK =
Review Comment:
You need update the RedisSourceFactory too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-seatunnel] Sachin-2206 commented on issue #3857: [Feature]Rename seatunnel-engine module
Sachin-2206 commented on issue #3857: URL: https://github.com/apache/incubator-seatunnel/issues/3857#issuecomment-1372409172 Sir I want to know that should I change only package name or complete import statements names ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] Sachin-2206 commented on issue #3857: [Feature]Rename seatunnel-engine module
Sachin-2206 commented on issue #3857: URL: https://github.com/apache/incubator-seatunnel/issues/3857#issuecomment-1372460377 Sir I completed my work . I want to send a pull request of it . Please check it and tell me whether my work is correct or not, -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #3591: [Feature] Support encryption configuration
github-actions[bot] commented on issue #3591: URL: https://github.com/apache/incubator-seatunnel/issues/3591#issuecomment-1372969193 This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #3431: [Feature][Metrics] SeaTunnel Metrics for Connector V2
github-actions[bot] commented on issue #3431: URL: https://github.com/apache/incubator-seatunnel/issues/3431#issuecomment-1372969235 This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] closed issue #3591: [Feature] Support encryption configuration
github-actions[bot] closed issue #3591: [Feature] Support encryption configuration URL: https://github.com/apache/incubator-seatunnel/issues/3591 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #3267: [Feature][starter][flink] Support transform-v2 for flink
github-actions[bot] commented on issue #3267: URL: https://github.com/apache/incubator-seatunnel/issues/3267#issuecomment-1372969260 This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #3222: [Bug][Translation][Flink] Convert BigDecimal type losing precsion
github-actions[bot] commented on issue #3222: URL: https://github.com/apache/incubator-seatunnel/issues/3222#issuecomment-1372969284 This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] closed issue #3222: [Bug][Translation][Flink] Convert BigDecimal type losing precsion
github-actions[bot] closed issue #3222: [Bug][Translation][Flink] Convert BigDecimal type losing precsion URL: https://github.com/apache/incubator-seatunnel/issues/3222 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer merged pull request #3862: [Docs]Add next version release note
TyrantLucifer merged PR #3862: URL: https://github.com/apache/incubator-seatunnel/pull/3862 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[incubator-seatunnel] branch dev updated: [Improve][Docs] Add next version release note (#3862)
This is an automated email from the ASF dual-hosted git repository. tyrantlucifer pushed a commit to branch dev in repository https://gitbox.apache.org/repos/asf/incubator-seatunnel.git The following commit(s) were added to refs/heads/dev by this push: new cef31fd9c [Improve][Docs] Add next version release note (#3862) cef31fd9c is described below commit cef31fd9cf90caeb1298c97aff68d5085db9ee2f Author: Kirs AuthorDate: Fri Jan 6 09:43:49 2023 +0800 [Improve][Docs] Add next version release note (#3862) And list the inspection items in the PR template --- .github/PULL_REQUEST_TEMPLATE.md | 3 ++- release-note.md | 50 2 files changed, 52 insertions(+), 1 deletion(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 736a8a517..ee1aace4a 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -29,4 +29,5 @@ Feel free to ping committers for the review! * [ ] If you are contributing the connector code, please check that the following files are updated: 1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/incubator-seatunnel/tree/dev/docs/en/connector-v2) 2. Update [plugin-mapping.properties](https://github.com/apache/incubator-seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it - 3. Update the pom file of [seatunnel-dist](https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-dist/pom.xml) \ No newline at end of file + 3. Update the pom file of [seatunnel-dist](https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-dist/pom.xml) +* [ ] Update the [`release-note`](https://github.com/apache/incubator-seatunnel/blob/dev/release-note.md). \ No newline at end of file diff --git a/release-note.md b/release-note.md new file mode 100644 index 0..a9ffc530c --- /dev/null +++ b/release-note.md @@ -0,0 +1,50 @@ +# Next-release + +## New Feature +### Zeta Engine +- [Script]Add support close engine instance shell +### Core +- [Starter][Flink]Support transform-v2 for flink #3396 +### Transformer +- [Spark] Support transform-v2 for spark (#3409) +- [ALL]Add FieldMapper Transform #3781 + +## Improves +### Connectors +- [CDC]Add mysql-cdc source factory #3791 +- [JDBC]Fix the problem that the exception cannot be thrown normally #3796 +- [JDBC]Remove unused options for jdbc source factory #3794 +- [JDBC]Add exactly-once for JDBC source connector #3750 +- [JDBC]Fix JDBC source xa transaction commit failure on pipeline restore #3809 +- [JDBC]Improve option check rule +- [JDBC]Support SAP HANA. (#3017) +- [MongoDB]Add source query capability #3697 +- [File]Fix file source connector option rule bug #3804 +- [File]Add lzo compression way +- [Kafka]Fix Source failed to parse offset format #3810 +- [Kafka]Fix source the default value of commit_on_checkpoint #3831 +- [Kafka & RabbitMQ & StarRocks & ClickHouse]Change Connector Custom Config Prefix To Map #3719 +- [Common]The log outputs detailed exception stack information #3805 +- [API]Add parallelism and column projection interface #3829 +- [API]Add get source method to all source connector #3846 +### Zeta Engine +- [Chore] Remove unnecessary dependencies #3795 +- [Core] Improve job restart of all node down #3784 +- [Checkpoint] Cancel CheckpointCoordinator First Before Cancel Task #3838 +- [Storage] Remove seatunnel-api from engine storage. #3834 + +## Bug Fixes +### Connectors +- [ClickHouse File] Fix ClickhouseFile Committer Serializable Problems #3803 + +### Zeta Engine +- [Checkpoint] Fix Checkpoint Continue Trigger After Job CANCELED #3808 +- [Core]Fix Local Mode can't deserialize split (#3817) +- [Metrics] Fix Metrics will lose when Job be canceled. #3797 + + +## Test +### E2E +- [SqlServer CDC] fix SqlServerCDC IT failure #3807 + +
[GitHub] [incubator-seatunnel] TyrantLucifer closed issue #3267: [Feature][starter][flink] Support transform-v2 for flink
TyrantLucifer closed issue #3267: [Feature][starter][flink] Support transform-v2 for flink URL: https://github.com/apache/incubator-seatunnel/issues/3267 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on issue #3267: [Feature][starter][flink] Support transform-v2 for flink
TyrantLucifer commented on issue #3267: URL: https://github.com/apache/incubator-seatunnel/issues/3267#issuecomment-1373037758 closed because of #3396 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on pull request #3771: [Feature][SeaTunnel Engine] Add a jvm file to define the SeaTunnel En…
Hisoka-X commented on PR #3771: URL: https://github.com/apache/incubator-seatunnel/pull/3771#issuecomment-1373046709 Hi, e2e same like have some problem.  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] liugddx commented on pull request #3818: [Improve][seatunnel-engine-common] move serializer to common
liugddx commented on PR #3818: URL: https://github.com/apache/incubator-seatunnel/pull/3818#issuecomment-1373055740 > Btw, A better way is to serialize as a separate module, and we may support other ways in the future, such as protobuf, json and more Done. PTAL thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] lhyundeadsoul commented on pull request #2832: [Feature][Connector-V2] add tdengine source
lhyundeadsoul commented on PR #2832: URL: https://github.com/apache/incubator-seatunnel/pull/2832#issuecomment-1373062196 hey guys, Thank you for all your suggestions. All checks have passed now. Could you help me to merge? @CalvinKirs @hailin0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[incubator-seatunnel] branch dev updated (cef31fd9c -> ce85a8c68)
This is an automated email from the ASF dual-hosted git repository. wanghailin pushed a change to branch dev in repository https://gitbox.apache.org/repos/asf/incubator-seatunnel.git from cef31fd9c [Improve][Docs] Add next version release note (#3862) add ce85a8c68 Fix Source Class Support Parallelism judge & Add UT for it (#3878) No new revisions were added by this update. Summary of changes: .../java/org/apache/seatunnel/api/table/factory/FactoryUtil.java | 2 +- .../seatunnel/connectors/seatunnel/jdbc/JdbcFactoryTest.java | 8 +++- 2 files changed, 8 insertions(+), 2 deletions(-)
[GitHub] [incubator-seatunnel] hailin0 merged pull request #3878: [Hotfix][API] Fix Source Class Support Parallelism judge & Add UT for it
hailin0 merged PR #3878: URL: https://github.com/apache/incubator-seatunnel/pull/3878 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 merged pull request #3864: [Improve][JDBC] improve jdbc sink option
EricJoy2048 merged PR #3864: URL: https://github.com/apache/incubator-seatunnel/pull/3864 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[incubator-seatunnel] branch dev updated: [Improve][JDBC] improve jdbc sink option (#3864)
This is an automated email from the ASF dual-hosted git repository.
gaojun2048 pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/incubator-seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 768a9300e [Improve][JDBC] improve jdbc sink option (#3864)
768a9300e is described below
commit 768a9300e8f6f01e4e04a27f684ce6ca15265639
Author: Eric
AuthorDate: Fri Jan 6 11:01:16 2023 +0800
[Improve][JDBC] improve jdbc sink option (#3864)
* improve jdbc sink option
* improve option default get
---
docs/en/connector-v2/sink/Jdbc.md | 2 +-
.../seatunnel/jdbc/config/JdbcConfig.java | 11 -
.../internal/options/JdbcConnectionOptions.java| 27 --
3 files changed, 16 insertions(+), 24 deletions(-)
diff --git a/docs/en/connector-v2/sink/Jdbc.md
b/docs/en/connector-v2/sink/Jdbc.md
index ec31ce87d..c01b173b2 100644
--- a/docs/en/connector-v2/sink/Jdbc.md
+++ b/docs/en/connector-v2/sink/Jdbc.md
@@ -38,7 +38,7 @@ support `Xa transactions`. You can set `is_exactly_once=true`
to enable it.
| support_upsert_by_query_primary_key_exist | Boolean | No | false
|
| connection_check_timeout_sec | Int | No | 30
|
| max_retries | Int | No | 3
|
-| batch_size| Int | No | 300
|
+| batch_size| Int | No | 1000
|
| batch_interval_ms | Int | No | 1000
|
| is_exactly_once | Boolean | No | false
|
| xa_data_source_class_name | String | No | -
|
diff --git
a/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/config/JdbcConfig.java
b/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/config/JdbcConfig.java
index b1d395abb..72efa6fc8 100644
---
a/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/config/JdbcConfig.java
+++
b/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/config/JdbcConfig.java
@@ -26,6 +26,7 @@ import org.apache.seatunnel.shade.com.typesafe.config.Config;
import java.io.Serializable;
import java.util.List;
+@SuppressWarnings("checkstyle:MagicNumber")
public class JdbcConfig implements Serializable {
private static final int DEFAULT_CONNECTION_CHECK_TIMEOUT_SEC = 30;
private static final boolean DEFAULT_AUTO_COMMIT = true;
@@ -46,22 +47,20 @@ public class JdbcConfig implements Serializable {
public static final Option AUTO_COMMIT =
Options.key("auto_commit").booleanType().defaultValue(DEFAULT_AUTO_COMMIT).withDescription("auto
commit");
-public static final Option BATCH_SIZE =
Options.key("batch_size").intType().noDefaultValue().withDescription("batch
size");
+public static final Option BATCH_SIZE =
Options.key("batch_size").intType().defaultValue(1000).withDescription("batch
size");
public static final Option FETCH_SIZE =
Options.key("fetch_size").intType().defaultValue(0).withDescription("For
queries that return a large number of objects, " +
"you can configure the row fetch size used in the query to improve
performance by reducing the number database hits required to satisfy the
selection criteria. Zero means use jdbc default value.");
-public static final Option BATCH_INTERVAL_MS =
Options.key("batch_interval_ms").intType().noDefaultValue().withDescription("batch
interval milliSecond");
-
+public static final Option BATCH_INTERVAL_MS =
Options.key("batch_interval_ms").intType().defaultValue(1000).withDescription("batch
interval milliSecond");
public static final Option IS_EXACTLY_ONCE =
Options.key("is_exactly_once").booleanType().defaultValue(true).withDescription("exactly
once");
public static final Option XA_DATA_SOURCE_CLASS_NAME =
Options.key("xa_data_source_class_name").stringType().noDefaultValue().withDescription("data
source class name");
+public static final Option MAX_COMMIT_ATTEMPTS =
Options.key("max_commit_attempts").intType().defaultValue(3).withDescription("max
commit attempts");
-public static final Option MAX_COMMIT_ATTEMPTS =
Options.key("max_commit_attempts").stringType().noDefaultValue().withDescription("max
commit attempts");
-
-public static final Option TRANSACTION_TIMEOUT_SEC =
Options.key("transaction_timeout_sec").stringType().noDefaultValue().withDescription("transaction
timeout (second)");
+public static final Option TRANSACTION_TIMEOUT_SEC =
Options.key("transaction_timeout_sec").intType().defaultValue(-1).withDescription("transaction
timeout (second)");
public static final Option TABLE =
Op
[GitHub] [incubator-seatunnel] lightzhao commented on pull request #3854: [Improve][connector-v2][FileSink] Allow the user to set the row delimiter as an empty string
lightzhao commented on PR #3854: URL: https://github.com/apache/incubator-seatunnel/pull/3854#issuecomment-1373154304 @TaoZex please review, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] cxy-java opened a new issue, #3879: [Bug] [Module Name] Bug title
cxy-java opened a new issue, #3879:
URL: https://github.com/apache/incubator-seatunnel/issues/3879
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22)
and found no similar issues.
### What happened
an error occurred when I used the seatunnel engine
### SeaTunnel Version
3.2.0
### SeaTunnel Config
```conf
Jdbc {
url = "jdbc:mysql://localhost/test?serverTimezone=GMT%2b8"
driver = "com.mysql.cj.jdbc.Driver"
connection_check_timeout_sec = 100
user = "root"
password = "123456"
query = "select * from user"
}
```
### Running Command
```shell
bin/seatunnel.sh --config config/jdbc_to_console.conf -e local
```
### Error Exception
```log
Exception
StackTrace:org.apache.seatunnel.connectors.seatunnel.jdbc.exception.JdbcConnectorException:
ErrorCode:[JDBC-06], ErrorDescription:[No suitable dialect factory found] -
Could not find any jdbc dialect factories that implement
'org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.JdbcDialectFactory'
in the classpath
```
### Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 opened a new issue, #3880: [Improve] [Doc] Improve OSSFile Connector title
EricJoy2048 opened a new issue, #3880: URL: https://github.com/apache/incubator-seatunnel/issues/3880 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened update `OssFile ` to `OssFile[Jindo] ` is better. https://user-images.githubusercontent.com/32193458/210947010-76f59da1-b717-42e1-bd0b-498558273af1.png";> ### SeaTunnel Version dev ### SeaTunnel Config ```conf * ``` ### Running Command ```shell * ``` ### Error Exception ```log * ``` ### Flink or Spark Version _No response_ ### Java or Scala Version _No response_ ### Screenshots _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #2832: [Feature][Connector-V2] add tdengine source
TyrantLucifer commented on PR #2832: URL: https://github.com/apache/incubator-seatunnel/pull/2832#issuecomment-1373219560 > @EricJoy2048 @hailin0 @Hisoka-X @TyrantLucifer @CalvinKirs Hey guys, please help to give me a CR if you are free. Thx! Hi @lhyundeadsoul , overall LGTM but a little nit, you should unified the exception in connectors and update user-quick-manaul. You can refer to https://github.com/apache/incubator-seatunnel/issues/2930 see more details. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on a diff in pull request #2832: [Feature][Connector-V2] add tdengine source
TyrantLucifer commented on code in PR #2832:
URL:
https://github.com/apache/incubator-seatunnel/pull/2832#discussion_r1063191401
##
seatunnel-connectors-v2/connector-common/src/main/java/org/apache/seatunnel/connectors/seatunnel/common/schema/SeaTunnelSchema.java:
##
@@ -103,7 +103,7 @@ private static int[] parseDecimalPS(String type) {
return new int[]{precision, scale};
}
-private static SeaTunnelDataType parseTypeByString(String type) {
+public static SeaTunnelDataType parseTypeByString(String type) {
Review Comment:
Revert?
##
seatunnel-connectors-v2/connector-tdengine/src/main/java/org/apache/seatunnel/connectors/seatunnel/tdengine/sink/TDengineSinkWriter.java:
##
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.tdengine.sink;
+
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
+import
org.apache.seatunnel.connectors.seatunnel.common.sink.AbstractSinkWriter;
+import
org.apache.seatunnel.connectors.seatunnel.tdengine.config.TDengineSourceConfig;
+
+import org.apache.seatunnel.shade.com.typesafe.config.Config;
+
+import com.google.common.base.Throwables;
+import com.google.common.collect.Lists;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.StringUtils;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.time.LocalDateTime;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Objects;
+
+@Slf4j
+public class TDengineSinkWriter extends AbstractSinkWriter
{
+
+private static final DateTimeFormatter FORMATTER =
DateTimeFormatter.ofPattern("-MM-dd HH:mm:ss.SSS");
+private final Connection conn;
+private final TDengineSourceConfig config;
+private int tagsNum;
+
+@SneakyThrows
+public TDengineSinkWriter(Config pluginConfig, SeaTunnelRowType
seaTunnelRowType) {
+config = TDengineSourceConfig.buildSourceConfig(pluginConfig);
+String jdbcUrl = StringUtils.join(config.getUrl(),
config.getDatabase(), "?user=", config.getUsername(), "&password=",
config.getPassword());
+conn = DriverManager.getConnection(jdbcUrl);
+try (Statement statement = conn.createStatement()) {
+final ResultSet metaResultSet = statement.executeQuery("desc " +
config.getDatabase() + "." + config.getStable());
+while (metaResultSet.next()) {
+if (StringUtils.equals("TAG",
metaResultSet.getString("note"))) {
+tagsNum++;
+}
+}
+}
+}
+
+@SneakyThrows
+@Override
+@SuppressWarnings("checkstyle:RegexpSingleline")
+public void write(SeaTunnelRow element) {
+final ArrayList tags = Lists.newArrayList();
+for (int i = element.getArity() - tagsNum; i < element.getArity();
i++) {
+tags.add(element.getField(i));
+}
+final String tagValues =
StringUtils.join(convertDataType(tags.toArray()), ",");
+
+final Object[] metrics = ArrayUtils.subarray(element.getFields(), 1,
element.getArity() - tagsNum);
+
+try (Statement statement =
conn.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY)) {
+String sql = "INSERT INTO " + element.getField(0)
Review Comment:
String.format is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer merged pull request #3859: Update Connector release status
TyrantLucifer merged PR #3859: URL: https://github.com/apache/incubator-seatunnel/pull/3859 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[incubator-seatunnel] branch dev updated: [Improve][Docs] Update Connector release status (#3859)
This is an automated email from the ASF dual-hosted git repository. tyrantlucifer pushed a commit to branch dev in repository https://gitbox.apache.org/repos/asf/incubator-seatunnel.git The following commit(s) were added to refs/heads/dev by this push: new 3cdf3f59c [Improve][Docs] Update Connector release status (#3859) 3cdf3f59c is described below commit 3cdf3f59c137bbdd1e4c382500bcfc49863c8d6e Author: Kirs AuthorDate: Fri Jan 6 15:12:58 2023 +0800 [Improve][Docs] Update Connector release status (#3859) * Update Connector release status After production testing and feedback from community users, we have updated the health status of some Connectors * set Cassandra status to beta --- docs/en/Connector-v2-release-state.md | 93 +-- 1 file changed, 56 insertions(+), 37 deletions(-) diff --git a/docs/en/Connector-v2-release-state.md b/docs/en/Connector-v2-release-state.md index 8611a7429..bd6cd940f 100644 --- a/docs/en/Connector-v2-release-state.md +++ b/docs/en/Connector-v2-release-state.md @@ -11,55 +11,74 @@ SeaTunnel uses a grading system for connectors to help you understand what to ex | Connector Name | Type | Status | Support Version | |-|||-| +| [AmazonDynamoDB](connector-v2/sink/AmazonDynamoDB.md) | Sink | Beta | 2.3.0 | +| [AmazonDynamoDB](connector-v2/source/AmazonDynamoDB.md) | Source | Beta | 2.3.0 | | [Asset](connector-v2/sink/Assert.md)| Sink | Beta | 2.2.0-beta | -| [ClickHouse](connector-v2/source/Clickhouse.md) | Source | Beta | 2.2.0-beta | -| [ClickHouse](connector-v2/sink/Clickhouse.md) | Sink | Beta | 2.2.0-beta | -| [ClickHouseFile](connector-v2/sink/ClickhouseFile.md) | Sink | Beta | 2.2.0-beta | -| [Console](connector-v2/sink/Console.md) | Sink | Beta | 2.2.0-beta | +| [Cassandra](connector-v2/sink/Cassandra.md) | Sink | Beta | 2.3.0 | +| [Cassandra](connector-v2/source/Cassandra.md) | Source | Beta | 2.3.0 | +| [ClickHouse](connector-v2/source/Clickhouse.md) | Source | GA | 2.2.0-beta | +| [ClickHouse](connector-v2/sink/Clickhouse.md) | Sink | GA | 2.2.0-beta | +| [ClickHouseFile](connector-v2/sink/ClickhouseFile.md) | Sink | GA | 2.2.0-beta | +| [Console](connector-v2/sink/Console.md) | Sink | GA | 2.2.0-beta | | [DataHub](connector-v2/sink/Datahub.md) | Sink | Alpha | 2.2.0-beta | +| [Doris](connector-v2/sink/Doris.md) | Sink | Beta | 2.3.0 | | [DingTalk](connector-v2/sink/DingTalk.md) | Sink | Alpha | 2.2.0-beta | -| [Elasticsearch](connector-v2/sink/Elasticsearch.md) | Sink | Beta | 2.2.0-beta | +| [Elasticsearch](connector-v2/sink/Elasticsearch.md) | Sink | GA | 2.2.0-beta | | [Email](connector-v2/sink/Email.md) | Sink | Alpha | 2.2.0-beta | | [Enterprise WeChat](connector-v2/sink/Enterprise-WeChat.md) | Sink | Alpha | 2.2.0-beta | | [FeiShu](connector-v2/sink/Feishu.md) | Sink | Alpha | 2.2.0-beta | -| [Fake](connector-v2/source/FakeSource.md) | Source | Beta | 2.2.0-beta | -| [FtpFile](connector-v2/sink/FtpFile.md) | Sink | Alpha | 2.2.0-beta | -| [Greenplum](connector-v2/sink/Greenplum.md) | Sink | Alpha | 2.2.0-beta | -| [Greenplum](connector-v2/source/Greenplum.md) | Source | Alpha | 2.2.0-beta | -| [HdfsFile](connector-v2/sink/HdfsFile.md) | Sink | Beta | 2.2.0-beta | -| [HdfsFile](connector-v2/source/HdfsFile.md) | Source | Beta | 2.2.0-beta | -| [Hive](connector-v2/sink/Hive.md) | Sink | Beta | 2.2.0-beta | -| [Hive](connector-v2/source/Hive.md) | Source | Beta | 2.2.0-beta | +| [Fake](connector-v2/source/FakeSource.md) | Source | GA | 2.2.0-beta | +| [FtpFile](connector-v2/sink/FtpFile.md) | Sink | Beta | 2.2.0-beta | +| [Greenplum](connector-v2/sink/Greenplum.md) | Sink | Beta | 2.2.0-beta | +| [Greenplum](connector-v2/source/Greenplum.md) | Source | Beta | 2.2.0-beta | +| [HdfsFile](connector-v2/sink/HdfsFile.md) | Sink | GA | 2.2.0-beta | +| [HdfsFile](connector-v2/source/HdfsFile.md) | Source | GA | 2.2.0-beta | +| [Hive](connector-v2/sink/Hive.md)
[GitHub] [incubator-seatunnel] TaoZex commented on pull request #3848: [Feature][Connector V2] Add GoogleSheets Sink
TaoZex commented on PR #3848: URL: https://github.com/apache/incubator-seatunnel/pull/3848#issuecomment-1373245323 @Hisoka-X @EricJoy2048 PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] MonsterChenzhuo commented on pull request #3874: [Bug] [connector][kafka] json deserialize exception log no content #3873
MonsterChenzhuo commented on PR #3874: URL: https://github.com/apache/incubator-seatunnel/pull/3874#issuecomment-1373276967 cc @hailin0 @Hisoka-X PTAL, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-seatunnel] Carl-Zhou-CN opened a new pull request, #3881: [Improve][Connector-V2][clickhouse] Special characters in column names are supported
Carl-Zhou-CN opened a new pull request, #3881: URL: https://github.com/apache/incubator-seatunnel/pull/3881 …s are supported ## Purpose of this pull request ## Check list * [ ] Code changed are covered with tests, or it does not need tests for reason: * [ ] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md) * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs * [ ] If you are contributing the connector code, please check that the following files are updated: 1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/incubator-seatunnel/tree/dev/docs/en/connector-v2) 2. Update [plugin-mapping.properties](https://github.com/apache/incubator-seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it 3. Update the pom file of [seatunnel-dist](https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-dist/pom.xml) * [ ] Update the [`release-note`](https://github.com/apache/incubator-seatunnel/blob/dev/release-note.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
