[seatunnel-website] branch main updated: updata logo (#258)
This is an automated email from the ASF dual-hosted git repository.
gaojun2048 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/seatunnel-website.git
The following commit(s) were added to refs/heads/main by this push:
new 19a37072cdc updata logo (#258)
19a37072cdc is described below
commit 19a37072cdc6047efdb5d8b6276c39608c698151
Author: lifeng <53861852+nielif...@users.noreply.github.com>
AuthorDate: Tue Jul 11 15:05:41 2023 +0800
updata logo (#258)
---
src/pages/user/images.json | 2 +-
static/user/shuguanguyun.jpg | Bin 0 -> 9391 bytes
static/user/yixin.png| Bin 8825 -> 0 bytes
3 files changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/pages/user/images.json b/src/pages/user/images.json
index 49beec66aba..4601f92cee7 100644
--- a/src/pages/user/images.json
+++ b/src/pages/user/images.json
@@ -17,7 +17,7 @@
{"url":"huya.png"},
{"url":"qunar.png"},
{"url":"anchnet.png"},
-{"url":"yixin.png"},
+{"url":"shuguanguyun.jpg"},
{"url":"GridSum.png"},
{"url":"数字广东.png"},
{"url":"guanyuzuiyou.png"},
diff --git a/static/user/shuguanguyun.jpg b/static/user/shuguanguyun.jpg
new file mode 100644
index 000..561fb9b406f
Binary files /dev/null and b/static/user/shuguanguyun.jpg differ
diff --git a/static/user/yixin.png b/static/user/yixin.png
deleted file mode 100644
index 171147927f1..000
Binary files a/static/user/yixin.png and /dev/null differ
[GitHub] [seatunnel-website] EricJoy2048 merged pull request #258: updata logo
EricJoy2048 merged PR #258: URL: https://github.com/apache/seatunnel-website/pull/258 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] liugddx closed pull request #4976: [Feature][Connector] support avro format
liugddx closed pull request #4976: [Feature][Connector] support avro format URL: https://github.com/apache/seatunnel/pull/4976 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] hailin0 merged pull request #5035: [BugFix] [Connector-V2] [MySQL-CDC] serverId from int to long (#5033)
hailin0 merged PR #5035: URL: https://github.com/apache/seatunnel/pull/5035 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[seatunnel] branch dev updated: [BugFix] [Connector-V2] [MySQL-CDC] serverId from int to long (#5033) (#5035)
This is an automated email from the ASF dual-hosted git repository.
wanghailin pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 4abc80e11 [BugFix] [Connector-V2] [MySQL-CDC] serverId from int to
long (#5033) (#5035)
4abc80e11 is described below
commit 4abc80e1116a1a742e7d161a7fd367690d443c4f
Author: 司马琦昂
AuthorDate: Tue Jul 11 15:48:45 2023 +0800
[BugFix] [Connector-V2] [MySQL-CDC] serverId from int to long (#5033)
(#5035)
* [bugfix] change MySQL CDC serverId from int to long (#5033)
* style: 🎨 optimize code style
---
.../cdc/mysql/config/MySqlSourceConfigFactory.java | 2 +-
.../seatunnel/cdc/mysql/config/ServerIdRange.java | 24 +++---
2 files changed, 13 insertions(+), 13 deletions(-)
diff --git
a/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/config/MySqlSourceConfigFactory.java
b/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/config/MySqlSourceConfigFactory.java
index ef697f2e1..7317b040f 100644
---
a/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/config/MySqlSourceConfigFactory.java
+++
b/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/config/MySqlSourceConfigFactory.java
@@ -89,7 +89,7 @@ public class MySqlSourceConfigFactory extends
JdbcSourceConfigFactory {
if (serverIdRange != null) {
props.setProperty("database.server.id.range",
String.valueOf(serverIdRange));
-int serverId = serverIdRange.getServerId(subtaskId);
+long serverId = serverIdRange.getServerId(subtaskId);
props.setProperty("database.server.id", String.valueOf(serverId));
}
if (databaseList != null) {
diff --git
a/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/config/ServerIdRange.java
b/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/config/ServerIdRange.java
index 55ac9c14b..c3319f8a8 100644
---
a/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/config/ServerIdRange.java
+++
b/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/config/ServerIdRange.java
@@ -32,27 +32,27 @@ public class ServerIdRange implements Serializable {
private static final long serialVersionUID = 1L;
/** Start of the range (inclusive). */
-private final int startServerId;
+private final long startServerId;
/** End of the range (inclusive). */
-private final int endServerId;
+private final long endServerId;
-public ServerIdRange(int startServerId, int endServerId) {
+public ServerIdRange(long startServerId, long endServerId) {
this.startServerId = startServerId;
this.endServerId = endServerId;
}
-public int getStartServerId() {
+public long getStartServerId() {
return startServerId;
}
-public int getEndServerId() {
+public long getEndServerId() {
return endServerId;
}
-public int getServerId(int subTaskId) {
+public long getServerId(int subTaskId) {
checkArgument(subTaskId >= 0, "Subtask ID %s shouldn't be a negative
number.", subTaskId);
-if (subTaskId > getNumberOfServerIds()) {
+if ((long) subTaskId > getNumberOfServerIds()) {
throw new IllegalArgumentException(
String.format(
"Subtask ID %s is out of server id range %s, "
@@ -64,8 +64,8 @@ public class ServerIdRange implements Serializable {
return startServerId + subTaskId;
}
-public int getNumberOfServerIds() {
-return endServerId - startServerId + 1;
+public long getNumberOfServerIds() {
+return endServerId - startServerId + 1L;
}
@Override
@@ -96,14 +96,14 @@ public class ServerIdRange implements Serializable {
return new ServerIdRange(
parseServerId(idArray[0].trim()),
parseServerId(idArray[1].trim()));
} else {
-int serverId = parseServerId(range);
+long serverId = parseServerId(range);
return new ServerIdRange(serverId, serverId);
}
}
-private static int parseServerId(String serverIdValue) {
+private static long parseServerId(String serverIdValue) {
try {
-return Integer.parseInt(serverIdValue);
+return Long.parseLong(serverIdValue);
} catch
[GitHub] [seatunnel] EricJoy2048 commented on a diff in pull request #5044: [Feature] make data block balance before importing data #5026
EricJoy2048 commented on code in PR #5044: URL: https://github.com/apache/seatunnel/pull/5044#discussion_r1259365835 ## docs/en/connector-v2/sink/common-options.md: ## @@ -19,6 +20,16 @@ When `parallelism` is not specified, the `parallelism` in env is used by default When parallelism is specified, it will override the parallelism in env. +### partition_balance [boolean] Review Comment: Please add comment that this option only supported by Flink/Spark. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] EricJoy2048 merged pull request #5030: [Feature][Connector-V2][cdc] Change the time zone to the default time zone
EricJoy2048 merged PR #5030: URL: https://github.com/apache/seatunnel/pull/5030 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[seatunnel] branch dev updated: [Feature][Connector-V2][cdc] Change the time zone to the default time zone (#5030)
This is an automated email from the ASF dual-hosted git repository.
gaojun2048 pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 3cff923a7 [Feature][Connector-V2][cdc] Change the time zone to the
default time zone (#5030)
3cff923a7 is described below
commit 3cff923a7954977ddea3b4b16fec50659e4a238f
Author: Carl-Zhou-CN <67902676+carl-zhou...@users.noreply.github.com>
AuthorDate: Tue Jul 11 16:23:36 2023 +0800
[Feature][Connector-V2][cdc] Change the time zone to the default time zone
(#5030)
---
docs/en/connector-v2/source/MySQL-CDC.md | 2 +-
.../seatunnel/connectors/cdc/base/option/JdbcSourceOptions.java| 7 +--
.../cdc/debezium/row/SeaTunnelRowDebeziumDeserializeSchema.java| 2 +-
3 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/docs/en/connector-v2/source/MySQL-CDC.md
b/docs/en/connector-v2/source/MySQL-CDC.md
index f26c1e60a..caeeca062 100644
--- a/docs/en/connector-v2/source/MySQL-CDC.md
+++ b/docs/en/connector-v2/source/MySQL-CDC.md
@@ -155,7 +155,7 @@ By default, a random number is generated between 5400 and
6400, though we recomm
### server-time-zone [String]
-The session time zone in database server.
+The session time zone in database server. If not set, then
ZoneId.systemDefault() is used to determine the server time zone.
### connect.timeout.ms [long]
diff --git
a/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/option/JdbcSourceOptions.java
b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/option/JdbcSourceOptions.java
index ea64cd3d2..715915c74 100644
---
a/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/option/JdbcSourceOptions.java
+++
b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/option/JdbcSourceOptions.java
@@ -21,6 +21,7 @@ import org.apache.seatunnel.api.configuration.Option;
import org.apache.seatunnel.api.configuration.Options;
import org.apache.seatunnel.connectors.cdc.base.source.IncrementalSource;
+import java.time.ZoneId;
import java.util.List;
/** Configurations for {@link IncrementalSource} of JDBC data source. */
@@ -61,8 +62,10 @@ public class JdbcSourceOptions extends SourceOptions {
public static final Option SERVER_TIME_ZONE =
Options.key("server-time-zone")
.stringType()
-.defaultValue("UTC")
-.withDescription("The session time zone in database
server.");
+.defaultValue(ZoneId.systemDefault().getId())
+.withDescription(
+"The session time zone in database server."
++ "If not set, then ZoneId.systemDefault()
is used to determine the server time zone");
public static final Option SERVER_ID =
Options.key("server-id")
diff --git
a/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/row/SeaTunnelRowDebeziumDeserializeSchema.java
b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/row/SeaTunnelRowDebeziumDeserializeSchema.java
index d9cdf2f50..2997fc78d 100644
---
a/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/row/SeaTunnelRowDebeziumDeserializeSchema.java
+++
b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/row/SeaTunnelRowDebeziumDeserializeSchema.java
@@ -211,7 +211,7 @@ public final class SeaTunnelRowDebeziumDeserializeSchema
private SeaTunnelDataType resultTypeInfo;
private MetadataConverter[] metadataConverters = new
MetadataConverter[0];
private ValueValidator validator = (rowData, rowKind) -> {};
-private ZoneId serverTimeZone = ZoneId.of("UTC");
+private ZoneId serverTimeZone = ZoneId.systemDefault();
private DebeziumDeserializationConverterFactory
userDefinedConverterFactory =
DebeziumDeserializationConverterFactory.DEFAULT;
[GitHub] [seatunnel] ddna1021 commented on a diff in pull request #5044: [Feature] make data block balance before importing data #5026
ddna1021 commented on code in PR #5044: URL: https://github.com/apache/seatunnel/pull/5044#discussion_r1259387052 ## docs/en/connector-v2/sink/common-options.md: ## @@ -19,6 +20,16 @@ When `parallelism` is not specified, the `parallelism` in env is used by default When parallelism is specified, it will override the parallelism in env. +### partition_balance [boolean] Review Comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] EricJoy2048 merged pull request #5011: [Bugfix][connector-cdc-mysql] Fix listener not released when BinlogClient reuse
EricJoy2048 merged PR #5011: URL: https://github.com/apache/seatunnel/pull/5011 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[seatunnel] branch dev updated: [Bugfix][connector-cdc-mysql] Fix listener not released when BinlogClient reuse (#5011)
This is an automated email from the ASF dual-hosted git repository.
gaojun2048 pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 3287b1d85 [Bugfix][connector-cdc-mysql] Fix listener not released when
BinlogClient reuse (#5011)
3287b1d85 is described below
commit 3287b1d8521b9fda08fa4774cf958071031ac313
Author: happyboy1024 <137260654+happyboy1...@users.noreply.github.com>
AuthorDate: Tue Jul 11 16:28:01 2023 +0800
[Bugfix][connector-cdc-mysql] Fix listener not released when BinlogClient
reuse (#5011)
---
.../reader/fetch/MySqlSourceFetchTaskContext.java | 18 +-
1 file changed, 17 insertions(+), 1 deletion(-)
diff --git
a/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/source/reader/fetch/MySqlSourceFetchTaskContext.java
b/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/source/reader/fetch/MySqlSourceFetchTaskContext.java
index c6aebc8aa..0b6ea40ea 100644
---
a/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/source/reader/fetch/MySqlSourceFetchTaskContext.java
+++
b/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/source/reader/fetch/MySqlSourceFetchTaskContext.java
@@ -18,6 +18,7 @@
package
org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.reader.fetch;
import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
+import org.apache.seatunnel.common.utils.ReflectionUtils;
import org.apache.seatunnel.connectors.cdc.base.config.JdbcSourceConfig;
import org.apache.seatunnel.connectors.cdc.base.dialect.JdbcDataSourceDialect;
import
org.apache.seatunnel.connectors.cdc.base.relational.JdbcSourceEventDispatcher;
@@ -70,6 +71,7 @@ import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import static
org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.offset.BinlogOffset.BINLOG_FILENAME_OFFSET_KEY;
import static
org.apache.seatunnel.connectors.seatunnel.cdc.mysql.utils.MySqlConnectionUtils.createBinaryClient;
@@ -326,13 +328,27 @@ public class MySqlSourceFetchTaskContext extends
JdbcSourceFetchTaskContext {
MySqlDatabaseSchema schema,
BinaryLogClient reusedBinaryLogClient) {
super(config, schema);
-this.reusedBinaryLogClient = reusedBinaryLogClient;
+this.reusedBinaryLogClient =
resetBinaryLogClient(reusedBinaryLogClient);
}
@Override
public BinaryLogClient getBinaryLogClient() {
return reusedBinaryLogClient;
}
+
+/** reset the listener of binaryLogClient before fetch task start. */
+private BinaryLogClient resetBinaryLogClient(BinaryLogClient
binaryLogClient) {
+Optional eventListenersField =
+ReflectionUtils.getField(
+binaryLogClient, BinaryLogClient.class,
"eventListeners");
+eventListenersField.ifPresent(o ->
((List) o).clear());
+Optional lifecycleListeners =
+ReflectionUtils.getField(
+binaryLogClient, BinaryLogClient.class,
"lifecycleListeners");
+lifecycleListeners.ifPresent(
+o -> ((List)
o).clear());
+return binaryLogClient;
+}
}
/** Copied from debezium for accessing here. */
[GitHub] [seatunnel] EricJoy2048 commented on a diff in pull request #5007: [Hotfix][Zeta] Fix wait job complete bug
EricJoy2048 commented on code in PR #5007:
URL: https://github.com/apache/seatunnel/pull/5007#discussion_r1259392309
##
seatunnel-engine/seatunnel-engine-client/src/main/java/org/apache/seatunnel/engine/client/job/ClientJobProxy.java:
##
@@ -93,21 +89,12 @@ private void submitJob(JobImmutableInformation
jobImmutableInformation) {
@Override
public JobStatus waitForJobComplete() {
try {
-jobResult =
-RetryUtils.retryWithException(
-() -> {
-PassiveCompletableFuture jobFuture =
-doWaitForJobComplete();
-return jobFuture.get();
-},
-new RetryUtils.RetryMaterial(
-10,
-true,
-exception ->
-
ExceptionUtil.isOperationNeedRetryException(exception),
-Constant.OPERATION_RETRY_SLEEP));
+PassiveCompletableFuture jobFuture =
doWaitForJobComplete();
+jobResult = jobFuture.get();
if (jobResult == null) {
-throw new SeaTunnelEngineException("failed to fetch job
result");
+LOGGER.severe(
+"Unable to obtain the status of the job, it may have
been running during the last cluster shutdown.");
Review Comment:
If the job can not get from running job and history job. It mean the job is
running when the cluster crash. So we can not get it from the history job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [seatunnel] ocean-zhc opened a new issue, #5060: [Bug] [kafka source] checkpoint 设置不生效
ocean-zhc opened a new issue, #5060:
URL: https://github.com/apache/seatunnel/issues/5060
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22)
and found no similar issues.
### What happened
kafka source 配置HDFS 的checkpoint检查点，不生效。
https://github.com/apache/seatunnel/assets/46189785/8774e129-9be0-4683-a72b-9221e2212af6";>
程序中断，在重新启动，插入mysql[primary_key]参数不设置的情况下，会重复，，看日志，offset会重复消费，从0开始。
### SeaTunnel Version
dev分支：2.3.2-SNAPSHOT
### SeaTunnel Config
```conf
env {
parallelism = 5
job.mode = "STREAMING"
job.name = "DZFP_DZDZ_DZDZ_FJXX_PTFP"
checkpoint.interval = "6"
checkpoint.data-uri = "hdfs://localhost:8020/flink-checKpoints"
}
source {
Kafka {
result_table_name = "DZFP_DZDZ_DZDZ_FJXX_PTFP1"
topic = "DZFP_DZDZ_DZDZ_FJXX_PTFP"
bootstrap.servers = "localhost:9200"
commit_on_checkpoint = "true"
format_error_handle_way = "skip"
schema = {
fields {
message = {
data = "map"
}
}
}
#semantics = EXACTLY_ONCE
start_mode = "earliest"
#start_mode.offsets = {
# 0 = 0
# 1 = 0
# 2 = 0
#}
kafka.config = {
auto.offset.reset = "earliest"
enable.auto.commit = "true"
# max.poll.interval.ms = 3000
#max.partition.fetch.bytes = "5242880"
#session.timeout.ms = "3"
request.timeout.ms = "6"
buffer.memory = "33554432"
#client.id = client_1
max.poll.records = "10"
}
}
}
transform {
sql{
source_table_name = "DZFP_DZDZ_DZDZ_FJXX_PTFP1"
result_table_name = "DZFP_DZDZ_DZDZ_FJXX_PTFP2"
# qdmx自定义UDF函数
query = "select qdmx(message,'xsfdzdh') as
xsfdzdh,qdmx(message,'gf_qxswjg_dm') as gf_qxswjg_dm,qdmx(message,'sblx_dm') as
sblx_dm,qdmx(message,'gmfdzdh') as gmfdzdh,qdmx(message,'hzxxb_bh') as
hzxxb_bh,qdmx(message,'mac') as mac,qdmx(message,'sjczbs') as
sjczbs,qdmx(message,'xsfyhzh') as xsfyhzh,qdmx(message,'cxswjg_dm') as
cxswjg_dm,qdmx(message,'fpbb_dm') as fpbb_dm,qdmx(message,'skm') as
skm,qdmx(message,'kpjh') as kpjh,qdmx(message,'kpr') as kpr,qdmx(message,'ydk')
as ydk,qdmx(message,'jsswjg_dm') as jsswjg_dm,qdmx(message,'ydfpbz') as
ydfpbz,qdmx(message,'skr') as skr,qdmx(message,'fpqm') as
fpqm,qdmx(message,'xf_qxswjg_dm') as xf_qxswjg_dm,qdmx(message,'jsr_mc') as
jsr_mc,qdmx(message,'xdjg') as xdjg,qdmx(message,'tslsh') as
tslsh,qdmx(message,'fpdm') as fpdm,qdmx(message,'cslx_dm') as
cslx_dm,qdmx(message,'sjtbsj') as sjtbsj,qdmx(message,'fpdmhm') as
fpdmhm,qdmx(message,'fhr') as fhr,qdmx(message,'gf_sjswjg_dm') as
gf_sjswjg_dm,qdmx(message,'fphm') as fphm,qdmx(mes
sage,'jsr_dm') as jsr_dm,qdmx(message,'zbxlh') as
zbxlh,qdmx(message,'bs_nsrsbh') as bs_nsrsbh,qdmx(message,'gf_dsswjg_dm') as
gf_dsswjg_dm,qdmx(message,'bdbz') as bdbz,qdmx(message,'xf_sjswjg_dm') as
xf_sjswjg_dm,qdmx(message,'data_category') as
data_category,qdmx(message,'gmfyhzh') as gmfyhzh,qdmx(message,'swjg_dm') as
swjg_dm,qdmx(message,'sjbb') as sjbb,qdmx(message,'tspz_dm') as
tspz_dm,qdmx(message,'wspzh') as wspzh,qdmx(message,'sbbh') as
sbbh,qdmx(message,'yqbz') as yqbz,qdmx(message,'ip') as ip,qdmx(message,'kpyf')
as kpyf,qdmx(message,'xf_dsswjg_dm') as xf_dsswjg_dm,qdmx(message,'sslkjly') as
sslkjly from DZFP_DZDZ_DZDZ_FJXX_PTFP1"
}
}
sink {
#Console {
#source_table_name = "DZFP_DZDZ_DZDZ_FJXX_PTFP2"
#}
jdbc {
source_table_name = "DZFP_DZDZ_DZDZ_FJXX_PTFP2"
url =
"jdbc:mysql://192.168.202.30:3306/sjjcb_dw?serverTimezone=GMT%2b8&rewriteBatchedStatements=true"
driver = "com.mysql.cj.jdbc.Driver"
user = "user"
password = "pwd"
batch_size = "20"
database = "dbname"
table = "tablename"
generate_sink_sql = true
#is_exactly_once = "true"
#xa_data_source_class_name = "com.mysql.cj.jdbc.MysqlXADataSource"
primary_keys = ["fpdmhm"]
}
}
```
### Running Command
```shell
./bin/seatunnel.sh --config config/qdmx_config/DZFP_DZDZ_DZDZ_FJXX_PTFP.conf
-e local
```
### Error Exception
```log
no
```
### Flink or Spark Version
no
### Java or Scala Version
_No response_
### Screenshots
1.8.0_172
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this projec
[GitHub] [seatunnel] wu-a-ge closed issue #5046: [Bug] [Zeta] The checkpoint manager did not delete the historical checkpoint file
wu-a-ge closed issue #5046: [Bug] [Zeta] The checkpoint manager did not delete the historical checkpoint file URL: https://github.com/apache/seatunnel/issues/5046 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] Hisoka-X merged pull request #4989: [Feature][Connector-V2][Jdbc] Add oceanbase dialect factory
Hisoka-X merged PR #4989: URL: https://github.com/apache/seatunnel/pull/4989 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[seatunnel] branch dev updated: [Feature][Connector-V2][Jdbc] Add oceanbase dialect factory (#4989)
This is an automated email from the ASF dual-hosted git repository. fanjia pushed a commit to branch dev in repository https://gitbox.apache.org/repos/asf/seatunnel.git The following commit(s) were added to refs/heads/dev by this push: new 7ba11cecd [Feature][Connector-V2][Jdbc] Add oceanbase dialect factory (#4989) 7ba11cecd is described below commit 7ba11cecdf7ac70c65ff5df501bbce8b499967c9 Author: He Wang AuthorDate: Tue Jul 11 17:34:02 2023 +0800 [Feature][Connector-V2][Jdbc] Add oceanbase dialect factory (#4989) - Co-authored-by: silenceland Co-authored-by: changhuyan <877018...@qq.com> --- docs/en/connector-v2/sink/Jdbc.md | 6 + docs/en/connector-v2/sink/OceanBase.md | 186 +++ docs/en/connector-v2/source/Jdbc.md| 6 + docs/en/connector-v2/source/OceanBase.md | 168 ++ .../jdbc/config/JdbcConnectionConfig.java | 13 ++ .../seatunnel/jdbc/config/JdbcOptions.java | 6 + .../seatunnel/jdbc/config/JdbcSourceConfig.java| 2 + .../jdbc/internal/dialect/JdbcDialectFactory.java | 10 + .../jdbc/internal/dialect/JdbcDialectLoader.java | 5 +- .../dialect/oceanbase/OceanBaseDialectFactory.java | 49 .../connectors/seatunnel/jdbc/sink/JdbcSink.java | 5 +- .../seatunnel/jdbc/sink/JdbcSinkFactory.java | 9 +- .../seatunnel/jdbc/source/JdbcSource.java | 4 +- .../seatunnel/jdbc/source/JdbcSourceFactory.java | 9 +- .../seatunnel/jdbc/JdbcOceanBaseITBase.java| 147 .../seatunnel/jdbc/JdbcOceanBaseMysqlIT.java | 256 + .../seatunnel/jdbc/JdbcOceanBaseOracleIT.java | 161 + .../jdbc_oceanbase_mysql_source_and_sink.conf | 55 + .../jdbc_oceanbase_oracle_source_and_sink.conf | 53 + .../e2e/connector/pulsar/PulsarBatchIT.java| 2 + 20 files changed, 1144 insertions(+), 8 deletions(-) diff --git a/docs/en/connector-v2/sink/Jdbc.md b/docs/en/connector-v2/sink/Jdbc.md index d472d9a33..f128f6b4b 100644 --- a/docs/en/connector-v2/sink/Jdbc.md +++ b/docs/en/connector-v2/sink/Jdbc.md @@ -33,6 +33,7 @@ support `Xa transactions`. You can set `is_exactly_once=true` to enable it. | user | String | No | - | | password | String | No | - | | query | String | No | - | +| compatible_mode | String | No | - | | database | String | No | - | | table | String | No | - | | primary_keys | Array | No | - | @@ -69,6 +70,10 @@ The URL of the JDBC connection. Refer to a case: jdbc:postgresql://localhost/tes Use this sql write upstream input datas to database. e.g `INSERT ...` +### compatible_mode [string] + +The compatible mode of database, required when the database supports multiple compatible modes. For example, when using OceanBase database, you need to set it to 'mysql' or 'oracle'. + ### database [string] Use this `database` and `table-name` auto-generate sql and receive upstream input datas write to database. @@ -168,6 +173,7 @@ there are some reference value for params above. | Redshift | com.amazon.redshift.jdbc42.Driver| jdbc:redshift://localhost:5439/testdb | com.amazon.redshift.xa.RedshiftXADataSource| https://mvnrepository.com/artifact/com.amazon.redshift/redshift-jdbc42 | | Snowflake | net.snowflake.client.jdbc.SnowflakeDriver| jdbc:snowflake://.snowflakecomputing.com | / | https://mvnrepository.com/artifact/net.snowflake/snowflake-jdbc | | Vertica| com.vertica.jdbc.Driver | jdbc:vertica://localhost:5433 | / | https://repo1.maven.org/maven2/com/vertica/jdbc/vertica-jdbc/12.0.3-0/vertica-jdbc-12.0.3-0.jar | +| OceanBase | com.oceanbase.jdbc.Driver| jdbc:oceanbase://localhost:2881| / | https://repo1.maven.org/maven2/com/oceanbase/oceanbase-client/2.4.3/oceanbase-client-2.4.3.jar | ## Example diff --git a/docs/en/connector-v2/sink/OceanBase.md b/docs/en/connector-v2/sink/OceanBase.md new file mode 100644 index 0..ec87ce3d3 --- /dev/null +++ b/docs/en/connector-v2/sink/OceanBase.md @@ -0,0 +1,186 @@ +# OceanBase + +> JDBC OceanBase Sink Connector + +## Support Those Engines + +> Spark +> Flink +>
[GitHub] [seatunnel] XiaoJiang521 opened a new pull request, #5061: [feature][doris] Doris factory type
XiaoJiang521 opened a new pull request, #5061: URL: https://github.com/apache/seatunnel/pull/5061 ## Purpose of this pull request ## Check list * [ ] Code changed are covered with tests, or it does not need tests for reason: * [ ] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md) * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs * [ ] If you are contributing the connector code, please check that the following files are updated: 1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2) 2. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it 3. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml) * [ ] Update the [`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] EricJoy2048 commented on a diff in pull request #5007: [Hotfix][Zeta] Fix wait job complete bug
EricJoy2048 commented on code in PR #5007:
URL: https://github.com/apache/seatunnel/pull/5007#discussion_r1259620087
##
seatunnel-engine/seatunnel-engine-client/src/main/java/org/apache/seatunnel/engine/client/job/ClientJobProxy.java:
##
@@ -93,21 +89,12 @@ private void submitJob(JobImmutableInformation
jobImmutableInformation) {
@Override
public JobStatus waitForJobComplete() {
try {
-jobResult =
-RetryUtils.retryWithException(
Review Comment:
> Why remove retry?
The test found that the retry here is meaningless because once the client
and server connection times out, the seaTunnelHazelcastClient will no longer be
available, so no matter how many retries are made here, it is useless
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [seatunnel] CheneyYin commented on pull request #4803: [Feature][Connector-V2] jdbc connector supports Kingbase database
CheneyYin commented on PR #4803: URL: https://github.com/apache/seatunnel/pull/4803#issuecomment-1630691160 > > > > please add e2e testcase https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e > > > > > > > > > need all success？ https://user-images.githubusercontent.com/72905543/251616238-826d67fb-2844-470d-9796-053c51ebbb71.png";> > > > > hello @TyrantLucifer ,when i test using spark container `Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: ErrorCode:[API-01], ErrorDescription:[Configuration item validate failed] - PluginName: jdbc, PluginType: source, Message: com.kingbase8.util.KSQLException: FATAL: invalid value for parameter "client_encoding": "ANSI_X3.4-1968" at org.apache.seatunnel.core.starter.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:63) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.spark.SeaTunnelSpark.main(SeaTunnelSpark.java:35) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start (SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)` > > https://github.com/apache/seatunnel/assets/72905543/a62cf1b3-4785-4a54-9db1-077fc63d3bb3";> > in local env is ok,I suspect something is wrong with the container [This link may help you to fix client encoding error](https://h2c.tech/p/os%E5%AD%97%E7%AC%A6%E9%9B%86%E5%BD%B1%E5%93%8Dkingbase%E6%95%B0%E6%8D%AE%E5%BA%93%E8%BF%9E%E6%8E%A5/) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] jackyyyyyssss opened a new issue, #5062: [Bug] [Module Name] Bug title
jacky opened a new issue, #5062: URL: https://github.com/apache/seatunnel/issues/5062 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened JdbcSourceConfigFactory hostname annotate is erro  ### SeaTunnel Version 2.3.2-SNAPSHOT ### SeaTunnel Config ```conf no have config ``` ### Running Command ```shell no have config ``` ### Error Exception ```log no have config ``` ### Flink or Spark Version _No response_ ### Java or Scala Version 1.8 ### Screenshots  ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] jackyyyyyssss closed issue #5062: annotate is erro
jacky closed issue #5062: annotate is erro URL: https://github.com/apache/seatunnel/issues/5062 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] javalover123 commented on pull request #5014: [Feature][Transform-V2][SQL] Support case when clause for SQL Transform plugin (#5013)
javalover123 commented on PR #5014: URL: https://github.com/apache/seatunnel/pull/5014#issuecomment-1630911862 OK, added. [https://github.com/apache/seatunnel-website/pull/253](https://github.com/apache/seatunnel-website/pull/253) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[seatunnel] branch dev updated: [HotFix][Zeta] fix after the savepoint job is restored, the checkpoint file cannot be generated #4985 (#5051)
This is an automated email from the ASF dual-hosted git repository.
wanghailin pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 7ed372897 [HotFix][Zeta] fix after the savepoint job is restored, the
checkpoint file cannot be generated #4985 (#5051)
7ed372897 is described below
commit 7ed372897930cb4d5ad40907d7099358b5839b59
Author: wu-a-ge
AuthorDate: Tue Jul 11 22:24:53 2023 +0800
[HotFix][Zeta] fix after the savepoint job is restored, the checkpoint file
cannot be generated #4985 (#5051)
* fix after the savepoint job is restored, the checkpoint file cannot be
generated
---
.../engine/server/checkpoint/CheckpointCoordinator.java | 13 -
1 file changed, 12 insertions(+), 1 deletion(-)
diff --git
a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/checkpoint/CheckpointCoordinator.java
b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/checkpoint/CheckpointCoordinator.java
index 3ae265850..9e0ef2a53 100644
---
a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/checkpoint/CheckpointCoordinator.java
+++
b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/checkpoint/CheckpointCoordinator.java
@@ -183,8 +183,19 @@ public class CheckpointCoordinator {
this.checkpointIdCounter = checkpointIdCounter;
this.readyToCloseStartingTask = new CopyOnWriteArraySet<>();
if (pipelineState != null) {
-this.latestCompletedCheckpoint =
+// fix after the savepoint job is restored, the checkpoint file
cannot be generated
+CompletedCheckpoint tmpCheckpoint =
serializer.deserialize(pipelineState.getStates(),
CompletedCheckpoint.class);
+this.latestCompletedCheckpoint =
+new CompletedCheckpoint(
+tmpCheckpoint.getJobId(),
+tmpCheckpoint.getPipelineId(),
+tmpCheckpoint.getCheckpointId(),
+tmpCheckpoint.getCheckpointTimestamp(),
+CheckpointType.CHECKPOINT_TYPE,
+tmpCheckpoint.getCompletedTimestamp(),
+tmpCheckpoint.getTaskStates(),
+tmpCheckpoint.getTaskStatistics());
}
this.checkpointCoordinatorFuture = new CompletableFuture();
[GitHub] [seatunnel] hailin0 merged pull request #5051: [HotFix][Zeta] fix after the savepoint job is restored, the checkpoint file cannot be generated #4985
hailin0 merged PR #5051: URL: https://github.com/apache/seatunnel/pull/5051 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] mgierdal commented on issue #4963: [Bug] [localFile-sink] Plugin saves open text tabular data regardless of fileFormat value
mgierdal commented on issue #4963: URL: https://github.com/apache/seatunnel/issues/4963#issuecomment-1631418600 Indeed, documentation was misleading. Changing a config field name from `fileFormat` to `file_format_type` made it work. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] mgierdal commented on issue #4599: Seatunnel with Spark: Caused by: java.lang.OutOfMemoryError: Java heap space
mgierdal commented on issue #4599:
URL: https://github.com/apache/seatunnel/issues/4599#issuecomment-1631449446
I am having a similar failure with ver.2.3.2, under java 17.0.7 2023-04-18
LTS.
```
2023-07-11 15:47:14,211 WARN
org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801
[seatunnel-495998] [5.1] Exception in
org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask@67644eb2
java.lang.OutOfMemoryError: Java heap space
...
2023-07-11 15:47:12,347 ERROR
org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job
SeaTunnel_Job (731233925082382337), Pipeline: [(1/1)], task: [pipeline-1
[Source[0]-JDBC-default-identifier]-SourceTask (1/2)] end with state FAILED and
Exception: java.lang.OutOfMemoryError: Java heap space
```
My config is:
```
env {
# You can set SeaTunnel environment configuration here
execution.parallelism = 2
job.mode = "BATCH"
checkpoint.interval = 1
#execution.checkpoint.interval = 1
#execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
}
source {
JDBC {
url = "jdbc:mysql:<>"
driver = "com.mysql.cj.jdbc.Driver"
user = <>
password = <>
query = "SELECT from <>"
parallelism = 2
fetch_size = 500
}
}
sink {
localFile {
path="test_log"
file_format_type="parquet"
}
}
```
The expected parquet folder is not being created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [seatunnel] CheneyYin commented on issue #4599: Seatunnel with Spark: Caused by: java.lang.OutOfMemoryError: Java heap space
CheneyYin commented on issue #4599:
URL: https://github.com/apache/seatunnel/issues/4599#issuecomment-1631673938
> I am having a similar failure with ver.2.3.2, under java 17.0.7 2023-04-18
LTS.
>
> ```
> 2023-07-11 15:47:14,211 WARN
org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801
[seatunnel-495998] [5.1] Exception in
org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask@67644eb2
> java.lang.OutOfMemoryError: Java heap space
> ...
> 2023-07-11 15:47:12,347 ERROR
org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job
SeaTunnel_Job (731233925082382337), Pipeline: [(1/1)], task: [pipeline-1
[Source[0]-JDBC-default-identifier]-SourceTask (1/2)] end with state FAILED and
Exception: java.lang.OutOfMemoryError: Java heap space
> ```
>
> My config is:
> ```
> env {
> # You can set SeaTunnel environment configuration here
> execution.parallelism = 2
> job.mode = "BATCH"
> checkpoint.interval = 1
> #execution.checkpoint.interval = 1
> #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
> }
>
> source {
>
> JDBC {
> url = "jdbc:mysql:<>"
> driver = "com.mysql.cj.jdbc.Driver"
> user = <>
> password = <>
> query = "SELECT from <>"
> parallelism = 2
> fetch_size = 500
> }
> }
> sink {
> localFile {
> path="test_log"
> file_format_type="parquet"
> }
> }
> ```
> The expected parquet folder is not being created.
>
Do you have the same problem when using the spark engine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] closed issue #4798: [Bug] [Jdbc] When Identity_ When Insert is set to OFF, it cannot be a table't '_ dest_ Insert explicit values for identity columns in mmx '
github-actions[bot] closed issue #4798: [Bug] [Jdbc] When Identity_ When Insert is set to OFF, it cannot be a table't '_ dest_ Insert explicit values for identity columns in mmx ' URL: https://github.com/apache/seatunnel/issues/4798 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] commented on issue #4669: There is an issue with the data format supported by seatunnel's HTTP source
github-actions[bot] commented on issue #4669: URL: https://github.com/apache/seatunnel/issues/4669#issuecomment-1631680430 This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] closed issue #4669: There is an issue with the data format supported by seatunnel's HTTP source
github-actions[bot] closed issue #4669: There is an issue with the data format supported by seatunnel's HTTP source URL: https://github.com/apache/seatunnel/issues/4669 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] gaopeng666 commented on a diff in pull request #4994: [Feature][Connector-V2][Clickhouse] Add clickhouse connector timezone key,default UTC
gaopeng666 commented on code in PR #4994:
URL: https://github.com/apache/seatunnel/pull/4994#discussion_r1260458766
##
seatunnel-connectors-v2/connector-clickhouse/src/main/java/org/apache/seatunnel/connectors/seatunnel/clickhouse/config/ClickhouseConfig.java:
##
@@ -75,6 +75,13 @@ public class ClickhouseConfig {
.noDefaultValue()
.withDescription("Clickhouse server password");
+/** Clickhouse server timezone */
+public static final Option SERVER_TIME_ZONE =
+Options.key("server_time_zone")
+.stringType()
+.defaultValue("UTC")
Review Comment:
I have set the timezone by default to the timezone of the seatunnel service
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [seatunnel] EricJoy2048 commented on a diff in pull request #5061: [feature][doris] Doris factory type
EricJoy2048 commented on code in PR #5061:
URL: https://github.com/apache/seatunnel/pull/5061#discussion_r1260475971
##
seatunnel-connectors-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/datatype/DorisDataTypeConvertor.java:
##
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.doris.datatype;
+
+import org.apache.seatunnel.api.table.catalog.DataTypeConvertException;
+import org.apache.seatunnel.api.table.catalog.DataTypeConvertor;
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.DecimalType;
+import org.apache.seatunnel.api.table.type.LocalTimeType;
+import org.apache.seatunnel.api.table.type.PrimitiveByteArrayType;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.api.table.type.SqlType;
+
+import org.apache.commons.collections4.MapUtils;
+
+import com.google.auto.service.AutoService;
+import com.google.common.collect.ImmutableMap;
+
+import java.util.Collections;
+import java.util.Locale;
+import java.util.Map;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+
+@AutoService(DataTypeConvertor.class)
+public class DorisDataTypeConvertor implements DataTypeConvertor {
+
+public static final String NULL = "NULL";
+public static final String BOOLEAN = "BOOLEAN";
+public static final String TINYINT = "TINYINT";
+public static final String SMALLINT = "SMALLINT";
+public static final String INT = "INT";
+public static final String BIGINT = "BIGINT";
+public static final String FLOAT = "FLOAT";
+public static final String DOUBLE = "DOUBLE";
+public static final String DECIMAL = "DECIMAL";
+public static final String DATE = "DATE";
+public static final String DATETIME = "DATETIME";
+public static final String CHAR = "CHAR";
+public static final String VARCHAR = "VARCHAR";
+public static final String BINARY = "BINARY";
+public static final String VARBINARY = "VARBINARY";
+public static final String ARRAY = "ARRAY";
+public static final String MAP = "MAP";
+public static final String STRUCT = "STRUCT";
+public static final String UNION = "UNION";
+public static final String INTERVAL = "INTERVAL";
+public static final String TIMESTAMP = "TIMESTAMP";
+public static final String YEAR = "YEAR";
+public static final String GEOMETRY = "GEOMETRY";
+public static final String IP = "IP";
+
+public static final String PRECISION = "precision";
+public static final String SCALE = "scale";
+
+public static final Integer DEFAULT_PRECISION = 10;
+
+public static final Integer DEFAULT_SCALE = 0;
+
+@Override
+public SeaTunnelDataType toSeaTunnelType(String connectorDataType) {
+checkNotNull(connectorDataType, "connectorDataType can not be null");
+Map dataTypeProperties;
+switch (connectorDataType.toUpperCase(Locale.ROOT)) {
+case DECIMAL:
+// parse precision and scale
+int left = connectorDataType.indexOf("(");
+int right = connectorDataType.indexOf(")");
+int precision = DEFAULT_PRECISION;
+int scale = DEFAULT_SCALE;
+if (left != -1 && right != -1) {
+String[] precisionAndScale =
+connectorDataType.substring(left + 1,
right).split(",");
+if (precisionAndScale.length == 2) {
+precision = Integer.parseInt(precisionAndScale[0]);
+scale = Integer.parseInt(precisionAndScale[1]);
+} else if (precisionAndScale.length == 1) {
+precision = Integer.parseInt(precisionAndScale[0]);
+}
+}
+dataTypeProperties = ImmutableMap.of(PRECISION, precision,
SCALE, scale);
+break;
+default:
+dataTypeProperties = Collections.emptyMap();
+break;
+}
+return toSeaTunnelType(connectorDataType, dataTypeProperties);
+}
+
+@Override
+public SeaTunnelDa
[GitHub] [seatunnel] liugddx opened a new pull request, #5063: [bugfix][zeta][restapi] Fix ClassNotFoundException.
liugddx opened a new pull request, #5063: URL: https://github.com/apache/seatunnel/pull/5063 ## Purpose of this pull request ## Check list * [ ] Code changed are covered with tests, or it does not need tests for reason: * [ ] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md) * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs * [ ] If you are contributing the connector code, please check that the following files are updated: 1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2) 2. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it 3. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml) * [ ] Update the [`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] ic4y commented on a diff in pull request #5057: [Improve][CDC] support exactly-once of cdc and fix the BinlogOffset comparing bug
ic4y commented on code in PR #5057:
URL: https://github.com/apache/seatunnel/pull/5057#discussion_r1260486937
##
seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/source/reader/external/IncrementalSourceScanFetcher.java:
##
@@ -223,14 +223,11 @@ public void close() {
private boolean isChangeRecordInChunkRange(SourceRecord record) {
if (taskContext.isDataChangeRecord(record)) {
+// fix the between condition
return taskContext.isRecordBetween(
record,
-null == currentSnapshotSplit.getSplitStart()
-? null
-: new Object[]
{currentSnapshotSplit.getSplitStart()},
-null == currentSnapshotSplit.getSplitEnd()
-? null
-: new Object[]
{currentSnapshotSplit.getSplitEnd()});
+currentSnapshotSplit.getSplitStart(),
Review Comment:
The revision is right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [seatunnel] sunxiaojian commented on pull request #5024: [Feature][SeaTunnel Engine IMap Storage] Imap storage supports kafka compact topic in cluster mode
sunxiaojian commented on PR #5024: URL: https://github.com/apache/seatunnel/pull/5024#issuecomment-1631760368 > > > > Yes, I think you can enable the test case by use kafka container. done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] liunaijie opened a new pull request, #5064: support avro format & support run action on fork repo
liunaijie opened a new pull request, #5064: URL: https://github.com/apache/seatunnel/pull/5064 ## Purpose of this pull request 1. support avro format 2. on kafka connect support avro format 3. update github action config then can run CI on fork repo ## Check list * [x] Code changed are covered with tests, or it does not need tests for reason: * [x] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md) * [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs * [ ] If you are contributing the connector code, please check that the following files are updated: 1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2) 4. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it 5. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml) * [ ] Update the [`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] liunaijie closed pull request #5064: support avro format & support run action on fork repo
liunaijie closed pull request #5064: support avro format & support run action on fork repo URL: https://github.com/apache/seatunnel/pull/5064 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] liunaijie opened a new pull request, #5065: [Feature] update action config to support run CI on fork repo
liunaijie opened a new pull request, #5065: URL: https://github.com/apache/seatunnel/pull/5065 ## Purpose of this pull request This update will let the CI run on fork repo. ## Check list * [ ] Code changed are covered with tests, or it does not need tests for reason: * [ ] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md) * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs * [ ] If you are contributing the connector code, please check that the following files are updated: 1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2) 2. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it 3. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml) * [ ] Update the [`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] liunaijie commented on pull request #5065: [Feature] update action config to support run CI on fork repo
liunaijie commented on PR #5065: URL: https://github.com/apache/seatunnel/pull/5065#issuecomment-1631775800 @liugddx PTAL, i changed some code base on your pr. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] liugddx commented on pull request #5065: [Feature] update action config to support run CI on fork repo
liugddx commented on PR #5065: URL: https://github.com/apache/seatunnel/pull/5065#issuecomment-1631779540 > @liugddx PTAL, i changed some code base on your pr. Good job ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] ruanwenjun commented on a diff in pull request #5065: [Feature] update action config to support run CI on fork repo
ruanwenjun commented on code in PR #5065: URL: https://github.com/apache/seatunnel/pull/5065#discussion_r1260523259 ## .github/workflows/backend.yml: ## @@ -18,9 +18,9 @@ name: Backend on: push: + pull_request: Review Comment: Do we need to remove below setting? ``` branches: - dev ``` This will cause only submit PR to dev will trigger the workflow? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] liugddx commented on a diff in pull request #5065: [Feature] update action config to support run CI on fork repo
liugddx commented on code in PR #5065: URL: https://github.com/apache/seatunnel/pull/5065#discussion_r1260530012 ## .github/workflows/backend.yml: ## @@ -18,9 +18,9 @@ name: Backend on: push: + pull_request: Review Comment: > Do we need to remove below setting? > > ``` > branches: >- dev > ``` > > This will cause only submit PR to dev will trigger the workflow? @EricJoy2048 @hailin0 @Hisoka-X @ic4y PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] liugddx commented on a diff in pull request #5065: [Feature] update action config to support run CI on fork repo
liugddx commented on code in PR #5065: URL: https://github.com/apache/seatunnel/pull/5065#discussion_r1260530263 ## .github/workflows/backend.yml: ## @@ -18,9 +18,9 @@ name: Backend on: push: + pull_request: Review Comment: > > Do we need to remove below setting? > > ``` > > branches: > >- dev > > ``` > > > > > > > > > > > > > > > > > > > > > > > > This will cause only submit PR to dev will trigger the workflow? > > @EricJoy2048 @hailin0 @Hisoka-X @ic4y PTAL @TyrantLucifer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] liugddx commented on a diff in pull request #5065: [Feature] update action config to support run CI on fork repo
liugddx commented on code in PR #5065: URL: https://github.com/apache/seatunnel/pull/5065#discussion_r1260534203 ## .github/workflows/backend.yml: ## @@ -18,9 +18,9 @@ name: Backend on: push: + pull_request: Review Comment: > Do we need to remove below setting? > > ``` > branches: >- dev > ``` > > This will cause only submit PR to dev will trigger the workflow? I think there is no problem with the dev branch, we are mainly developing in the dev branch now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] evan766 commented on pull request #4918: [feature][Connector-V2][cdc] Support flink running cdc job
evan766 commented on PR #4918: URL: https://github.com/apache/seatunnel/pull/4918#issuecomment-1631807054 > ## Purpose of this pull request > ## Check list > * [ ] Code changed are covered with tests, or it does not need tests for reason: > * [ ] If any new Jar binary package adding in your PR, please add License Notice according > [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md) > * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs > * [ ] If you are contributing the connector code, please check that the following files are updated: > > 1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2) > 2. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it > 3. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml) > * [ ] Update the [`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md). 兄弟，我clone了你的项目，用的flink-cdc分支编译，再创建mysql-cdc任务到 flink, flink还是报错，你这边是OK的吗 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] Carl-Zhou-CN commented on pull request #4918: [feature][Connector-V2][cdc] Support flink running cdc job
Carl-Zhou-CN commented on PR #4918: URL: https://github.com/apache/seatunnel/pull/4918#issuecomment-1631834268 @evan766 Can you share your json configuration? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] liugddx merged pull request #5063: [bugfix][zeta][restapi] Fix ClassNotFoundException.
liugddx merged PR #5063: URL: https://github.com/apache/seatunnel/pull/5063 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[seatunnel] branch dev updated: fix class not found exception (#5063)
This is an automated email from the ASF dual-hosted git repository. liugddx pushed a commit to branch dev in repository https://gitbox.apache.org/repos/asf/seatunnel.git The following commit(s) were added to refs/heads/dev by this push: new 6060b13f9 fix class not found exception (#5063) 6060b13f9 is described below commit 6060b13f9f9a223fd0a85c33d98e5f86df3aa254 Author: Guangdong Liu <804167...@qq.com> AuthorDate: Wed Jul 12 12:39:08 2023 +0800 fix class not found exception (#5063) --- .../engine/server/rest/RestHttpGetCommandProcessor.java| 14 +- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/RestHttpGetCommandProcessor.java b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/RestHttpGetCommandProcessor.java index 1540f99c9..6c71ac2fe 100644 --- a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/RestHttpGetCommandProcessor.java +++ b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/RestHttpGetCommandProcessor.java @@ -22,6 +22,7 @@ import org.apache.seatunnel.shade.com.fasterxml.jackson.databind.JsonNode; import org.apache.seatunnel.shade.com.fasterxml.jackson.databind.ObjectMapper; import org.apache.seatunnel.engine.common.Constant; +import org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader; import org.apache.seatunnel.engine.core.dag.logical.LogicalDag; import org.apache.seatunnel.engine.core.job.JobImmutableInformation; import org.apache.seatunnel.engine.core.job.JobInfo; @@ -42,6 +43,7 @@ import com.hazelcast.internal.json.JsonObject; import com.hazelcast.internal.json.JsonValue; import com.hazelcast.internal.util.JsonUtil; import com.hazelcast.internal.util.StringUtil; +import com.hazelcast.jet.impl.execution.init.CustomClassLoadedObject; import com.hazelcast.map.IMap; import com.hazelcast.spi.impl.NodeEngine; @@ -227,12 +229,14 @@ public class RestHttpGetCommandProcessor extends HttpCommandProcessor
[GitHub] [seatunnel] sunxiaojian opened a new pull request, #5066: [Feature][connector][kafka] Support read debezium format message from kafka
sunxiaojian opened a new pull request, #5066: URL: https://github.com/apache/seatunnel/pull/5066 https://github.com/apache/seatunnel/issues/3743 Support read debezium format message from kafka ## Check list * [ ] Code changed are covered with tests, or it does not need tests for reason: * [ ] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md) * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs * [ ] If you are contributing the connector code, please check that the following files are updated: 1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2) 2. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it 3. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml) * [ ] Update the [`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] sunxiaojian commented on pull request #5066: [Feature][connector][kafka] Support read debezium format message from kafka
sunxiaojian commented on PR #5066: URL: https://github.com/apache/seatunnel/pull/5066#issuecomment-1631838075 https://github.com/apache/seatunnel/pull/3981 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] liugddx closed pull request #5065: [Feature] update action config to support run CI on fork repo
liugddx closed pull request #5065: [Feature] update action config to support run CI on fork repo URL: https://github.com/apache/seatunnel/pull/5065 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] liugddx commented on a diff in pull request #5024: [Feature][SeaTunnel Engine IMap Storage] Imap storage supports kafka compact topic in cluster mode
liugddx commented on code in PR #5024:
URL: https://github.com/apache/seatunnel/pull/5024#discussion_r1260583929
##
pom.xml:
##
@@ -146,6 +146,7 @@
2.4.7
3.1.4
4.1.60.Final
+3.4.1
Review Comment:
We support those versions of kafka?
##
seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/pom.xml:
##
@@ -92,6 +99,19 @@
${netty-buffer.version}
test
+
+
+org.testcontainers
+kafka
+${testcontainer.version}
+test
+
+
+junit
+junit
Review Comment:
Do we need this dependency?
##
seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-kafka/pom.xml:
##
@@ -0,0 +1,69 @@
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+org.apache.seatunnel
+imap-storage-plugins
+${revision}
+
+
+org.apache.seatunnel
+imap-storage-kafka
+SeaTunnel : Engine : Storage : IMap Storage Plugins : Kafka
+
+
+UTF-8
+
+
+
+
+org.apache.seatunnel
+serializer-protobuf
+${project.version}
+
+
+org.apache.kafka
+kafka-clients
+
+
+org.awaitility
+awaitility
+
+
+org.apache.commons
+commons-lang3
+
+
+org.testcontainers
+kafka
+${testcontainer.version}
+test
+
+
+junit
+junit
+4.13.2
+test
Review Comment:
Same as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [seatunnel] ruanwenjun commented on a diff in pull request #5065: [Feature] update action config to support run CI on fork repo
ruanwenjun commented on code in PR #5065: URL: https://github.com/apache/seatunnel/pull/5065#discussion_r1260599986 ## .github/workflows/backend.yml: ## @@ -18,9 +18,9 @@ name: Backend on: push: + pull_request: Review Comment: If we cp to release-branch, the ci will not trigger, it would be better to keep the origin rule. Why we add the new rule "only trigger on dev"? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] liunaijie commented on a diff in pull request #5065: [Feature] update action config to support run CI on fork repo
liunaijie commented on code in PR #5065: URL: https://github.com/apache/seatunnel/pull/5065#discussion_r1260614300 ## .github/workflows/backend.yml: ## @@ -18,9 +18,9 @@ name: Backend on: push: + pull_request: Review Comment: only trigger on dev rule is added on this pr: https://github.com/apache/seatunnel/pull/4522/files. For this pr, my purpose is enable the CI on fork repo. Then it can run CI on fork repo first to check, fix error to reduce everyone's time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [seatunnel] evan766 commented on pull request #4918: [feature][Connector-V2][cdc] Support flink running cdc job
evan766 commented on PR #4918:
URL: https://github.com/apache/seatunnel/pull/4918#issuecomment-1631908573
> @evan766 Can you share your json configuration?
thank you very much to reply me!
steps I do, where was wrong?
1、fork your project:
https://github.com/Carl-Zhou-CN/incubator-seatunnel/tree/flink-cdc
2、here is my forked project: `https://github.com/evan766/incubator-seatunnel`
3、clone https://github.com/evan766/incubator-seatunnel to my local machine.
4 、
```
git clone g...@github.com:evan766/incubator-seatunnel.git
checkout flink-cdc
checkout -b dev-cdc
git remote add apache-seatunnel https://github.com/apache/seatunnel.git
git merge apache-seatunnel/dev
git push origin
```
5、follow the steps in the tutorial to build binanry package：
https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/setup.md
```
#Install Subproject Locally
./mvnw install -Dmaven.test.skip
#Building seaTunnel from source
mvn clean package -pl seatunnel-dist -am -Dmaven.test.skip=true
```
6、terminal output target in dir: incubator-seatunnel/seatunnel-dist/target
```
$: ls -l incubator-seatunnel/seatunnel-dist/target
-rw-r--r--@ 1 hong staff 1593852848 Jul 12 11:56
apache-seatunnel-2.3.2-SNAPSHOT-bin.tar.gz
-rw-r--r--@ 1 hong staff 4838534 Jul 12 11:56
apache-seatunnel-2.3.2-SNAPSHOT-src.tar.gz
drwxr-xr-x@ 2 hong staff 64 Jul 12 11:55 archive-tmp
drwxr-xr-x@ 3 hong staff 96 Jul 12 11:55
maven-shared-archive-resources
-rw-r--r--@ 1 hong staff 81 Jul 12 11:55 spotless-index
```
7、 push job to flink , flink Version: 1.16.2
```
tar zxvf apache-seatunnel-2.3.2-SNAPSHOT-bin.tar.gz
bash bin/start-seatunnel-flink-15-connector-v2.sh --config
config/cdc.template
```
cdc.templte
```
env {
execution.parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 1
}
source {
MySQL-CDC {
debezium {
poll.interval.ms = 1000
}
result_table_name = "temp_organization"
parallelism = 1
server-id = 5656
username = "root"
password = "root"
table-names = ["testbeat.organization"]
base-url = "jdbc:mysql://192.168.66.227:3306/testbeat"
}
}
sink {
jdbc {
source_table_name:"temp_organization"
url = "jdbc:mysql://192.168.66.227:3306/seatunnel"
driver = "com.mysql.cj.jdbc.Driver"
user = "root"
password = "root"
generate_sink_sql = "true"
database = "seatunnel"
table = "organization2"
primary_keys = ["id"]
}
}
```
8、flink job failed:
```
2023-07-12 14:10:02
java.lang.UnsupportedOperationException: Flink ParallelSource don't support
sending SourceEvent. Please implement the `SupportCoordinate` marker interface
on the SeaTunnel source.
at
org.apache.seatunnel.translation.source.ParallelReaderContext.sendSourceEventToEnumerator(ParallelReaderContext.java:61)
at
org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.reportFinishedSnapshotSplitsIfNeed(IncrementalSourceReader.java:151)
at
org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.onSplitFinished(IncrementalSourceReader.java:133)
at
org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.finishCurrentFetch(SourceReaderBase.java:204)
at
org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.moveToNextSplit(SourceReaderBase.java:180)
at
org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:161)
at
org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:92)
at
org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.pollNext(IncrementalSourceReader.java:92)
at
org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:112)
at
org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.pollNext(IncrementalSourceReader.java:92)
at
org.apache.seatunnel.translation.source.ParallelSource.run(ParallelSource.java:136)
at
org.apache.seatunnel.translation.flink.source.BaseSeaTunnelSourceFunction.run(BaseSeaTunnelSourceFunction.java:86)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:67)
at
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:333)
```
 Whether this was removed when merged？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
