Re: [I] [Feature][Kafka] Support XML format read/write kafka record [seatunnel]
litiliu commented on issue #8749: URL: https://github.com/apache/seatunnel/issues/8749#issuecomment-2670502974 I'll handle this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Feature][Connector-V2] Add `file_extension` parameter for read/write file [seatunnel]
Hisoka-X commented on code in PR #8769:
URL: https://github.com/apache/seatunnel/pull/8769#discussion_r1962909577
##
docs/en/connector-v2/sink/CosFile.md:
##
@@ -46,6 +46,7 @@ By default, we use 2PC commit to ensure `exactly-once`
| file_name_expression | string | no |
"${transactionId}" | Only used when custom_filename is
true
|
| filename_time_format | string | no | ".MM.dd"
| Only used when custom_filename is true
|
| file_format_type | string | no | "csv"
|
|
+| file_extension| string | no | -
| Override the default file extensions with custom
file extensions
|
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] [Feature][Module Name] TiDB CDC data into Doris [seatunnel]
remones commented on issue #8764: URL: https://github.com/apache/seatunnel/issues/8764#issuecomment-2670310716 @hailin0 you can assign it to me -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [Feature] Add export functionality for job configuration [seatunnel-web]
liunaijie opened a new pull request, #279: URL: https://github.com/apache/seatunnel-web/pull/279 ## Purpose of this pull request add api to export job conf ## Check list * [ ] Code changed are covered with tests, or it does not need tests for reason: * [ ] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md) * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Doc][Improve] support chinese [docs/zh/connector-v2/sink/TDengine.md] [seatunnel]
duzhendi commented on code in PR #8732: URL: https://github.com/apache/seatunnel/pull/8732#discussion_r1962762391 ## docs/zh/connector-v2/sink/TDengine.md: ## @@ -0,0 +1,71 @@ +# TDengine + +> TDengine 数据接收器 + +## 描述 + +用于将数据写入TDengine. 在运行 seatunnel 任务之前，你需要创建稳定的环境。 + +## 主要特性 + +- [x] [exactly-once](../../concept/connector-v2-features.md) +- [ ] [cdc](../../concept/connector-v2-features.md) Review Comment: 对比了 en/connector-v2/sink/TDengine.md文件夹下的TDengine.md文件，没发现有缺少的内容呢 https://github.com/user-attachments/assets/9d90499b-972c-4212-9fa5-4e6c75898537 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Doc][Improve] support chinese [docs/zh/connector-v2/sink/TDengine.md] [seatunnel]
duzhendi commented on code in PR #8732: URL: https://github.com/apache/seatunnel/pull/8732#discussion_r1962767390 ## docs/zh/connector-v2/sink/TDengine.md: ## @@ -0,0 +1,71 @@ +# TDengine + +> TDengine 数据接收器 + +## 描述 + +用于将数据写入TDengine. 在运行 seatunnel 任务之前，你需要创建稳定的环境。 + +## 主要特性 + +- [x] [exactly-once](../../concept/connector-v2-features.md) +- [ ] [cdc](../../concept/connector-v2-features.md) Review Comment: 您好，对比了docs/en/connector-v2/sink/TDengine.md, 没发现有缺少内容呢 https://github.com/user-attachments/assets/3106cd0c-06bb-4601-9c48-f9e470fd8634 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Doc][Improve] support chinese [docs/zh/connector-v2/sink/TDengine.md] [seatunnel]
duzhendi commented on code in PR #8732: URL: https://github.com/apache/seatunnel/pull/8732#discussion_r1962762391 ## docs/zh/connector-v2/sink/TDengine.md: ## @@ -0,0 +1,71 @@ +# TDengine + +> TDengine 数据接收器 + +## 描述 + +用于将数据写入TDengine. 在运行 seatunnel 任务之前，你需要创建稳定的环境。 + +## 主要特性 + +- [x] [exactly-once](../../concept/connector-v2-features.md) +- [ ] [cdc](../../concept/connector-v2-features.md) Review Comment: 对比了 en/connector-v2/sink/TDengine.md文件夹下的TDengine.md文件，没发现有缺少的内容呢 https://github.com/user-attachments/assets/9d90499b-972c-4212-9fa5-4e6c75898537 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Doc][Improve] support chinese [docs/zh/connector-v2/sink/Tablestore.md] [seatunnel]
duzhendi commented on code in PR #8731: URL: https://github.com/apache/seatunnel/pull/8731#discussion_r1962764998 ## docs/zh/connector-v2/sink/Tablestore.md: ## @@ -0,0 +1,73 @@ +# Tablestore + +> Tablestore 数据接收器 + +## 描述 + +用于将数据写入 Tablestore + +## 主要特性 + +- [ ] [exactly-once](../../concept/connector-v2-features.md) + Review Comment: 对比了docs/eh/connector-v2/sink/Tablestore.md下的Tablestore.md，没发现有缺少内容呢 https://github.com/user-attachments/assets/f4bddc1e-72ee-4cfa-8438-cf65cde477e6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Feature][Connector-V2] Add `file_extension` parameter for read/write file [seatunnel]
hailin0 commented on code in PR #8769:
URL: https://github.com/apache/seatunnel/pull/8769#discussion_r1962764747
##
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/BaseSinkConfig.java:
##
@@ -214,6 +214,12 @@ public class BaseSinkConfig {
.defaultValue(FileFormat.CSV)
.withDescription("File format type, e.g. csv, orc,
parquet, text");
+public static final Option FILE_EXTENSION =
+Options.key("file_extension")
+.stringType()
+.noDefaultValue()
+.withDescription("The file extension of files");
Review Comment:
add example into description
##
docs/en/connector-v2/sink/CosFile.md:
##
@@ -46,6 +46,7 @@ By default, we use 2PC commit to ensure `exactly-once`
| file_name_expression | string | no |
"${transactionId}" | Only used when custom_filename is
true
|
| filename_time_format | string | no | ".MM.dd"
| Only used when custom_filename is true
|
| file_format_type | string | no | "csv"
|
|
+| file_extension| string | no | -
| Override the default file extensions with custom
file extensions
|
Review Comment:
add example into all docs
##
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/BaseSinkConfig.java:
##
@@ -214,6 +214,12 @@ public class BaseSinkConfig {
.defaultValue(FileFormat.CSV)
.withDescription("File format type, e.g. csv, orc,
parquet, text");
+public static final Option FILE_EXTENSION =
+Options.key("file_extension")
Review Comment:
or use `file_suffix`?
##
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/BaseSourceConfigOptions.java:
##
@@ -34,6 +34,12 @@ public class BaseSourceConfigOptions {
.withDescription(
"File format type, e.g. json, csv, text, parquet,
orc, avro");
+public static final Option FILE_EXTENSION =
+Options.key("file_extension")
+.stringType()
+.noDefaultValue()
+.withDescription("The file extension of files");
Review Comment:
add example into description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [Feature][Connector-V2] Add `file_extension` parameter for read/write file [seatunnel]
hailin0 commented on code in PR #8769:
URL: https://github.com/apache/seatunnel/pull/8769#discussion_r1962767560
##
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/BaseSinkConfig.java:
##
@@ -214,6 +214,12 @@ public class BaseSinkConfig {
.defaultValue(FileFormat.CSV)
.withDescription("File format type, e.g. csv, orc,
parquet, text");
+public static final Option FILE_EXTENSION =
+Options.key("file_extension")
Review Comment:
or use `file_suffix` 、`filename_extension `?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix possible data loss in certain scenarios of starrocks [seatunnel]
xiaochen-zhou commented on PR #8768: URL: https://github.com/apache/seatunnel/pull/8768#issuecomment-2670323737 > Please add testcase (reproduce & verify) Ok. I'll try to complete it today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve] update clickhouse connector config option [seatunnel]
Hisoka-X merged PR #8755: URL: https://github.com/apache/seatunnel/pull/8755 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(seatunnel) branch dev updated (b16aac232a -> b964189b75)
This is an automated email from the ASF dual-hosted git repository.
fanjia pushed a change to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
from b16aac232a [Feature][Config] Support custom config keys for
encrypt/decrypt (#8739)
add b964189b75 [improve] update clickhouse connector config option (#8755)
No new revisions were added by this update.
Summary of changes:
.../seatunnel/api/ConnectorOptionCheckTest.java| 3 -
.../clickhouse/catalog/ClickhouseCatalog.java | 14 +-
.../catalog/ClickhouseCatalogFactory.java | 10 +-
.../clickhouse/config/ClickhouseBaseOptions.java} | 52 ---
.../config/ClickhouseFileSinkOptions.java | 87
...houseConfig.java => ClickhouseSinkOptions.java} | 116 +--
.../config/ClickhouseSourceOptions.java} | 11 +-
.../clickhouse/sink/client/ClickhouseSink.java | 30 ++--
.../sink/client/ClickhouseSinkFactory.java | 39 +++--
.../clickhouse/sink/file/ClickhouseFileSink.java | 158 +
.../sink/file/ClickhouseFileSinkFactory.java | 139 --
.../clickhouse/source/ClickhouseSourceFactory.java | 15 +-
.../seatunnel/clickhouse/util/ClickhouseProxy.java | 4 +-
.../seatunnel/clickhouse/util/ClickhouseUtil.java | 12 +-
.../clickhouse/ClickhouseCreateTableTest.java | 14 +-
15 files changed, 328 insertions(+), 376 deletions(-)
copy
seatunnel-connectors-v2/{connector-starrocks/src/main/java/org/apache/seatunnel/connectors/seatunnel/starrocks/config/StarRocksBaseOptions.java
=>
connector-clickhouse/src/main/java/org/apache/seatunnel/connectors/seatunnel/clickhouse/config/ClickhouseBaseOptions.java}
(52%)
create mode 100644
seatunnel-connectors-v2/connector-clickhouse/src/main/java/org/apache/seatunnel/connectors/seatunnel/clickhouse/config/ClickhouseFileSinkOptions.java
rename
seatunnel-connectors-v2/connector-clickhouse/src/main/java/org/apache/seatunnel/connectors/seatunnel/clickhouse/config/{ClickhouseConfig.java
=> ClickhouseSinkOptions.java} (52%)
copy
seatunnel-connectors-v2/{connector-web3j/src/main/java/org/apache/seatunnel/connectors/seatunnel/config/Web3jConfig.java
=>
connector-clickhouse/src/main/java/org/apache/seatunnel/connectors/seatunnel/clickhouse/config/ClickhouseSourceOptions.java}
(76%)
Re: [I] [Bug] [MySQL CDC] Memory Leak [seatunnel]
github-actions[bot] closed issue #8475: [Bug] [MySQL CDC] Memory Leak URL: https://github.com/apache/seatunnel/issues/8475 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] [MySQL CDC] Memory Leak [seatunnel]
github-actions[bot] commented on issue #8475: URL: https://github.com/apache/seatunnel/issues/8475#issuecomment-2670089196 This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Feature][Elasticsearch] When I want to get the _id of es, it is difficult [seatunnel]
github-actions[bot] closed issue #8506: [Feature][Elasticsearch] When I want to get the _id of es, it is difficult URL: https://github.com/apache/seatunnel/issues/8506 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Feature][Elasticsearch] When I want to get the _id of es, it is difficult [seatunnel]
github-actions[bot] commented on issue #8506: URL: https://github.com/apache/seatunnel/issues/8506#issuecomment-2670089160 This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] [MongoDB-CDC] Can not find catalog with factoryId [MongoDB] [seatunnel]
github-actions[bot] commented on issue #8562: URL: https://github.com/apache/seatunnel/issues/8562#issuecomment-2670089083 This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Feature][Config] Support custom config keys for encrypt/decrypt [seatunnel]
Hisoka-X merged PR #8739: URL: https://github.com/apache/seatunnel/pull/8739 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(seatunnel) branch dev updated (4eec9be012 -> b16aac232a)
This is an automated email from the ASF dual-hosted git repository. fanjia pushed a change to branch dev in repository https://gitbox.apache.org/repos/asf/seatunnel.git from 4eec9be012 [Improve] rabbit mq options (#8740) add b16aac232a [Feature][Config] Support custom config keys for encrypt/decrypt (#8739) No new revisions were added by this update. Summary of changes: .../connector-v2/Config-Encryption-Decryption.md | 16 ++- .../connector-v2/Config-Encryption-Decryption.md | 132 +++-- .../common/config/TypesafeConfigUtils.java | 2 +- .../core/starter/command/ConfDecryptCommand.java | 6 +- .../core/starter/utils/ConfigBuilder.java | 41 +-- .../core/starter/utils/ConfigShadeUtils.java | 36 +- .../core/starter/utils/ConfigBuilderTest.java | 4 +- .../core/starter/utils/ConfigShadeTest.java| 26 +++- .../src/test/resources/config.shade.conf | 6 + .../src/test/resources/config.shade.json | 11 +- 10 files changed, 189 insertions(+), 91 deletions(-)
Re: [I] [Feature][Config] Support custom config keys for encrypt/decrypt [seatunnel]
Hisoka-X closed issue #8511: [Feature][Config] Support custom config keys for encrypt/decrypt URL: https://github.com/apache/seatunnel/issues/8511 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug] [Oracle-CDC] startup.mode not supported specific [seatunnel]
Quilian opened a new issue, #8771:
URL: https://github.com/apache/seatunnel/issues/8771
### Search before asking
- [x] I had searched in the
[issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22)
and found no similar issues.
### What happened
根据官网文档描述，Oracle-cdc配置了startup.mode = specific ，但是启动的时候报错：the value(SPECIFIC)
must be one of the optionValues([INITIAL, LATEST])
### SeaTunnel Version
2.3.9
### SeaTunnel Config
```conf
env{
execution.parallelism = 1
job.mode = "STREAMING"
checkpoint.interval =1000
}
source {
Oracle-CDC {
username = "xx"
password = "xxx"
database-names = ["db1"]
schema-names = ["sc1"]
table-names = ["db1.sc1.table1"]
startup.mode = specific
startup.specific-offset.file = "/OSdata/oradata_ncdb/db1/redo03.log"
startup.specific-offset.pos = 14316
connect.timeout.ms = 6
connect.max-retries = 5
base-url = "jdbc:oracle:thin:@xx.xxx.xxx.xx::db1"
}
}
sink {
Doris {
fenodes = "xx.xxx.xx.xxx:"
username = xxx
password = "x"
database = "xxx"
#table = ""
sink.enable-2pc = "false"
sink.label-prefix = "xxx_111"
sink.enable-delete = "true"
doris.config = {
format="json"
read_json_by_line="true"
}
}
}
```
### Running Command
```shell
bin/seatunnel.sh -c xx.template -n jobname --async
```
### Error Exception
```log
2025-02-20 15:05:13,055 ERROR [o.a.s.c.s.SeaTunnel ] [main] -
Exception
StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException:
SeaTunnel job executed failed
at
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:228)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at
org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.api.table.factory.FactoryException:
ErrorCode:[API-06], ErrorDescription:[Factory initialize failed] - Unable to
create a source for identifier 'Oracle-CDC'.
at
org.apache.seatunnel.api.table.factory.FactoryUtil.restoreAndPrepareSource(FactoryUtil.java:119)
at
org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:68)
at
org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:404)
at
org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:236)
at
org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:123)
at
org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:191)
at
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:165)
... 2 more
Caused by:
org.apache.seatunnel.api.configuration.util.OptionValidationException:
ErrorCode:[API-02], ErrorDescription:[Option item validate failed] - These
options('startup.mode') are SingleChoiceOption, the value(SPECIFIC) must be one
of the optionValues([INITIAL, LATEST]).
at
org.apache.seatunnel.api.configuration.util.ConfigValidator.validateSingleChoice(ConfigValidator.java:89)
at
org.apache.seatunnel.api.configuration.util.ConfigValidator.validate(ConfigValidator.java:64)
at
org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:129)
at
org.apache.seatunnel.api.table.factory.FactoryUtil.restoreAndPrepareSource(FactoryUtil.java:91)
... 8 more
```
### Zeta or Flink or Spark Version
zeta
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [Feature][Connector-V2] Add `file_extension` parameter for read/write file [seatunnel]
litiliu commented on code in PR #8769:
URL: https://github.com/apache/seatunnel/pull/8769#discussion_r1962948805
##
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/AbstractWriteStrategy.java:
##
@@ -224,8 +224,16 @@ public LinkedHashMap>
generatorPartitionDir(SeaTunnelRow se
public final String generateFileName(String transactionId) {
String fileNameExpression = fileSinkConfig.getFileNameExpression();
FileFormat fileFormat = fileSinkConfig.getFileFormat();
-String suffix = fileFormat.getSuffix();
-suffix = compressFormat.getCompressCodec() + suffix;
+String suffix;
+if (StringUtils.isNotEmpty(fileSinkConfig.getFilenameExtension())) {
+suffix =
+fileSinkConfig.getFilenameExtension().startsWith(".")
+? fileSinkConfig.getFilenameExtension()
+: "." + fileSinkConfig.getFilenameExtension();
Review Comment:
`.` may be widely used in this project, should we extract to the constant
instead of using the magic value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[I] [Bug] [Oracle-CDC] 报错Oracle日志保存点断裂，但是数据库日志查看正常，未发生断裂。 [seatunnel]
Quilian opened a new issue, #8772:
URL: https://github.com/apache/seatunnel/issues/8772
### Search before asking
- [x] I had searched in the
[issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22)
and found no similar issues.
### What happened
Oracle-cdc任务运行一个月发生报错，报错：Caused by: java.sql.SQLException: ORA-00310:
archived log contains sequence 14319; sequence 14316 requiredORA-00334:
archived log: '/OSdata/oradata_ncdb/SJZTNCDB/redo03.log'。但是检查数据库保存点是连续正常的：
-rw-r- 1 oracle oinstall 202161688 Feb 19 10:23 o1_mf_14313_mvbht21b_.arc
-rw-r- 1 oracle oinstall 109754880 Feb 19 10:23 o1_mf_14314_mvb5wy6_.arc
-rw-r- 1 oracle oinstall 53207744 Feb 19 10:23 o1_mf_14315_mvb57yh_.arc
-rw-r- 1 oracle oinstall 196684888 Feb 19 10:24 o1_mf_14316_mvb6r7x_.arc
-rw-r- 1 oracle oinstall 195910144 Feb 19 10:24 o1_mf_14317_mvb6j6t_.arc
-rw-r- 1 oracle oinstall 195910144 Feb 19 10:24 o1_mf_14318_mvb6jtw_.arc
-rw-r- 1 oracle oinstall 195910144 Feb 19 10:24 o1_mf_14319_mvb6vxm_.arc
-rw-r- 1 oracle oinstall 196322528 Feb 19 10:24 o1_mf_14320_mvb6y1h_.arc
-rw-r- 1 oracle oinstall 198780288 Feb 19 10:24 o1_mf_14322_mvb74vy_.arc
-rw-r- 1 oracle oinstall 201594728 Feb 19 10:25 o1_mf_14324_mvb8y0t_.arc
-rw-r- 1 oracle oinstall 93376824 Feb 19 10:35 o1_mf_14327_mvbvsyf_.arc
### SeaTunnel Version
2.3.9
### SeaTunnel Config
```conf
env{
execution.parallelism = 1
job.mode = "STREAMING"
checkpoint.interval =1000
}
source {
Oracle-CDC {
username = "xx"
password = "xxx"
database-names = ["db1"]
schema-names = ["sc1"]
table-names = ["db1.sc1.table1"]
connect.timeout.ms = 6
connect.max-retries = 5
base-url = "jdbc:oracle:thin:@xx.xxx.xxx.xx::db1"
}
}
sink {
Doris {
fenodes = "xx.xxx.xx.xxx:"
username = xxx
password = "x"
database = "xxx"
#table = ""
sink.enable-2pc = "false"
sink.label-prefix = "xxx_111"
sink.enable-delete = "true"
doris.config = {
format="json"
read_json_by_line="true"
}
}
}
```
### Running Command
```shell
bin/seatunnel.sh -c cdc.template --async
```
### Error Exception
```log
2025-02-19 10:29:10,909 ERROR [o.a.s.e.s.d.p.PhysicalVertex ]
[hz.main.generic-operation.thread-5] - Job db_oracle_cdc (933615545725485057),
Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Oracle-CDC]-SourceTask (1/1)]
end with state FAILED and Exception: java.lang.RuntimeException: One or more
fetchers have encountered exception
at
org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:147)
at
org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:167)
at
org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:93)
at
org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.pollNext(IncrementalSourceReader.java:119)
at
org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:159)
at
org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:127)
at
org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:169)
at
org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:132)
at
org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:694)
at
org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1019)
at
org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.RuntimeException: SplitFetcher thread 0 received
unexpected exception while polling the records
at
org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:165)
at
org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:81)
... 5 more
Caused by: org.apache.kafka.connect.errors.ConnectException: A
Re: [I] [Feature][Module Name] When synchronizing multiple tables in CDC, define a table name mapping to map the source table to target table names with different names [seatunnel]
sin70611 commented on issue #8756: URL: https://github.com/apache/seatunnel/issues/8756#issuecomment-2670119808 Thanks ,I used 2.3.8 and I have seen this in 2.3.9. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Feature][Module Name] When synchronizing multiple tables in CDC, define a table name mapping to map the source table to target table names with different names [seatunnel]
sin70611 closed issue #8756: [Feature][Module Name] When synchronizing multiple tables in CDC, define a table name mapping to map the source table to target table names with different names URL: https://github.com/apache/seatunnel/issues/8756 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve] update clickhouse connector config option [seatunnel]
liunaijie commented on PR #8755: URL: https://github.com/apache/seatunnel/pull/8755#issuecomment-2670125176 LGTM, Thanks! @lyne7-sc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [Fix][E2e] Optimized pom file name tag [seatunnel]
corgy-w opened a new pull request, #8770: URL: https://github.com/apache/seatunnel/pull/8770 ### Purpose of this pull request ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? ### Check list * [ ] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md) * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs * [ ] If you are contributing the connector code, please check that the following files are updated: 1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it 2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml) 3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml) 4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/) 5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config) * [ ] Update the [`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix assert connector assertion errors [seatunnel]
corgy-w commented on PR #8746: URL: https://github.com/apache/seatunnel/pull/8746#issuecomment-2670386880 > > Let's add a new check to make sure the assert can throw error normally. > > +1 , We need to verify that exceptions can be thrown correctly when an error occurs. get -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Feature][Connector-V2] Add `file_extension` parameter for read/write file [seatunnel]
litiliu commented on code in PR #8769:
URL: https://github.com/apache/seatunnel/pull/8769#discussion_r1962959943
##
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/BaseSinkConfig.java:
##
@@ -214,6 +214,13 @@ public class BaseSinkConfig {
.defaultValue(FileFormat.CSV)
.withDescription("File format type, e.g. csv, orc,
parquet, text");
+public static final Option FILENAME_EXTENSION =
+Options.key("filename_extension")
+.stringType()
+.noDefaultValue()
+.withDescription(
+"Override the default file name extensions with
custom file name extensions. E.g. `.xml`, `.json`, `dat`, `.customtype`");
Review Comment:
Should we add some validataions to this param? Like user input `.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [Feature][Config] Support custom config keys for encrypt/decrypt [seatunnel]
remones commented on PR #8739: URL: https://github.com/apache/seatunnel/pull/8739#issuecomment-2667843073 I noticed that every time I push a new commit, the GitHub Actions test workflow runs for over an hour, and some e2e test cases are not stable so it may fail sometimes. do you have any suggestions for this? cc @Hisoka-X @hailin0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Feature][JDBC] Improve the jdbc parallel-reader generated sql [seatunnel]
liunaijie commented on issue #8767: URL: https://github.com/apache/seatunnel/issues/8767#issuecomment-2668023369 Related code: - - https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/FixedChunkSplitter.java#L119 - https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/FixedChunkSplitter.java#L191 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix assert connector assertion errors [seatunnel]
zhangshenghang commented on PR #8746: URL: https://github.com/apache/seatunnel/pull/8746#issuecomment-2668029870 > Let's add a new check to make sure the assert can throw error normally. +1 , We need to verify that exceptions can be thrown correctly when an error occurs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Feature][Config] Support custom config keys for encrypt/decrypt [seatunnel]
hailin0 commented on PR #8739: URL: https://github.com/apache/seatunnel/pull/8739#issuecomment-2667944796 > I noticed that every time I push a new commit, the GitHub Actions test workflow runs for over an hour, and some e2e test cases are not stable so it may fail sometimes. do you have any suggestions for this? cc @Hisoka-X @hailin0 If you are interested, you can fix the unstable test case -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Feature][Kafka] Support native format read/write kafka record [seatunnel]
hailin0 commented on code in PR #8724: URL: https://github.com/apache/seatunnel/pull/8724#discussion_r1961236747 ## docs/en/connector-v2/sink/Kafka.md: ## @@ -30,21 +30,21 @@ They can be downloaded via install-plugin.sh or from the Maven central repositor ## Sink Options -| Name | Type | Required | Default | Description | -|---||--|-|-| -| topic | String | Yes | - | When the table is used as sink, the topic name is the topic to write data to. | -| bootstrap.servers | String | Yes | - | Comma separated list of Kafka brokers. | -| kafka.config | Map| No | - | In addition to the above parameters that must be specified by the `Kafka producer` client, the user can also specify multiple non-mandatory parameters for the `producer` client, covering [all the producer parameters specified in the official Kafka document](https://kafka.apache.org/documentation.html#producerconfigs). | -| semantics | String | No | NON | Semantics that can be chosen EXACTLY_ONCE/AT_LEAST_ONCE/NON, default NON. | -| partition_key_fields | Array | No | - | Configure which fields are used as the key of the kafka message. | -| partition | Int| No | - | We can specify the partition, all messages will be sent to this partition. | -| assign_partitions | Array | No | - | We can decide which partition to send based on the content of the message. The function of this parameter is to distribute information. | -| transaction_prefix| String | No | - | If semantic is specified as EXACTLY_ONCE, the producer will write all messages in a Kafka transaction,kafka distinguishes different transac
[PR] [Feature][Connector-V2] Add `file_extension` parameter for read/write file [seatunnel]
Hisoka-X opened a new pull request, #8769: URL: https://github.com/apache/seatunnel/pull/8769 ### Purpose of this pull request Add `file_extension` parameter for read/write file. When read file, it used to filter files. When write file, it used to override default file extension. ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? add new test. ### Check list * [ ] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md) * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs * [ ] If you are contributing the connector code, please check that the following files are updated: 1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it 2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml) 3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml) 4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/) 5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config) * [ ] Update the [`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Fix][Connector-V2] Fix possible data loss in certain scenarios of starrocks [seatunnel]
hailin0 commented on PR #8768: URL: https://github.com/apache/seatunnel/pull/8768#issuecomment-2668689941 Please add testcase (reproduce & verify) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(seatunnel) branch dev updated (6b6d8fae54 -> d78b143d90)
This is an automated email from the ASF dual-hosted git repository. wanghailin pushed a change to branch dev in repository https://gitbox.apache.org/repos/asf/seatunnel.git from 6b6d8fae54 [Improve] re-struct Zeta Engine config options (#8741) add d78b143d90 [Doc][Improve] support chinese [docs/zh/connector-v2/sink/Milvus.md] (#8727) No new revisions were added by this update. Summary of changes: docs/zh/connector-v2/sink/Milvus.md | 63 + 1 file changed, 63 insertions(+) create mode 100644 docs/zh/connector-v2/sink/Milvus.md
Re: [PR] [Doc][Improve] support chinese [docs/zh/connector-v2/sink/Milvus.md] [seatunnel]
hailin0 merged PR #8727: URL: https://github.com/apache/seatunnel/pull/8727 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(seatunnel) branch dev updated: [Doc][Improve] support chinese [docs/zh/connector-v2/sink/Socket.md] (#8729)
This is an automated email from the ASF dual-hosted git repository.
wanghailin pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new b83076c0df [Doc][Improve] support chinese
[docs/zh/connector-v2/sink/Socket.md] (#8729)
b83076c0df is described below
commit b83076c0df32483f2843100bf311ef38a76d03fe
Author: duzhendi <35251879+duzhe...@users.noreply.github.com>
AuthorDate: Wed Feb 19 21:42:25 2025 +0800
[Doc][Improve] support chinese [docs/zh/connector-v2/sink/Socket.md] (#8729)
---
docs/zh/connector-v2/sink/Socket.md | 79 +
1 file changed, 79 insertions(+)
diff --git a/docs/zh/connector-v2/sink/Socket.md
b/docs/zh/connector-v2/sink/Socket.md
new file mode 100644
index 00..22b180f714
--- /dev/null
+++ b/docs/zh/connector-v2/sink/Socket.md
@@ -0,0 +1,79 @@
+# Socket
+
+> Socket 数据接收器
+
+## 支持引擎
+
+> Spark
+> Flink
+> SeaTunnel Zeta
+
+## 主要特性
+
+- [ ] [exactly-once](../../concept/connector-v2-features.md)
+
+## 描述
+
+用于向Socket Server发送数据。两者都支持流媒体和批处理模式。
+
+>
例如，如果来自上游的数据是[`age:12，name:jared`]，则发送到Socket服务器的内容如下：`{"name"："jared"，"age"：17}`
+
+## Sink 选项
+
+| 名称 | 类型 | 是否必传 | 默认值 |
描述 |
+||-|--|-|-|
+| host | String | 是 | | socket 服务器主机
|
+| port | Integer | 是 | | socket 服务器端口
|
+| max_retries| Integer | 否 | 3 | 发送记录的重试失败次数
|
+| common-options | | 否 | - | 源插件常用参数，详见[Source common
Options]（../sink common-Options.md） |
+
+## 任务示例
+
+> 这是写入Socket端的随机生成数据
+
+```hocon
+env {
+ parallelism = 1
+ job.mode = "STREAMING"
+}
+
+source {
+ FakeSource {
+plugin_output = "fake"
+schema = {
+ fields {
+name = "string"
+age = "int"
+ }
+}
+ }
+}
+
+sink {
+ Socket {
+host = "localhost"
+port =
+ }
+}
+```
+
+* 启动端口侦听
+
+```shell
+nc -l -v
+```
+
+* 启动SeaTunnel任务
+
+* Socket 服务器控制台打印数据
+
+```text
+{"name":"jared","age":17}
+```
+
+## 更改日志
+
+### 2.2.0-beta 2022-09-26
+
+- 添加Socket 数据接收器
+
Re: [PR] [Doc][Improve] support chinese [docs/zh/connector-v2/sink/Socket.md] [seatunnel]
hailin0 merged PR #8729: URL: https://github.com/apache/seatunnel/pull/8729 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve] rabbit mq options [seatunnel]
hailin0 merged PR #8740: URL: https://github.com/apache/seatunnel/pull/8740 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve] update clickhouse connector config option [seatunnel]
lyne7-sc commented on PR #8755: URL: https://github.com/apache/seatunnel/pull/8755#issuecomment-2668837296 Hi , @liunaijie I noticed that the CI license header check failed in the previous PR. To address this, I've added a new commit to include the missing license headers for the following files: - `ClickhouseFileSinkOptions.java` - `ClickhouseSinkOptions.java` - `ClickhouseSourceOptions.java` Please review this update at your convenience. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(seatunnel) branch dev updated: [Improve] rabbit mq options (#8740)
This is an automated email from the ASF dual-hosted git repository.
wanghailin pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 4eec9be012 [Improve] rabbit mq options (#8740)
4eec9be012 is described below
commit 4eec9be012a685fa2be13590c5c967b7cd6eef3b
Author: Jarvis
AuthorDate: Wed Feb 19 21:42:57 2025 +0800
[Improve] rabbit mq options (#8740)
---
.../seatunnel/api/ConnectorOptionCheckTest.java| 2 -
.../rabbitmq/config/RabbitmqBaseOptions.java | 135 ++
.../seatunnel/rabbitmq/config/RabbitmqConfig.java | 271 -
.../rabbitmq/config/RabbitmqSinkOptions.java | 35 +++
.../rabbitmq/config/RabbitmqSourceOptions.java | 63 +
.../seatunnel/rabbitmq/sink/RabbitmqSink.java | 59 +
.../rabbitmq/sink/RabbitmqSinkFactory.java | 54 ++--
.../seatunnel/rabbitmq/source/RabbitmqSource.java | 62 +
.../rabbitmq/source/RabbitmqSourceFactory.java | 74 +++---
9 files changed, 371 insertions(+), 384 deletions(-)
diff --git
a/seatunnel-ci-tools/src/test/java/org/apache/seatunnel/api/ConnectorOptionCheckTest.java
b/seatunnel-ci-tools/src/test/java/org/apache/seatunnel/api/ConnectorOptionCheckTest.java
index 7dce5ee5f5..533c622602 100644
---
a/seatunnel-ci-tools/src/test/java/org/apache/seatunnel/api/ConnectorOptionCheckTest.java
+++
b/seatunnel-ci-tools/src/test/java/org/apache/seatunnel/api/ConnectorOptionCheckTest.java
@@ -171,7 +171,6 @@ public class ConnectorOptionCheckTest {
Set whiteList = new HashSet<>();
whiteList.add("JdbcSinkOptions");
whiteList.add("TypesenseSourceOptions");
-whiteList.add("RabbitmqSourceOptions");
whiteList.add("TypesenseSinkOptions");
whiteList.add("EmailSinkOptions");
whiteList.add("HudiSinkOptions");
@@ -188,7 +187,6 @@ public class ConnectorOptionCheckTest {
whiteList.add("MongodbSinkOptions");
whiteList.add("IoTDBSinkOptions");
whiteList.add("EasysearchSourceOptions");
-whiteList.add("RabbitmqSinkOptions");
whiteList.add("IcebergSourceOptions");
whiteList.add("HbaseSourceOptions");
whiteList.add("PaimonSourceOptions");
diff --git
a/seatunnel-connectors-v2/connector-rabbitmq/src/main/java/org/apache/seatunnel/connectors/seatunnel/rabbitmq/config/RabbitmqBaseOptions.java
b/seatunnel-connectors-v2/connector-rabbitmq/src/main/java/org/apache/seatunnel/connectors/seatunnel/rabbitmq/config/RabbitmqBaseOptions.java
new file mode 100644
index 00..ff71f860e3
--- /dev/null
+++
b/seatunnel-connectors-v2/connector-rabbitmq/src/main/java/org/apache/seatunnel/connectors/seatunnel/rabbitmq/config/RabbitmqBaseOptions.java
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.rabbitmq.config;
+
+import org.apache.seatunnel.api.configuration.Option;
+import org.apache.seatunnel.api.configuration.Options;
+import org.apache.seatunnel.api.options.ConnectorCommonOptions;
+
+public class RabbitmqBaseOptions extends ConnectorCommonOptions {
+
+public static final Option HOST =
+Options.key("host")
+.stringType()
+.noDefaultValue()
+.withDescription("the default host to use for
connections");
+
+public static final Option PORT =
+Options.key("port")
+.intType()
+.noDefaultValue()
+.withDescription("the default port to use for
connections");
+
+public static final Option VIRTUAL_HOST =
+Options.key("virtual_host")
+.stringType()
+.noDefaultValue()
+.withDescription("the virtual host to use when connecting
to the broker");
+
+public static final Option QUEUE_NAME =
+Options.key("queue_name")
+.stringType()
+.noDefaultValue()
+.withDescription("the queue to write the message to");
+
+public static final Option USERNAME =
+Options.
Re: [I] [Feature][Module Name] TiDB CDC data into Doris [seatunnel]
hailin0 commented on issue #8764: URL: https://github.com/apache/seatunnel/issues/8764#issuecomment-2668917656 Waiting for contributors to claim this issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Feature][MySQL 8.4 CDC Source] Any New version support MySQL 8.4 CDC source ? [seatunnel]
hailin0 closed issue #8675: [Feature][MySQL 8.4 CDC Source] Any New version support MySQL 8.4 CDC source ? URL: https://github.com/apache/seatunnel/issues/8675 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Feature][Module Name] When synchronizing multiple tables in CDC, define a table name mapping to map the source table to target table names with different names [seatunnel]
hailin0 commented on issue #8756: URL: https://github.com/apache/seatunnel/issues/8756#issuecomment-2668924278 https://github.com/apache/seatunnel/blob/dev/docs/en/transform-v2/table-rename.md https://github.com/apache/seatunnel/blob/dev/docs/en/transform-v2/field-rename.md -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] [connector-elasticsearch] Cannot replace capitalized dynamic index names [seatunnel]
hailin0 commented on issue #8752:
URL: https://github.com/apache/seatunnel/issues/8752#issuecomment-2668931333
What is ${OP_DAY}?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
