[GitHub] [seatunnel] immustard opened a new pull request, #5072: [Feature][connector-v2][Iceberg] sink plugin

2023-07-13 Thread via GitHub 


immustard opened a new pull request, #5072:
URL: https://github.com/apache/seatunnel/pull/5072

   
   
   ## Purpose of this pull request
   
   
   
   ## Check list
   
   * [x] Code changed are covered with tests, or it does not need tests for 
reason:
   * [x] If any new Jar binary package adding in your PR, please add License 
Notice according
 [New License 
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new 
feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If you are contributing the connector code, please check that the 
following files are updated:
 1. Update change log that in connector document. For more details you can 
refer to 
[connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2)
 2. Update 
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
 and add new connector information in it
 3. Update the pom file of 
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
   * [ ] Update the 
[`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] immustard commented on pull request #5072: [Feature][connector-v2][Iceberg] sink plugin

2023-07-13 Thread via GitHub 


immustard commented on PR #5072:
URL: https://github.com/apache/seatunnel/pull/5072#issuecomment-1633695031

   @hailin0 @EricJoy2048 PTAL


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] gaopeng666 closed pull request #4994: [Feature][Connector-V2][Clickhouse] Add clickhouse connector timezone key,default UTC

2023-07-13 Thread via GitHub 


gaopeng666 closed pull request #4994: [Feature][Connector-V2][Clickhouse] Add 
clickhouse connector timezone key,default UTC
URL: https://github.com/apache/seatunnel/pull/4994


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] lihjChina opened a new pull request, #5073: add DM upset support

2023-07-13 Thread via GitHub 


lihjChina opened a new pull request, #5073:
URL: https://github.com/apache/seatunnel/pull/5073

   
   
   ## Purpose of this pull request
   
   
   
   ## Check list
   
   * [ ] Code changed are covered with tests, or it does not need tests for 
reason:
   * [ ] If any new Jar binary package adding in your PR, please add License 
Notice according
 [New License 
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new 
feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If you are contributing the connector code, please check that the 
following files are updated:
 1. Update change log that in connector document. For more details you can 
refer to 
[connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2)
 2. Update 
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
 and add new connector information in it
 3. Update the pom file of 
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
   * [ ] Update the 
[`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] voyagertanyao opened a new issue, #5074: [Bug] [connector-hive] source hive to sink hdfs with parquet format error

2023-07-13 Thread via GitHub 


voyagertanyao opened a new issue, #5074:
URL: https://github.com/apache/seatunnel/issues/5074

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22)
 and found no similar issues.
   
   
   ### What happened
   
   using seatunnel 2.3.2 default engine zeta, collect hive table to hdfs, 
(CDH-6.2.1), run failed
   
   using connector-hive-2.3.2, connector-file-hadoop-2.3.2
   
   **_finally, I found why the error came out. Because both these two 
connnector jar have the class ParquetWriteStrategy.class
   
connector-hive-2.3.2/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/x.class_**
   
   
   
   ### SeaTunnel Version
   
   2.3.2
   
   ### SeaTunnel Config
   
   ```conf
   env {
 execution.parallelism = 1
 job.mode = "BATCH"
   }
   
   source {
 Hive {
   table_name = "pro_smb_snapshot.c_device"
   metastore_uri = "thrift://slave002:9083"
   read_partitions = ["dt=2023-07-12"]
 }
   }
   
   sink {
 HdfsFile {
   fs.defaultFS = "hdfs://hadoop001:8020"
   path = "/tmp/seatunnel/prof_smb/c_device"
   have_partition = no
   batch_size = 1000
   custom_filename = true
   file_name_expression = "${transactionId}_${now}"
   filename_time_format = "-MM-dd"
   file_format_type = "parquet"
 }
   }
   ```
   
   
   ### Running Command
   
   ```shell
   bin/seatunnel.sh --config hive.yaml -e local
   bin/seatunnel.sh --config hive.yaml -e cluster
   ```
   
   
   ### Error Exception
   
   ```log
   2023-07-13 14:22:27,104 INFO  com.hazelcast.core.LifecycleService - 
hz.client_1 [seatunnel-2.3.2] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is 
CLIENT_DISCONNECTED
2023-07-13 14:22:27,106 INFO  com.hazelcast.core.LifecycleService - 
hz.client_1 [seatunnel-2.3.2] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is 
SHUTDOWN
2023-07-13 14:22:27,106 INFO  
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - 
Closed SeaTunnel client..
2023-07-13 14:22:27,106 ERROR 
org.apache.seatunnel.core.starter.SeaTunnel - 


===


2023-07-13 14:22:27,106 ERROR 
org.apache.seatunnel.core.starter.SeaTunnel - Fatal Error, 

2023-07-13 14:22:27,106 ERROR 
org.apache.seatunnel.core.starter.SeaTunnel - Please submit bug report in 
https://github.com/apache/seatunnel/issues

2023-07-13 14:22:27,107 ERROR 
org.apache.seatunnel.core.starter.SeaTunnel - Reason:SeaTunnel job executed 
failed 

2023-07-13 14:22:27,109 ERROR 
org.apache.seatunnel.core.starter.SeaTunnel - Exception 
StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: 
SeaTunnel job executed failed
at 
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:188)
at 
org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at 
org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.util.concurrent.CompletionException: 
com.hazelcast.nio.serialization.HazelcastSerializationException: 
java.io.InvalidClassException: 
org.apache.seatunnel.connectors.seatunnel.file.sink.writer.ParquetWriteStrategy;
 local class incompatible: stream classdesc serialVersionUID = 
3902709188302080196, local class serialVersionUID = 3615746833076381224
at 
com.hazelcast.spi.impl.AbstractInvocationFuture.wrapInCompletionException(AbstractInvocationFuture.java:1347)
at 
com.hazelcast.spi.impl.AbstractInvocationFuture.cascadeException(AbstractInvocationFuture.java:1340)
at 
com.hazelcast.spi.impl.AbstractInvocationFuture.access$200(AbstractInvocationFuture.java:65)
at 
com.hazelcast.spi.impl.AbstractInvocationFuture$ApplyNode.execute(AbstractInvocationFuture.java:1478)
at 
com.hazelcast.spi.impl.AbstractInvocationFuture.unblockOtherNode(AbstractInvocationFuture.java:797)
at 
com.hazelcast.spi.impl.AbstractInvocationFuture.unblockAll(AbstractInvocationFuture.java:759)
at 
com.hazelcast.spi.impl.AbstractInvocationFuture.complete0(AbstractInvocationFuture.java:1235)
at 
com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionallyInternal(AbstractInvocationFuture.java:1223)
at 
com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionally(AbstractInvocationFuture.java:709)
at 
com.hazelcast.client.impl.spi.impl.ClientInvocation.completeExceptionally(ClientInvocation.java:294)
at 
com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:321)
at 
com.hazelcast.client

[GitHub] [seatunnel] voyagertanyao commented on issue #5074: [Bug] [connector-hive] source hive to sink hdfs with parquet format error

2023-07-13 Thread via GitHub 


voyagertanyao commented on issue #5074:
URL: https://github.com/apache/seatunnel/issues/5074#issuecomment-1633785462

   This is because both connectors have the same file writer class in 
connector-hive and connector-file-hdfs, the same error for orc\json format. So 
why we should avoid this happened? hive to hdfs


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] jobmission opened a new issue, #5075: [Feature][Connector-V2-RabbitMQ] support plain text payload

2023-07-13 Thread via GitHub 


jobmission opened a new issue, #5075:
URL: https://github.com/apache/seatunnel/issues/5075

   ### Search before asking
   
   - [X] I had searched in the 
[feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22)
 and found no similar feature requirement.
   
   
   ### Description
   
   Rabbitmq source connector should support non-json schema payload, like plain 
text or xml  , and so on.
   
   ### Usage Scenario
   
   the rabbitmq message is plain text, not json ,no shcema.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] EricJoy2048 closed pull request #4626: [Feature][Jdbc-Connector] Add OceanBase Connector

2023-07-13 Thread via GitHub 


EricJoy2048 closed pull request #4626: [Feature][Jdbc-Connector] Add OceanBase 
Connector
URL: https://github.com/apache/seatunnel/pull/4626


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] EricJoy2048 commented on pull request #5073: [Feature][JDBC Sink] Add DM upset support

2023-07-13 Thread via GitHub 


EricJoy2048 commented on PR #5073:
URL: https://github.com/apache/seatunnel/pull/5073#issuecomment-1633811298

   Please add e2e for this pr.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] whhe opened a new issue, #5076: [Feature][Connector-V2][Jdbc] Add OceanBase catalog

2023-07-13 Thread via GitHub 


whhe opened a new issue, #5076:
URL: https://github.com/apache/seatunnel/issues/5076

   ### Search before asking
   
   - [X] I had searched in the 
[feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22)
 and found no similar feature requirement.
   
   
   ### Description
   
   Add oceanbase catalog to support DataSaveMode, which  is necessary for 
multiple table job.
   
   ### Usage Scenario
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] jobmission opened a new issue, #5077: [Feature][ConnectorV2-http-sink] support custom post body

2023-07-13 Thread via GitHub 


jobmission opened a new issue, #5077:
URL: https://github.com/apache/seatunnel/issues/5077

   ### Search before asking
   
   - [X] I had searched in the 
[feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22)
 and found no similar feature requirement.
   
   
   ### Description
   
   support custom http body payload json schema maybe like http source 
connnector:
   { 
 "store": {
   "book": [
 {
   "category": "reference",
   "author": "Nigel Rees",
   "title": "Sayings of the Century",
   "price": 8.95
 },
 {
   "category": "fiction",
   "author": "Evelyn Waugh",
   "title": "Sword of Honour",
   "price": 12.99
 }
   ],
   "bicycle": {
 "color": "red",
 "price": 19.95
   }
 },
 "expensive": 10
   }
   
   ### Usage Scenario
   
   the source input is mysql table, but the target http api need  body format 
like below,
   {
 "time":"2023-07-13 12:12:12",
  "total":3,
  "data":[
{"k":"v"}, {}, {}
  ]
   }
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] liunaijie closed pull request #4976: [Feature][Connector] support avro format

2023-07-13 Thread via GitHub 


liunaijie closed pull request #4976: [Feature][Connector] support avro format 
URL: https://github.com/apache/seatunnel/pull/4976


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel-web] zhuchaoling opened a new pull request, #68: Fix: login redirect path

2023-07-13 Thread via GitHub 


zhuchaoling opened a new pull request, #68:
URL: https://github.com/apache/seatunnel-web/pull/68

   
   
   ## Purpose of this pull request
   
   
   
   ## Check list
   
   * [ ] Code changed are covered with tests, or it does not need tests for 
reason:
   * [ ] If any new Jar binary package adding in your PR, please add License 
Notice according
 [New License 
Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new 
feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] gaopeng666 opened a new pull request, #5078: [Feature][Connector-V2][Clickhouse] Add clickhouse connector time zone key,default system time zone

2023-07-13 Thread via GitHub 


gaopeng666 opened a new pull request, #5078:
URL: https://github.com/apache/seatunnel/pull/5078

   
   
   ## Purpose of this pull request
   
   
   
   ## Check list
   
   * [ ] Code changed are covered with tests, or it does not need tests for 
reason:
   * [ ] If any new Jar binary package adding in your PR, please add License 
Notice according
 [New License 
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new 
feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If you are contributing the connector code, please check that the 
following files are updated:
 1. Update change log that in connector document. For more details you can 
refer to 
[connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2)
 2. Update 
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
 and add new connector information in it
 3. Update the pom file of 
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
   * [ ] Update the 
[`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] zhong16602192407 commented on issue #4689: [Feature][CDC] Oracle CDC support

2023-07-13 Thread via GitHub 


zhong16602192407 commented on issue #4689:
URL: https://github.com/apache/seatunnel/issues/4689#issuecomment-1633965800

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] sprybee closed issue #5053: [ question] Connector error and prompt NullPointERROR

2023-07-13 Thread via GitHub 


sprybee closed issue #5053: [ question]  Connector error  and  prompt 
NullPointERROR
URL: https://github.com/apache/seatunnel/issues/5053


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] arjun-guptha opened a new issue, #5079: CDC in Seatunnel Phoenic

2023-07-13 Thread via GitHub 


arjun-guptha opened a new issue, #5079:
URL: https://github.com/apache/seatunnel/issues/5079

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22)
 and found no similar issues.
   
   
   ### What happened
   
   I successfully integrated Seatunnel with Phoenix DB and conducted a test on 
the batch mode job, which yielded positive results. However, when attempting to 
test the streaming job of Seatunnel with Phoenix, I encountered the same 
outcome as the batch mode job. Streaming is not working in phoenix. I would 
greatly appreciate your assistance in resolving this issue.
   
   ### SeaTunnel Version
   
   2.3.1
   
   ### SeaTunnel Config
   
   ```conf
   env{
 #parallelism = 3
 job.mode = "STREAMING"
 job.name="test_hbase_streaming"
   }
   
   source {
   Jdbc {
   driver = org.apache.phoenix.jdbc.PhoenixDriver
   url = "jdbc:phoenix:instance-1:/hbase-unsecure"
   query = "select ROWID, MSISDN, ID, STATUS_FIELD from 
TEST_MERGE_SPLIT"
   }
   }
   
   sink{
   Jdbc {
   driver = org.apache.phoenix.jdbc.PhoenixDriver
   url = "jdbc:phoenix:instance-2:/hbase-unsecure"
   query = "upsert into TEST_MERGE_SPLIT(ROWID, MSISDN, ID, 
STATUS_FIELD) values(?,?,?,?)"
   }
   }
   ```
   
   
   ### Running Command
   
   ```shell
   ./bin/seatunnel.sh --config ./config/hbase-batch-3.conf -e local
   ```
   
   
   ### Error Exception
   
   ```log
   Although no error exceptions are occurring, the streaming mode is behaving 
similarly to the batch mode. Specifically, it is transferring all the records 
from the source to the sink table, and the job is subsequently marked as 
completed.
   ```
   
   
   ### Flink or Spark Version
   
   _No response_
   
   ### Java or Scala Version
   
   _No response_
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel-web] zhangchengming601 opened a new pull request, #69: [ADD][Switcher]Add SqlServerDataSourceConfigSwitcher And MysqlDatasourceConfigSwitcher

2023-07-13 Thread via GitHub 


zhangchengming601 opened a new pull request, #69:
URL: https://github.com/apache/seatunnel-web/pull/69

   
   
   ## Purpose of this pull request
   
   
   
   ## Check list
   
   * [ ] Code changed are covered with tests, or it does not need tests for 
reason:
   * [ ] If any new Jar binary package adding in your PR, please add License 
Notice according
 [New License 
Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new 
feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] gaopeng666 opened a new pull request, #5080: [Fix][Connector-V2][Clickhouse] fix clickhouse source connector read Nullable() type is not null,exam…

2023-07-13 Thread via GitHub 


gaopeng666 opened a new pull request, #5080:
URL: https://github.com/apache/seatunnel/pull/5080

   …ple:Nullable(Float64) while value is null the result is 0.0
   
   
   
   ## Purpose of this pull request
   
   
   
   ## Check list
   
   * [ ] Code changed are covered with tests, or it does not need tests for 
reason:
   * [ ] If any new Jar binary package adding in your PR, please add License 
Notice according
 [New License 
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new 
feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If you are contributing the connector code, please check that the 
following files are updated:
 1. Update change log that in connector document. For more details you can 
refer to 
[connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2)
 2. Update 
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
 and add new connector information in it
 3. Update the pom file of 
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
   * [ ] Update the 
[`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel-web] EricJoy2048 merged pull request #69: [ADD][Switcher]Add SqlServerDataSourceConfigSwitcher And MysqlDatasourceConfigSwitcher

2023-07-13 Thread via GitHub 


EricJoy2048 merged PR #69:
URL: https://github.com/apache/seatunnel-web/pull/69


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[seatunnel-web] branch add_canvas_job_define updated: [ADD][Switcher]Add SqlServerDataSourceConfigSwitcher And MysqlDatasourceConfigSwitcher (#69)

2023-07-13 Thread gaojun2048 
This is an automated email from the ASF dual-hosted git repository.

gaojun2048 pushed a commit to branch add_canvas_job_define
in repository https://gitbox.apache.org/repos/asf/seatunnel-web.git


The following commit(s) were added to refs/heads/add_canvas_job_define by this 
push:
 new 5970c19d [ADD][Switcher]Add SqlServerDataSourceConfigSwitcher And 
MysqlDatasourceConfigSwitcher (#69)
5970c19d is described below

commit 5970c19d23cbf42c9637223226bb6edb43ccf2a1
Author: zhangchengming601 <86779821+zhangchengming...@users.noreply.github.com>
AuthorDate: Thu Jul 13 19:54:05 2023 +0800

[ADD][Switcher]Add SqlServerDataSourceConfigSwitcher And 
MysqlDatasourceConfigSwitcher (#69)

* [ADD][SYNC TASK DEFINITION]

* [ADD][SYNC TASK DEFINITION] POM File

* [ADD][FUNCTION]Add synchronization task instance function

* [ADD][FUNCTION]Add synchronization task instance function

* [ADD][Switcher]Add SqlServerDataSourceConfigSwitcher And 
MysqlDatasourceConfigSwitcher

* [ADD][Switcher]Add SqlServerDataSourceConfigSwitcher And 
MysqlDatasourceConfigSwitcher
---
 pom.xml|  7 
 seatunnel-server/seatunnel-app/pom.xml | 10 +++--
 .../impl/MysqlDatasourceConfigSwitcher.java| 15 
 .../impl/SqlServerDataSourceConfigSwitcher.java| 44 +-
 4 files changed, 70 insertions(+), 6 deletions(-)

diff --git a/pom.xml b/pom.xml
index dc3af7dd..ef6b0822 100644
--- a/pom.xml
+++ b/pom.xml
@@ -433,6 +433,7 @@
 org.apache.seatunnel
 connector-jdbc
 ${seatunnel-framework.version}
+test
 
 
 
@@ -580,6 +581,7 @@
 org.apache.seatunnel
 connector-kafka
 ${seatunnel-framework.version}
+test
 
 
 org.apache.seatunnel
@@ -633,6 +635,7 @@
 org.apache.seatunnel
 connector-jdbc
 ${seatunnel-framework.version}
+test
 
 
 org.apache.seatunnel
@@ -854,11 +857,13 @@
 org.apache.seatunnel
 connector-cdc-mysql
 ${seatunnel-framework.version}
+test
 
 
 org.apache.seatunnel
 connector-cdc-sqlserver
 ${seatunnel-framework.version}
+test
 
 
 org.apache.seatunnel
@@ -880,11 +885,13 @@
 org.apache.seatunnel
 connector-cdc-sqlserver
 ${seatunnel-framework.version}
+test
 
 
 org.apache.seatunnel
 connector-cdc-mysql
 ${seatunnel-framework.version}
+test
 
 
 org.projectlombok
diff --git a/seatunnel-server/seatunnel-app/pom.xml 
b/seatunnel-server/seatunnel-app/pom.xml
index f6e585cd..0d330ec3 100644
--- a/seatunnel-server/seatunnel-app/pom.xml
+++ b/seatunnel-server/seatunnel-app/pom.xml
@@ -93,7 +93,7 @@
 
 org.apache.seatunnel
 connector-common
-
+test
 
 
 
@@ -104,7 +104,7 @@
 
 org.apache.seatunnel
 connector-console
-
+test
 
 
 
@@ -116,7 +116,7 @@
 
 org.apache.seatunnel
 connector-kafka
-
+test
 
 
 org.apache.seatunnel
@@ -161,15 +161,17 @@
 
 org.apache.seatunnel
 connector-jdbc
-
+test
 
 
 org.apache.seatunnel
 connector-cdc-mysql
+test
 
 
 org.apache.seatunnel
 connector-cdc-sqlserver
+test
 
 
 org.apache.seatunnel
diff --git 
a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/thirdparty/datasource/impl/MysqlDatasourceConfigSwitcher.java
 
b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/thirdparty/datasource/impl/MysqlDatasourceConfigSwitcher.java
index 8f27d2de..a034a643 100644
--- 
a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/thirdparty/datasource/impl/MysqlDatasourceConfigSwitcher.java
+++ 
b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/thirdparty/datasource/impl/MysqlDatasourceConfigSwitcher.java
@@ -17,8 +17,23 @@
 
 package org.apache.seatunnel.app.thirdparty.datasource.impl;
 
+import java.util.Optional;
+
 public class MysqlDatasourceConfigSwitcher extends 
BaseJdbcDataSourceConfigSwitcher {
 public static MysqlDatasourceConfigSwitcher INSTANCE = new 
MysqlDatasourceConfigSwitcher();
+private static final String CATALOG_NAME = "MySQL";
 
 private MysqlD

[GitHub] [seatunnel] gaopeng666 commented on pull request #5078: [Feature][Connector-V2][Clickhouse] Add clickhouse connector time zone key,default system time zone

2023-07-13 Thread via GitHub 


gaopeng666 commented on PR #5078:
URL: https://github.com/apache/seatunnel/pull/5078#issuecomment-1634132430

[Bug] [zeta] clickhouse to Doris，8 hours difference in datetime type #4935 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] NickYoungPeng commented on pull request #4803: [Feature][Connector-V2] jdbc connector supports Kingbase database

2023-07-13 Thread via GitHub 


NickYoungPeng commented on PR #4803:
URL: https://github.com/apache/seatunnel/pull/4803#issuecomment-1634137905

   > 
![image](https://user-images.githubusercontent.com/29588249/253173019-3a221d3e-3664-417e-b800-f45c66b65cd0.png)
 It can fix `ansi_x3.4-1968` error.
   
   thx sir


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel-web] EricJoy2048 merged pull request #68: Fix: login redirect path

2023-07-13 Thread via GitHub 


EricJoy2048 merged PR #68:
URL: https://github.com/apache/seatunnel-web/pull/68


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[seatunnel-web] branch add_canvas_job_define updated: Fix: login redirect path (#68)

2023-07-13 Thread gaojun2048 
This is an automated email from the ASF dual-hosted git repository.

gaojun2048 pushed a commit to branch add_canvas_job_define
in repository https://gitbox.apache.org/repos/asf/seatunnel-web.git


The following commit(s) were added to refs/heads/add_canvas_job_define by this 
push:
 new 28c18f2d Fix: login redirect path (#68)
28c18f2d is described below

commit 28c18f2d4f32cc8da15f9e863b3740de23f04d1c
Author: chaoling <126888254+zhuchaol...@users.noreply.github.com>
AuthorDate: Thu Jul 13 21:04:30 2023 +0800

Fix: login redirect path (#68)
---
 seatunnel-ui/index.html  |   1 +
 seatunnel-ui/public/logo-favicon.png | Bin 0 -> 7975 bytes
 seatunnel-ui/src/views/login/use-form.ts |   2 +-
 3 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/seatunnel-ui/index.html b/seatunnel-ui/index.html
index 9c457bea..9ffc8128 100644
--- a/seatunnel-ui/index.html
+++ b/seatunnel-ui/index.html
@@ -20,6 +20,7 @@
 
   
 
+
 
 SeaTunnel Admin
   
diff --git a/seatunnel-ui/public/logo-favicon.png 
b/seatunnel-ui/public/logo-favicon.png
new file mode 100644
index ..bb225968
Binary files /dev/null and b/seatunnel-ui/public/logo-favicon.png differ
diff --git a/seatunnel-ui/src/views/login/use-form.ts 
b/seatunnel-ui/src/views/login/use-form.ts
index b432883c..79e23056 100644
--- a/seatunnel-ui/src/views/login/use-form.ts
+++ b/seatunnel-ui/src/views/login/use-form.ts
@@ -57,7 +57,7 @@ export function useForm() {
   const handleLogin = () => {
 userLogin({ ...state.loginForm }).then((res: any) => {
   userStore.setUserInfo(res)
-  router.push({ path: '/data-pipes' })
+  router.push({ name: 'tasks' })
 })
   }

[GitHub] [seatunnel] NickYoungPeng commented on pull request #4803: [Feature][Connector-V2] jdbc connector supports Kingbase database

2023-07-13 Thread via GitHub 


NickYoungPeng commented on PR #4803:
URL: https://github.com/apache/seatunnel/pull/4803#issuecomment-1634345386

   https://github.com/apache/seatunnel/assets/72905543/bb4f0926-6f20-4dd8-bc68-f8a46cab6fc3";>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] liugddx opened a new pull request, #5081: [doc][zeta]add savepoint doc

2023-07-13 Thread via GitHub 


liugddx opened a new pull request, #5081:
URL: https://github.com/apache/seatunnel/pull/5081

   
   
   ## Purpose of this pull request
   
   
   
   ## Check list
   
   * [ ] Code changed are covered with tests, or it does not need tests for 
reason:
   * [ ] If any new Jar binary package adding in your PR, please add License 
Notice according
 [New License 
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new 
feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If you are contributing the connector code, please check that the 
following files are updated:
 1. Update change log that in connector document. For more details you can 
refer to 
[connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2)
 2. Update 
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
 and add new connector information in it
 3. Update the pom file of 
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
   * [ ] Update the 
[`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] github-actions[bot] commented on issue #4912: [Bug] [Module Name] Bug title

2023-07-13 Thread via GitHub 


github-actions[bot] commented on issue #4912:
URL: https://github.com/apache/seatunnel/issues/4912#issuecomment-1635088149

   This issue has been automatically marked as stale because it has not had 
recent activity for 30 days. It will be closed in next 7 days if no further 
activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] github-actions[bot] closed issue #4893: Incremental Synchronization of Oracle Heterogeneous Data

2023-07-13 Thread via GitHub 


github-actions[bot] closed issue #4893: Incremental Synchronization of Oracle 
Heterogeneous Data
URL: https://github.com/apache/seatunnel/issues/4893


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] github-actions[bot] closed issue #4879: [Bug] [seatunnel-connectors-v2] java.lang.NoClassDefFoundError: org/apache/spark/sql/connector/write/Write

2023-07-13 Thread via GitHub 


github-actions[bot] closed issue #4879: [Bug] [seatunnel-connectors-v2] 
java.lang.NoClassDefFoundError: org/apache/spark/sql/connector/write/Write
URL: https://github.com/apache/seatunnel/issues/4879


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] github-actions[bot] commented on issue #4676: [Feature][Zeta] hdfs checkpoint support HA configuration

2023-07-13 Thread via GitHub 


github-actions[bot] commented on issue #4676:
URL: https://github.com/apache/seatunnel/issues/4676#issuecomment-1635088307

   This issue has been closed because it has not received response for too long 
time. You could reopen it if you encountered similar problems in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] github-actions[bot] commented on issue #4879: [Bug] [seatunnel-connectors-v2] java.lang.NoClassDefFoundError: org/apache/spark/sql/connector/write/Write

2023-07-13 Thread via GitHub 


github-actions[bot] commented on issue #4879:
URL: https://github.com/apache/seatunnel/issues/4879#issuecomment-1635088248

   This issue has been closed because it has not received response for too long 
time. You could reopen it if you encountered similar problems in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] github-actions[bot] closed issue #4676: [Feature][Zeta] hdfs checkpoint support HA configuration

2023-07-13 Thread via GitHub 


github-actions[bot] closed issue #4676: [Feature][Zeta] hdfs checkpoint support 
HA configuration
URL: https://github.com/apache/seatunnel/issues/4676


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] Tangsonghuai closed issue #5069: [Bug] [Module Name] Cannot run in Centos6.6's env

2023-07-13 Thread via GitHub 


Tangsonghuai closed issue #5069: [Bug] [Module Name] Cannot run in Centos6.6's 
env
URL: https://github.com/apache/seatunnel/issues/5069


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] Carl-Zhou-CN opened a new issue, #5082: [Feature][connector-v2][oracle]Support source oracle cdc

2023-07-13 Thread via GitHub 


Carl-Zhou-CN opened a new issue, #5082:
URL: https://github.com/apache/seatunnel/issues/5082

   ### Search before asking
   
   - [X] I had searched in the 
[feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22)
 and found no similar feature requirement.
   
   
   ### Description
   
   Support source oracle cdc 
   
   ### Usage Scenario
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] Carl-Zhou-CN commented on issue #5082: [Feature][connector-v2][oracle]Support source oracle cdc

2023-07-13 Thread via GitHub 


Carl-Zhou-CN commented on issue #5082:
URL: https://github.com/apache/seatunnel/issues/5082#issuecomment-1635130470

   I will try to complete this source, please assign it to me


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] liunaijie opened a new pull request, #5083: [Feature] support compress on Text File Read

2023-07-13 Thread via GitHub 


liunaijie opened a new pull request, #5083:
URL: https://github.com/apache/seatunnel/pull/5083

   
   
   ## Purpose of this pull request
   support compress on local text file read. 
   
   
   
   
   ## Check list
   
   * [x] Code changed are covered with tests, or it does not need tests for 
reason:
   * [ ] If any new Jar binary package adding in your PR, please add License 
Notice according
 [New License 
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new 
feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If you are contributing the connector code, please check that the 
following files are updated:
 1. Update change log that in connector document. For more details you can 
refer to 
[connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2)
 2. Update 
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
 and add new connector information in it
 3. Update the pom file of 
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
   * [ ] Update the 
[`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] liunaijie opened a new pull request, #5084: [Feature] support avro format

2023-07-13 Thread via GitHub 


liunaijie opened a new pull request, #5084:
URL: https://github.com/apache/seatunnel/pull/5084

   - added avro format and UT
   - kafka support avro format and e2e test
   
   
   
   ## Purpose of this pull request
   close #4821 
   same pr with https://github.com/apache/seatunnel/pull/5064
   
   
   
   
   
   
   
   ## Check list
   
   * [x] Code changed are covered with tests, or it does not need tests for 
reason:
   * [x] If any new Jar binary package adding in your PR, please add License 
Notice according
 [New License 
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
   * [x] If necessary, please update the documentation to describe the new 
feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If you are contributing the connector code, please check that the 
following files are updated:
 1. Update change log that in connector document. For more details you can 
refer to 
[connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2)
 2. Update 
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
 and add new connector information in it
 3. Update the pom file of 
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
   * [ ] Update the 
[`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] liugddx closed pull request #5073: [Feature][JDBC Sink] Add DM upset support

2023-07-13 Thread via GitHub 


liugddx closed pull request #5073: [Feature][JDBC Sink] Add DM upset support
URL: https://github.com/apache/seatunnel/pull/5073


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] liugddx commented on pull request #5072: [Feature][connector-v2][Iceberg] sink plugin

2023-07-13 Thread via GitHub 


liugddx commented on PR #5072:
URL: https://github.com/apache/seatunnel/pull/5072#issuecomment-1635193941

   Thanks for your contribution, Here are some comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] liugddx commented on a diff in pull request #5072: [Feature][connector-v2][Iceberg] sink plugin

2023-07-13 Thread via GitHub 


liugddx commented on code in PR #5072:
URL: https://github.com/apache/seatunnel/pull/5072#discussion_r1263239895


##
seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/config/SinkConfig.java:
##
@@ -0,0 +1,48 @@
+package org.apache.seatunnel.connectors.seatunnel.iceberg.config;

Review Comment:
   Please add apache license.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] liugddx commented on a diff in pull request #5072: [Feature][connector-v2][Iceberg] sink plugin

2023-07-13 Thread via GitHub 


liugddx commented on code in PR #5072:
URL: https://github.com/apache/seatunnel/pull/5072#discussion_r1263242679


##
seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/data/DataConverter.java:
##
@@ -0,0 +1,19 @@
+package org.apache.seatunnel.connectors.seatunnel.iceberg.data;
+
+import org.apache.iceberg.data.Record;
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+
+/**
+ *
+ *
+ * @author mustard
+ * @version 1.0
+ * Create by 2023-07-05
+ */

Review Comment:
   The same above



##
seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/config/SinkConfig.java:
##
@@ -0,0 +1,48 @@
+package org.apache.seatunnel.connectors.seatunnel.iceberg.config;
+
+import lombok.Getter;
+import lombok.ToString;
+import org.apache.seatunnel.api.configuration.Option;
+import org.apache.seatunnel.api.configuration.Options;
+import org.apache.seatunnel.shade.com.typesafe.config.Config;
+
+/**
+ *
+ *
+ * @author mustard
+ * @version 1.0
+ * Create by 2023-07-05
+ */

Review Comment:
   Remove these.



##
seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/data/DefaultDataConverter.java:
##
@@ -0,0 +1,219 @@
+package org.apache.seatunnel.connectors.seatunnel.iceberg.data;

Review Comment:
   The same above



##
seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/data/DefaultDataConverter.java:
##
@@ -0,0 +1,219 @@
+package org.apache.seatunnel.connectors.seatunnel.iceberg.data;
+
+import lombok.NonNull;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.seatunnel.api.table.type.*;
+import org.apache.seatunnel.common.exception.CommonErrorCode;
+import 
org.apache.seatunnel.connectors.seatunnel.iceberg.exception.IcebergConnectorException;
+
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.time.*;
+import java.util.*;
+
+import static org.apache.iceberg.types.Type.TypeID.*;
+
+/**
+ *
+ *
+ * @author mustard
+ * @version 1.0
+ * Create by 2023-07-05
+ */

Review Comment:
   The same above



##
seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/data/DefaultDataConverter.java:
##
@@ -0,0 +1,219 @@
+package org.apache.seatunnel.connectors.seatunnel.iceberg.data;
+
+import lombok.NonNull;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.seatunnel.api.table.type.*;
+import org.apache.seatunnel.common.exception.CommonErrorCode;
+import 
org.apache.seatunnel.connectors.seatunnel.iceberg.exception.IcebergConnectorException;
+
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.time.*;
+import java.util.*;
+
+import static org.apache.iceberg.types.Type.TypeID.*;
+
+/**
+ *
+ *
+ * @author mustard
+ * @version 1.0
+ * Create by 2023-07-05
+ */
+@RequiredArgsConstructor

Review Comment:
   The same above



##
seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/IcebergSink.java:
##
@@ -0,0 +1,118 @@
+package org.apache.seatunnel.connectors.seatunnel.iceberg.sink;
+
+import com.google.auto.service.AutoService;
+import lombok.SneakyThrows;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+import org.apache.seatunnel.api.common.PrepareFailException;
+import org.apache.seatunnel.api.sink.SeaTunnelSink;
+import org.apache.seatunnel.api.sink.SinkWriter;
+import org.apache.seatunnel.api.table.catalog.CatalogTableUtil;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
+import org.apache.seatunnel.common.config.CheckConfigUtil;
+import org.apache.seatunnel.common.config.CheckResult;
+import 
org.apache.seatunnel.connectors.seatunnel.common.sink.AbstractSimpleSink;
+import 
org.apache.seatunnel.connectors.seatunnel.common.sink.AbstractSinkWriter;
+import org.apache.seatunnel.connectors.seatunnel.iceberg.IcebergTableLoader;
+import org.apache.seatunnel.connectors.seatunnel.iceberg.config.CommonConfig;
+import org.apache.seatunnel.connectors.seatunnel.iceberg.config.SinkConfig;
+import 
org.apache.seatunnel.connectors.seatunnel.iceberg.data.IcebergTypeMapper;
+import 
org.apache.seatunnel.connectors.seatunnel.iceberg.sink.writer.IcebergSink

[GitHub] [seatunnel] liugddx commented on pull request #5072: [Feature][connector-v2][Iceberg] sink plugin

2023-07-13 Thread via GitHub 


liugddx commented on PR #5072:
URL: https://github.com/apache/seatunnel/pull/5072#issuecomment-1635199431

   Run `mvn spotless:apply` before submitting the code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] liugddx commented on pull request #5072: [Feature][connector-v2][Iceberg] sink plugin

2023-07-13 Thread via GitHub 


liugddx commented on PR #5072:
URL: https://github.com/apache/seatunnel/pull/5072#issuecomment-163512

   If you are contributing the connector code, please check that the following 
files are updated:
   
   1. Update change log that in connector document. For more details you can 
refer to 
[connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2)
   
   2. Update 
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
 and add new connector information in it
   
   3. Update the pom file of 
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
Update the 
[release-note](https://github.com/apache/seatunnel/blob/dev/release-note.md).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[seatunnel] branch dev updated: [Bugfix]fix clickhouse source connector read Nullable() type is not null,example:Nullable(Float64) while value is null the result is 0.0 (#5080)

2023-07-13 Thread liugddx 
This is an automated email from the ASF dual-hosted git repository.

liugddx pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git


The following commit(s) were added to refs/heads/dev by this push:
 new cf3d0bba2 [Bugfix]fix clickhouse source connector read Nullable() type 
is not null,example:Nullable(Float64) while value is null the result is 0.0 
(#5080)
cf3d0bba2 is described below

commit cf3d0bba2e64869f9683f5e4ac0df1abed64310e
Author: 阿丙 <50567478+gaopeng...@users.noreply.github.com>
AuthorDate: Fri Jul 14 11:07:46 2023 +0800

[Bugfix]fix clickhouse source connector read Nullable() type is not 
null,example:Nullable(Float64) while value is null the result is 0.0 (#5080)
---
 .../seatunnel/clickhouse/source/ClickhouseSourceReader.java  | 12 
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git 
a/seatunnel-connectors-v2/connector-clickhouse/src/main/java/org/apache/seatunnel/connectors/seatunnel/clickhouse/source/ClickhouseSourceReader.java
 
b/seatunnel-connectors-v2/connector-clickhouse/src/main/java/org/apache/seatunnel/connectors/seatunnel/clickhouse/source/ClickhouseSourceReader.java
index 8d3de88ab..bb4d3905e 100644
--- 
a/seatunnel-connectors-v2/connector-clickhouse/src/main/java/org/apache/seatunnel/connectors/seatunnel/clickhouse/source/ClickhouseSourceReader.java
+++ 
b/seatunnel-connectors-v2/connector-clickhouse/src/main/java/org/apache/seatunnel/connectors/seatunnel/clickhouse/source/ClickhouseSourceReader.java
@@ -83,10 +83,14 @@ public class ClickhouseSourceReader implements 
SourceReader

[GitHub] [seatunnel] liugddx merged pull request #5078: [Feature][Connector-V2][Clickhouse] Add clickhouse connector time zone key,default system time zone

2023-07-13 Thread via GitHub 


liugddx merged PR #5078:
URL: https://github.com/apache/seatunnel/pull/5078


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[seatunnel] branch dev updated: [Feature][Connector-V2][Clickhouse] Add clickhouse connector time zone key,default system time zone (#5078)

2023-07-13 Thread liugddx 
This is an automated email from the ASF dual-hosted git repository.

liugddx pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git


The following commit(s) were added to refs/heads/dev by this push:
 new 309b58d12 [Feature][Connector-V2][Clickhouse] Add clickhouse connector 
time zone key,default system time zone (#5078)
309b58d12 is described below

commit 309b58d12d30f6ec7d523c2b9afec9181d3c4c6e
Author: 阿丙 <50567478+gaopeng...@users.noreply.github.com>
AuthorDate: Fri Jul 14 11:08:04 2023 +0800

[Feature][Connector-V2][Clickhouse] Add clickhouse connector time zone 
key,default system time zone (#5078)

* Add clickhouse connector time zone key,default system time zone

* Modify the document and add clickhouse server_time_zone configuration
---
 docs/en/connector-v2/source/Clickhouse.md  | 22 ++
 .../clickhouse/config/ClickhouseConfig.java| 10 ++
 .../clickhouse/sink/client/ClickhouseSink.java |  4 
 .../clickhouse/sink/file/ClickhouseFileSink.java   |  2 ++
 .../clickhouse/source/ClickhouseSource.java| 11 +++
 .../seatunnel/clickhouse/util/ClickhouseUtil.java  |  8 +++-
 6 files changed, 48 insertions(+), 9 deletions(-)

diff --git a/docs/en/connector-v2/source/Clickhouse.md 
b/docs/en/connector-v2/source/Clickhouse.md
index ef5d99c05..07384875c 100644
--- a/docs/en/connector-v2/source/Clickhouse.md
+++ b/docs/en/connector-v2/source/Clickhouse.md
@@ -20,14 +20,15 @@ supports query SQL and can achieve projection effect.
 
 ## Options
 
-|  name  |  type  | required | default value |
-|||--|---|
-| host   | string | yes  | - |
-| database   | string | yes  | - |
-| sql| string | yes  | - |
-| username   | string | yes  | - |
-| password   | string | yes  | - |
-| common-options || no   | - |
+|   name   |  type  | required | default value  |
+|--||--||
+| host | string | yes  | -  |
+| database | string | yes  | -  |
+| sql  | string | yes  | -  |
+| username | string | yes  | -  |
+| password | string | yes  | -  |
+| server_time_zone | string | no   | ZoneId.systemDefault() |
+| common-options   || no   | -  |
 
 ### host [string]
 
@@ -49,6 +50,10 @@ The query sql used to search data though Clickhouse server
 
 `ClickHouse` user password
 
+### server_time_zone [string]
+
+The session time zone in database server. If not set, then 
ZoneId.systemDefault() is used to determine the server time zone.
+
 ### common options
 
 Source plugin common parameters, please refer to [Source Common 
Options](common-options.md) for details
@@ -64,6 +69,7 @@ source {
 sql = "select * from test where age = 20 limit 100"
 username = "default"
 password = ""
+server_time_zone = "UTC"
 result_table_name = "test"
   }
   
diff --git 
a/seatunnel-connectors-v2/connector-clickhouse/src/main/java/org/apache/seatunnel/connectors/seatunnel/clickhouse/config/ClickhouseConfig.java
 
b/seatunnel-connectors-v2/connector-clickhouse/src/main/java/org/apache/seatunnel/connectors/seatunnel/clickhouse/config/ClickhouseConfig.java
index 3d4b19edb..f7c8e032c 100644
--- 
a/seatunnel-connectors-v2/connector-clickhouse/src/main/java/org/apache/seatunnel/connectors/seatunnel/clickhouse/config/ClickhouseConfig.java
+++ 
b/seatunnel-connectors-v2/connector-clickhouse/src/main/java/org/apache/seatunnel/connectors/seatunnel/clickhouse/config/ClickhouseConfig.java
@@ -20,6 +20,7 @@ package 
org.apache.seatunnel.connectors.seatunnel.clickhouse.config;
 import org.apache.seatunnel.api.configuration.Option;
 import org.apache.seatunnel.api.configuration.Options;
 
+import java.time.ZoneId;
 import java.util.Collections;
 import java.util.List;
 import java.util.Map;
@@ -75,6 +76,15 @@ public class ClickhouseConfig {
 .noDefaultValue()
 .withDescription("Clickhouse server password");
 
+/** Clickhouse server timezone */
+public static final Option SERVER_TIME_ZONE =
+Options.key("server_time_zone")
+.stringType()
+.defaultValue(ZoneId.systemDefault().getId())
+.withDescription(
+"The session time zone in database server."
++ "If not set, then ZoneId.systemDefault() 
is used to determine the server time zone");
+
 /** Split mode when table is distributed engine */
 public static final Option SPLIT_MODE =
 Options.key("split_mode")
diff --git 
a/

[GitHub] [seatunnel] liugddx merged pull request #5080: [Fix][Connector-V2][Clickhouse] fix clickhouse source connector read Nullable() type is not null,exam…

2023-07-13 Thread via GitHub 


liugddx merged PR #5080:
URL: https://github.com/apache/seatunnel/pull/5080


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [seatunnel] liunaijie commented on pull request #5084: [Feature] support avro format

2023-07-13 Thread via GitHub 


liunaijie commented on PR #5084:
URL: https://github.com/apache/seatunnel/pull/5084#issuecomment-1635312636

   pls re-trigger the CI, no failed job, only one cancelled job


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org