BIGflynb opened a new issue, #9039: URL: https://github.com/apache/seatunnel/issues/9039
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened The seatunnel task does not stop automatically after reading and writing. After a period of time, the task is repeated, and an error message is displayed indicating that the postgresql key is repeated, resulting in a task failure. ### SeaTunnel Version 2.3.8 ### SeaTunnel Config ```conf env { parallelism = 1 job.mode = "BATCH" } source { SftpFile { plugin_output = "table1" host = "" port = user = "" password = "" path = "/path/.zip" file_format_type = "text" field_delimiter = "," skip_header_row_number = 7 archive_compress_codec = "ZIP" schema = { fields { } } } } transform { Sql { plugin_input = "table1" plugin_output = "result1" query = """ SELECT } } sink { Jdbc { plugin_input = "result1" url = "jdbc:postgresql:" driver = org.postgresql.Driver user = password = query = """ INSERT INTO } } ``` ### Running Command ```shell ./bin/seatunnel.sh --config config/v2.streaming.conf.template -m local ``` ### Error Exception ```log [INFO] 2025-03-25 10:23:16.524 +0800 - -> 2025-03-25 10:23:15,732 INFO [o.a.s.e.c.j.JobMetricsRunner ] [job-metrics-runner-956729158640074753] - *********************************************** Job Progress Information *********************************************** Job Id : 956729158640074753 Read Count So Far : 162666 Write Count So Far : 160608 Average Read Count : 33/s Average Write Count : 33/s Last Statistic Time : 2025-03-25 10:22:15 Current Statistic Time : 2025-03-25 10:23:15 *********************************************** [INFO] 2025-03-25 10:24:04.530 +0800 - -> 2025-03-25 10:24:03,887 INFO [o.a.s.e.s.CoordinatorService ] [pool-6-thread-1] - [localhost]:5801 [seatunnel-406852] [5.1] *********************************************** CoordinatorService Thread Pool Status *********************************************** activeCount : 1 corePoolSize : 0 maximumPoolSize : 2147483647 poolSize : 1 completedTaskCount : 462 taskCount : 463 *********************************************** 2025-03-25 10:24:03,889 INFO [o.a.s.e.s.CoordinatorService ] [pool-6-thread-1] - [localhost]:5801 [seatunnel-406852] [5.1] *********************************************** Job info detail *********************************************** createdJobCount : 0 scheduledJobCount : 0 runningJobCount : 1 failingJobCount : 0 failedJobCount : 0 cancellingJobCount : 0 canceledJobCount : 0 finishedJobCount : 0 *********************************************** [INFO] 2025-03-25 10:24:16.533 +0800 - -> 2025-03-25 10:24:15,734 INFO [o.a.s.e.c.j.JobMetricsRunner ] [job-metrics-runner-956729158640074753] - *********************************************** Job Progress Information *********************************************** Job Id : 956729158640074753 Read Count So Far : 162697 Write Count So Far : 162688 Average Read Count : 0/s Average Write Count : 34/s Last Statistic Time : 2025-03-25 10:23:15 Current Statistic Time : 2025-03-25 10:24:15 *********************************************** [INFO] 2025-03-25 10:25:04.538 +0800 - -> 2025-03-25 10:25:03,887 INFO [o.a.s.e.s.CoordinatorService ] [pool-6-thread-1] - [localhost]:5801 [seatunnel-406852] [5.1] *********************************************** CoordinatorService Thread Pool Status *********************************************** activeCount : 1 corePoolSize : 0 maximumPoolSize : 2147483647 poolSize : 1 completedTaskCount : 462 taskCount : 463 *********************************************** 2025-03-25 10:25:03,890 INFO [o.a.s.e.s.CoordinatorService ] [pool-6-thread-1] - [localhost]:5801 [seatunnel-406852] [5.1] *********************************************** Job info detail *********************************************** createdJobCount : 0 scheduledJobCount : 0 runningJobCount : 1 failingJobCount : 0 failedJobCount : 0 cancellingJobCount : 0 canceledJobCount : 0 finishedJobCount : 0 *********************************************** [INFO] 2025-03-25 10:25:06.540 +0800 - -> 2025-03-25 10:25:06,462 WARN [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=956729158640074753, pipelineId=1, taskGroupId=30003}] - [localhost]:5801 [seatunnel-406852] [5.1] Exception in org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask@69a298fe org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[COMMON-01], ErrorDescription:[SeaTunnel read file 'sftp://ip/export/home/omc/objectstorage/var/prs/result_file/Cell_Hour_ToNMS/.zip' failed.] at org.apache.seatunnel.common.exception.CommonError.fileOperationFailed(CommonError.java:68) ~[seatunnel-starter.jar:2.3.8] at org.apache.seatunnel.connectors.seatunnel.file.source.BaseFileSourceReader.pollNext(BaseFileSourceReader.java:65) ~[connector-file-sftp-2.3.8.jar:2.3.8] at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:159) ~[seatunnel-starter.jar:2.3.8] at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:127) ~[seatunnel-starter.jar:2.3.8] at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168) ~[seatunnel-starter.jar:2.3.8] at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:132) ~[seatunnel-starter.jar:2.3.8] at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:693) [seatunnel-starter.jar:2.3.8] at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1018) [seatunnel-starter.jar:2.3.8] at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:39) [seatunnel-starter.jar:2.3.8] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_422] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_422] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_422] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_422] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_422] Caused by: java.io.IOException: error at com.jcraft.jsch.ChannelSftp$2.close(ChannelSftp.java:1532) ~[connector-file-sftp-2.3.8.jar:2.3.8] at org.apache.seatunnel.connectors.seatunnel.file.sftp.system.SFTPInputStream.close(SFTPInputStream.java:113) ~[connector-file-sftp-2.3.8.jar:2.3.8] at java.io.FilterInputStream.close(FilterInputStream.java:181) ~[?:1.8.0_422] at java.io.PushbackInputStream.close(PushbackInputStream.java:379) ~[?:1.8.0_422] at java.util.zip.InflaterInputStream.close(InflaterInputStream.java:227) ~[?:1.8.0_422] at java.util.zip.ZipInputStream.close(ZipInputStream.java:266) ~[?:1.8.0_422] at org.apache.seatunnel.connectors.seatunnel.file.source.reader.AbstractReadStrategy.resolveArchiveCompressedInputStream(AbstractReadStrategy.java:201) ~[connector-file-sftp-2.3.8.jar:2.3.8] at org.apache.seatunnel.connectors.seatunnel.file.source.reader.TextReadStrategy.read(TextReadStrategy.java:70) ~[connector-file-sftp-2.3.8.jar:2.3.8] at org.apache.seatunnel.connectors.seatunnel.file.source.BaseFileSourceReader.pollNext(BaseFileSourceReader.java:63) ~[connector-file-sftp-2.3.8.jar:2.3.8] ... 12 more 2025-03-25 10:25:06,490 INFO [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=956729158640074753, pipelineId=1, taskGroupId=30003}] - [localhost]:5801 [seatunnel-406852] [5.1] taskDone, taskId = 40003, taskGroup = TaskGroupLocation{jobId=956729158640074753, pipelineId=1, taskGroupId=30003} 2025-03-25 10:25:06,490 INFO [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=956729158640074753, pipelineId=1, taskGroupId=30003}] - [localhost]:5801 [seatunnel-406852] [5.1] task 40003 error with exception: [org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[COMMON-01], ErrorDescription:[SeaTunnel read file 'sftp://ip/export/home/omc/objectstorage/var/prs/result_file/Cell_Hour_ToNMS/.zip' failed.]], cancel other task in taskGroup TaskGroupLocation{jobId=956729158640074753, pipelineId=1, taskGroupId=30003}. 2025-03-25 10:25:06,491 WARN [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=956729158640074753, pipelineId=1, taskGroupId=30003}] - [localhost]:5801 [seatunnel-406852] [5.1] Interrupted task 50003 - org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask@318555ab 2025-03-25 10:25:06,491 INFO [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=956729158640074753, pipelineId=1, taskGroupId=30003}] - [localhost]:5801 [seatunnel-406852] [5.1] taskDone, taskId = 50003, taskGroup = TaskGroupLocation{jobId=956729158640074753, pipelineId=1, taskGroupId=30003} 2025-03-25 10:25:06,492 INFO [o.a.s.c.s.f.s.s.SFTPFileSystem] [BlockingWorker-TaskGroupLocation{jobId=956729158640074753, pipelineId=1, taskGroupId=30003}] - Inside shutdown, con2infoMap size=1 2025-03-25 10:25:06,492 INFO [o.a.s.a.e.LoggingEventHandler ] [hz.main.generic-operation.thread-22] - log event: ReaderCloseEvent(createdTime=1742869506491, jobId=956729158640074753, eventType=LIFECYCLE_READER_CLOSE) 2025-03-25 10:25:06,501 INFO [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=956729158640074753, pipelineId=1, taskGroupId=30003}] - [localhost]:5801 [seatunnel-406852] [5.1] taskGroup TaskGroupLocation{jobId=956729158640074753, pipelineId=1, taskGroupId=30003} complete with FAILED 2025-03-25 10:25:06,503 INFO [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=956729158640074753, pipelineId=1, taskGroupId=30003}] - [localhost]:5801 [seatunnel-406852] [5.1] task 50003 error with exception: [org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[COMMON-01], ErrorDescription:[SeaTunnel read file 'sftp://ip/export/home/omc/objectstorage/var/prs/result_file/Cell_Hour_ToNMS/.zip' failed.]], cancel other task in taskGroup TaskGroupLocation{jobId=956729158640074753, pipelineId=1, taskGroupId=30003}. 2025-03-25 10:25:06,503 INFO [o.a.s.e.s.TaskExecutionService] [hz.main.seaTunnel.task.thread-11] - [localhost]:5801 [seatunnel-406852] [5.1] Task TaskGroupLocation{jobId=956729158640074753, pipelineId=1, taskGroupId=30003} complete with state FAILED 2025-03-25 10:25:06,503 INFO [o.a.s.e.s.CoordinatorService ] [hz.main.seaTunnel.task.thread-11] - [localhost]:5801 [seatunnel-406852] [5.1] Received task end from execution TaskGroupLocation{jobId=956729158640074753, pipelineId=1, taskGroupId=30003}, state FAILED 2025-03-25 10:25:06,507 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [hz.main.seaTunnel.task.thread-11] - Job SeaTunnel_Job (956729158640074753), Pipeline: [(1/9)], task: [pipeline-1 [Source[0]-SftpFile]-SourceTask (4/9)] turned from state RUNNING to FAILED. 2025-03-25 10:25:06,507 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [hz.main.seaTunnel.task.thread-11] - Job SeaTunnel_Job (956729158640074753), Pipeline: [(1/9)], task: [pipeline-1 [Source[0]-SftpFile]-SourceTask (4/9)] state process is stopped 2025-03-25 10:25:06,508 ERROR [o.a.s.e.s.d.p.PhysicalVertex ] [hz.main.seaTunnel.task.thread-11] - Job SeaTunnel_Job (956729158640074753), Pipeline: [(1/9)], task: [pipeline-1 [Source[0]-SftpFile]-SourceTask (4/9)] end with state FAILED and Exception: org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[COMMON-01], ErrorDescription:[SeaTunnel read file 'sftp://ip/export/home/omc/objectstorage/var/prs/result_file/Cell_Hour_ToNMS/.zip' failed.] at org.apache.seatunnel.common.exception.CommonError.fileOperationFailed(CommonError.java:68) at org.apache.seatunnel.connectors.seatunnel.file.source.BaseFileSourceReader.pollNext(BaseFileSourceReader.java:65) at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:159) at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:127) at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168) at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:132) at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:693) at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1018) at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:39) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.io.IOException: error at com.jcraft.jsch.ChannelSftp$2.close(ChannelSftp.java:1532) at org.apache.seatunnel.connectors.seatunnel.file.sftp.system.SFTPInputStream.close(SFTPInputStream.java:113) at java.io.FilterInputStream.close(FilterInputStream.java:181) at java.io.PushbackInputStream.close(PushbackInputStream.java:379) at java.util.zip.InflaterInputStream.close(InflaterInputStream.java:227) at java.util.zip.ZipInputStream.close(ZipInputStream.java:266) at org.apache.seatunnel.connectors.seatunnel.file.source.reader.AbstractReadStrategy.resolveArchiveCompressedInputStream(AbstractReadStrategy.java:201) at org.apache.seatunnel.connectors.seatunnel.file.source.reader.TextReadStrategy.read(TextReadStrategy.java:70) at org.apache.seatunnel.connectors.seatunnel.file.source.BaseFileSourceReader.pollNext(BaseFileSourceReader.java:63) ... 12 more ``` ### Zeta or Flink or Spark Version _No response_ ### Java or Scala Version _No response_ ### Screenshots _No response_ ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
