This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-13009-84e47e957539de49ef2cd36d38d46a580e041227 in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit 3119df4ad90655190584ba1877ffabd5c60f7b52 Author: SiriusTL666 <[email protected]> AuthorDate: Wed Nov 4 15:07:11 2026 +0000 [Fix][Connector-V2][FTP] bound the control connection reply timeout (#13009) Co-authored-by: SiriusTL666 <[email protected]> --- docs/en/connectors/sink/FtpFile.md | 1 + docs/en/connectors/source/FtpFile.md | 1 + .../introduction/concepts/incompatible-changes.md | 7 + docs/zh/connectors/sink/FtpFile.md | 1 + docs/zh/connectors/source/FtpFile.md | 1 + .../introduction/concepts/incompatible-changes.md | 7 + .../seatunnel/file/ftp/config/FtpConf.java | 3 + .../file/ftp/config/FtpFileBaseOptions.java | 7 + .../file/ftp/sink/FtpFileSinkFactory.java | 6 + .../file/ftp/source/FtpFileSourceFactory.java | 6 + .../file/ftp/system/SeaTunnelFTPFileSystem.java | 104 +++- .../system/SeaTunnelFTPFileSystemTimeoutTest.java | 603 +++++++++++++++++++++ .../e2e/connector/file/ftp/FtpFileIT.java | 22 + .../test/resources/text/ftp_control_timeout.conf | 51 ++ 14 files changed, 790 insertions(+), 30 deletions(-) diff --git a/docs/en/connectors/sink/FtpFile.md b/docs/en/connectors/sink/FtpFile.md index b0bc827ea4..e3663c630d 100644 --- a/docs/en/connectors/sink/FtpFile.md +++ b/docs/en/connectors/sink/FtpFile.md @@ -51,6 +51,7 @@ By default, we use 2PC commit to ensure `exactly-once` | connection_mode | string | no | active_local | The target ftp connection mode | | remote_verification_enabled | boolean | no | true | Whether to enable remote host verification for FTP data channels | | control_encoding | string | no | UTF-8 | Character encoding for the FTP control connection, useful for paths with spaces or non-ASCII characters | +| control_timeout | int | no | 60000 | Timeout in milliseconds for reading a reply from the FTP server on the control connection; 0 waits indefinitely | | custom_filename | boolean | no | false | Whether you need custom the filename | | file_name_expression | string | no | "${transactionId}" | Only used when custom_filename is true | | filename_time_format | string | no | "yyyy.MM.dd" | Only used when custom_filename is true | diff --git a/docs/en/connectors/source/FtpFile.md b/docs/en/connectors/source/FtpFile.md index ccc056e7cc..d7302d7f0e 100644 --- a/docs/en/connectors/source/FtpFile.md +++ b/docs/en/connectors/source/FtpFile.md @@ -59,6 +59,7 @@ If you use SeaTunnel Engine, It automatically integrated the hadoop jar when you | connection_mode | string | no | active_local | | remote_verification_enabled | boolean | no | true | | control_encoding | string | no | UTF-8 | +| control_timeout | int | no | 60000 | | delimiter/field_delimiter | string | no | \001 | | row_delimiter | string | no | \n | | read_columns | list | no | - | diff --git a/docs/en/introduction/concepts/incompatible-changes.md b/docs/en/introduction/concepts/incompatible-changes.md index 2540e6c74c..e98a52c541 100644 --- a/docs/en/introduction/concepts/incompatible-changes.md +++ b/docs/en/introduction/concepts/incompatible-changes.md @@ -108,6 +108,13 @@ protocol still uses `234`; its wrapper scripts continue to translate help to exi The Markdown file source now declares `child_ids` as `ARRAY<STRING>` and emits a `String[]` for every row. Previously, it declared `STRING`, emitted comma-separated IDs for parents, and used `null` for leaves. Update downstream schemas and consumers to read an array; leaves now contain an empty array. The element IDs, types, page numbers, and position indexes retain their existing formats. +### FTP connector: control connection replies are now bounded by a 60 second timeout + +- **Behavior change: the FTP source and sink stop waiting for a control-connection reply after 60 seconds** + - **Description**: The FTP file system created its client and called `connect(host, port)` without setting a read timeout. The underlying client's read timeout defaults to `0`, which means wait indefinitely, so a server that accepted the TCP connection but never sent a reply — a stalled server, or a connection accepted by a firewall or load balancer that is silently dropping traffic — left the calling thread blocked forever. The new `control_timeout` option bounds every reply read on t [...] + - **Impact**: Jobs against a server that is reachable but does not answer the control channel now fail with a timeout instead of hanging. A job that previously stayed in `RUNNING` with no progress and no error now fails with the `SocketTimeoutException` raised by the underlying client, which reports the read that timed out but not the FTP host and port. Connections that answer within 60 seconds are unaffected. This option bounds control-channel replies only: the TCP connect is not cove [...] + - **Migration Guide**: No action is required. If a server legitimately needs more than 60 seconds to answer a control command, for example on a slow link, raise `control_timeout` on the connector. Set it to `0` to restore the previous wait-indefinitely behavior. Note that the upper bound applies per reply, so a sink that uploads a large file and whose server post-processes it can wait longer than 60 seconds for the final `226 Transfer complete`; raise `control_timeout` for such servers. + ### Runtime Requirements - **Breaking Change: Minimum Java runtime raised from Java 8 to Java 11** diff --git a/docs/zh/connectors/sink/FtpFile.md b/docs/zh/connectors/sink/FtpFile.md index ceeb8af3b1..64c0c4827a 100644 --- a/docs/zh/connectors/sink/FtpFile.md +++ b/docs/zh/connectors/sink/FtpFile.md @@ -52,6 +52,7 @@ import ChangeLog from '../changelog/connector-file-ftp.md'; | connection_mode | string | 否 | active_local | 目标FTP连接模式 | | remote_verification_enabled | boolean | 否 | true | 是否启用FTP数据通道的远程主机验证 | | control_encoding | string | 否 | UTF-8 | FTP 控制连接的字符编码,路径包含空格或非 ASCII 字符时很有用 | +| control_timeout | int | 否 | 60000 | 读取 FTP 服务端控制连接应答的超时时间(毫秒),0 表示无限等待 | | custom_filename | boolean | 否 | false | 是否需要自定义文件名 | | file_name_expression | string | 否 | "${transactionId}" | 仅在 `custom_filename` 为 `true` 时使用 | | filename_time_format | string | 否 | "yyyy.MM.dd" | 仅在 `custom_filename` 为 `true` 时使用 | diff --git a/docs/zh/connectors/source/FtpFile.md b/docs/zh/connectors/source/FtpFile.md index d15f3cf44c..be15ba625f 100644 --- a/docs/zh/connectors/source/FtpFile.md +++ b/docs/zh/connectors/source/FtpFile.md @@ -58,6 +58,7 @@ import ChangeLog from '../changelog/connector-file-ftp.md'; | connection_mode | string | 否 | active_local | | remote_verification_enabled | boolean | 否 | true | | control_encoding | string | 否 | UTF-8 | +| control_timeout | int | 否 | 60000 | | delimiter/field_delimiter | string | 否 | \001 | | row_delimiter | string | 否 | \n | | read_columns | list | 否 | - | diff --git a/docs/zh/introduction/concepts/incompatible-changes.md b/docs/zh/introduction/concepts/incompatible-changes.md index 6f51a64e40..2c5f9190f6 100644 --- a/docs/zh/introduction/concepts/incompatible-changes.md +++ b/docs/zh/introduction/concepts/incompatible-changes.md @@ -103,6 +103,13 @@ MySQL JDBC Sink 现在直接绑定 SeaTunnel `TIMESTAMP`(`LocalDateTime`), Markdown File Source 现在将 `child_ids` 声明为 `ARRAY<STRING>`,每行输出 `String[]`。此前该字段声明为 `STRING`,父元素输出逗号分隔的 ID,叶子元素输出 `null`。请将下游 schema 和消费逻辑改为读取数组;叶子元素现在输出空数组。元素 ID、类型、页码和位置索引保持原有格式。 +### FTP 连接器:控制连接应答现在受 60 秒超时约束 + +- **行为变更:FTP source 和 sink 在等待控制连接应答超过 60 秒后不再继续等待** + - **描述**:FTP 文件系统创建客户端后直接调用 `connect(host, port)`,没有设置读超时。底层客户端的读超时默认值是 `0`,即无限等待。因此当服务端接受了 TCP 连接却永不回复时——例如服务端卡死,或连接被防火墙、负载均衡接受后静默丢包——调用线程会永久阻塞。新增的 `control_timeout` 选项约束控制连接上的每一次应答读取,包括 greeting、登录交互以及传输前各条命令的应答,默认 60000 毫秒,source 和 sink 均生效。 + - **影响**:对于可以连通但不回复控制通道的服务端,作业现在会以超时失败,而不是挂死。此前会一直停留在 `RUNNING`、既无进展也无错误的作业,现在会抛出底层客户端给出的 `SocketTimeoutException`,该异常指明超时的那次读取,但**不包含** FTP 主机与端口。60 秒内能应答的连接不受影响。该选项只约束控制通道的应答:TCP 连接建立本身不受它约束,数据连接也不受它约束,因此初始 SYN 被静默丢弃的主机、以及传输开始后停滞的数据连接,都不会因本选项而有界。 + - **升级指南**:无需操作。若某个服务端确实需要超过 60 秒才能应答某条控制命令(例如在慢速链路上),调大连接器上的 `control_timeout`。将其设为 `0` 可恢复此前无限等待的行为。注意该上限是按单次应答计算的,因此 sink 上传大文件时,如果服务端会在接收后做后处理,最后那条 `226 Transfer complete` 可能超过 60 秒,这类服务端需要调大 `control_timeout`。 + ### 运行环境要求 - **破坏性变更:最低 Java 运行时从 Java 8 提升到 Java 11** diff --git a/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/config/FtpConf.java b/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/config/FtpConf.java index aba74db3bd..d031a0ea15 100644 --- a/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/config/FtpConf.java +++ b/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/config/FtpConf.java @@ -63,6 +63,9 @@ public class FtpConf extends HadoopConf { String.valueOf(config.get(FtpFileBaseOptions.FTP_REMOTE_VERIFICATION_ENABLED))); ftpOptions.put( "fs.ftp.control.encoding", config.get(FtpFileBaseOptions.FTP_CONTROL_ENCODING)); + ftpOptions.put( + "fs.ftp.control.timeout", + String.valueOf(config.get(FtpFileBaseOptions.FTP_CONTROL_TIMEOUT))); hadoopConf.setExtraOptions(ftpOptions); return hadoopConf; } diff --git a/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/config/FtpFileBaseOptions.java b/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/config/FtpFileBaseOptions.java index 1ad33259b4..f9cb2c6ac8 100644 --- a/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/config/FtpFileBaseOptions.java +++ b/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/config/FtpFileBaseOptions.java @@ -56,4 +56,11 @@ public class FtpFileBaseOptions extends FileBaseOptions { .defaultValue("UTF-8") .withDescription( "Character encoding for FTP control connection. Use UTF-8 to support special characters in file paths"); + public static final Option<Integer> FTP_CONTROL_TIMEOUT = + Options.key("control_timeout") + .intType() + .defaultValue(60_000) + .withDescription( + "Timeout in milliseconds for reading a reply from the FTP server on the control connection. " + + "A value of 0 waits indefinitely. Defaults to 60000 ms"); } diff --git a/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/sink/FtpFileSinkFactory.java b/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/sink/FtpFileSinkFactory.java index 7ad7cd9c84..075229716d 100644 --- a/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/sink/FtpFileSinkFactory.java +++ b/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/sink/FtpFileSinkFactory.java @@ -18,6 +18,7 @@ package org.apache.seatunnel.connectors.seatunnel.file.ftp.sink; import org.apache.seatunnel.api.configuration.ReadonlyConfig; +import org.apache.seatunnel.api.configuration.util.Conditions; import org.apache.seatunnel.api.configuration.util.OptionRule; import org.apache.seatunnel.api.options.SinkConnectorCommonOptions; import org.apache.seatunnel.api.table.catalog.CatalogTable; @@ -123,6 +124,11 @@ public class FtpFileSinkFactory extends BaseMultipleTableFileSinkFactory { .optional(FileBaseSinkOptions.FILENAME_EXTENSION) .optional(FtpFileSourceOptions.FTP_REMOTE_VERIFICATION_ENABLED) .optional(FtpFileSourceOptions.FTP_CONTROL_ENCODING) + .optional( + FtpFileSourceOptions.FTP_CONTROL_TIMEOUT, + // A negative value reaches Socket.setSoTimeout, which throws an + // IllegalArgumentException that does not name the offending option. + Conditions.greaterOrEqual(FtpFileSourceOptions.FTP_CONTROL_TIMEOUT, 0)) .build(); } diff --git a/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/source/FtpFileSourceFactory.java b/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/source/FtpFileSourceFactory.java index 2b2c0c26b1..25949a916a 100644 --- a/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/source/FtpFileSourceFactory.java +++ b/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/source/FtpFileSourceFactory.java @@ -17,6 +17,7 @@ package org.apache.seatunnel.connectors.seatunnel.file.ftp.source; +import org.apache.seatunnel.api.configuration.util.Conditions; import org.apache.seatunnel.api.configuration.util.OptionRule; import org.apache.seatunnel.api.options.ConnectorCommonOptions; import org.apache.seatunnel.api.source.SeaTunnelSource; @@ -116,6 +117,11 @@ public class FtpFileSourceFactory implements TableSourceFactory { FileBaseSourceOptions.PDF_RAG_METADATA_ENABLED) .optional(FtpFileSourceOptions.FTP_REMOTE_VERIFICATION_ENABLED) .optional(FtpFileSourceOptions.FTP_CONTROL_ENCODING) + .optional( + FtpFileSourceOptions.FTP_CONTROL_TIMEOUT, + // A negative value reaches Socket.setSoTimeout, which throws an + // IllegalArgumentException that does not name the offending option. + Conditions.greaterOrEqual(FtpFileSourceOptions.FTP_CONTROL_TIMEOUT, 0)) .optional(FileBaseSourceOptions.QUOTE_CHAR) .optional(FileBaseSourceOptions.ESCAPE_CHAR) .optional(ConnectorCommonOptions.METALAKE_TYPE) diff --git a/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/system/SeaTunnelFTPFileSystem.java b/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/system/SeaTunnelFTPFileSystem.java index e708a4cfc2..f7070a6c02 100644 --- a/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/system/SeaTunnelFTPFileSystem.java +++ b/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/system/SeaTunnelFTPFileSystem.java @@ -74,6 +74,7 @@ public class SeaTunnelFTPFileSystem extends FileSystem implements StreamingFileS public static final String FS_FTP_REMOTE_VERIFICATION_ENABLED = "fs.ftp.remote.verification.enabled"; public static final String FS_FTP_CONTROL_ENCODING = "fs.ftp.control.encoding"; + public static final String FS_FTP_CONTROL_TIMEOUT = "fs.ftp.control.timeout"; public static final String E_SAME_DIRECTORY_ONLY = "only same directory renames are supported"; @@ -158,41 +159,84 @@ public class SeaTunnelFTPFileSystem extends FileSystem implements StreamingFileS String user = conf.get(FS_FTP_USER_PREFIX + host); String password = conf.get(FS_FTP_PASSWORD_PREFIX + host); - // Connect to the FTP server - client.connect(host, port); - int reply = client.getReplyCode(); - if (!FTPReply.isPositiveCompletion(reply)) { - throw NetUtils.wrapException( - host, - port, - NetUtils.UNKNOWN_HOST, - 0, - new ConnectException("Server response " + reply)); - } + // Bound the control channel reads. Without this a server that accepts the TCP connection + // but never answers leaves the calling thread blocked forever, because the client default + // read timeout is 0 (wait indefinitely). + client.setDefaultTimeout( + conf.getInt( + FS_FTP_CONTROL_TIMEOUT, + FtpFileBaseOptions.FTP_CONTROL_TIMEOUT.defaultValue())); + + // Everything from here on can fail with an exception while the socket is already open. + // The caller only receives the client on success, so it cannot release it on the failure + // path: without this catch, a timeout raised by the greeting or login read would leak a + // connected control socket until the garbage collector runs. + try { + // Connect to the FTP server + client.connect(host, port); + int reply = client.getReplyCode(); + if (!FTPReply.isPositiveCompletion(reply)) { + throw NetUtils.wrapException( + host, + port, + NetUtils.UNKNOWN_HOST, + 0, + new ConnectException("Server response " + reply)); + } - // Log in to the FTP server - if (!client.login(user, password)) { - throw new IOException( - String.format( - "Login failed on server - %s, port - %d as user '%s', reply code: %d", - host, port, user, client.getReplyCode())); - } + // Log in to the FTP server + if (!client.login(user, password)) { + throw new IOException( + String.format( + "Login failed on server - %s, port - %d as user '%s', reply code: %d", + host, port, user, client.getReplyCode())); + } - // Set the file type to binary and buffer size - client.setFileType(FTP.BINARY_FILE_TYPE); - client.setBufferSize(DEFAULT_BUFFER_SIZE); - client.setFileTransferMode(FTP.BLOCK_TRANSFER_MODE); + // Set the file type to binary and buffer size + client.setFileType(FTP.BINARY_FILE_TYPE); + client.setBufferSize(DEFAULT_BUFFER_SIZE); + client.setFileTransferMode(FTP.BLOCK_TRANSFER_MODE); - // Set the connection mode - setFsFtpConnectionMode(client, connectionMode); + // Set the connection mode + setFsFtpConnectionMode(client, connectionMode); - // Log successful connection information - LOG.info( - String.format( - "Successfully connected to FTP server %s:%d in %s", - host, port, connectionMode)); + // Log successful connection information + LOG.info( + String.format( + "Successfully connected to FTP server %s:%d in %s", + host, port, connectionMode)); - return client; + return client; + } catch (IOException | RuntimeException e) { + releaseQuietly(client, host, port); + throw e; + } + } + + /** + * Close a client whose handshake failed, without sending QUIT. + * + * <p>{@link #disconnect(FTPClient)} logs out first, which writes QUIT and waits for its reply + * on the very control channel that just timed out. On a stalled server that wait costs another + * {@code control_timeout}, so the failure this change is meant to surface promptly would take + * twice as long. Closing the socket drops the connection immediately. + */ + private void releaseQuietly(FTPClient client, String host, int port) { + if (client == null) { + return; + } + // Not gated on isConnected(): a connect() that failed after the socket was allocated + // reports itself as not connected, yet still holds the descriptor. FTPClient.disconnect() + // skips the QUIT when not connected and closes the socket either way. + try { + client.disconnect(); + } catch (IOException e) { + LOG.warn( + String.format( + "Failed to close the FTP connection to %s:%d after a failed attempt", + host, port), + e); + } } /** diff --git a/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/system/SeaTunnelFTPFileSystemTimeoutTest.java b/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/system/SeaTunnelFTPFileSystemTimeoutTest.java new file mode 100644 index 0000000000..9180bf1219 --- /dev/null +++ b/seatunnel-connectors-v2/connector-file/connector-file-ftp/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/ftp/system/SeaTunnelFTPFileSystemTimeoutTest.java @@ -0,0 +1,603 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.connectors.seatunnel.file.ftp.system; + +import org.apache.seatunnel.api.configuration.ReadonlyConfig; +import org.apache.seatunnel.api.configuration.util.ConfigValidator; +import org.apache.seatunnel.api.configuration.util.OptionRule; +import org.apache.seatunnel.api.configuration.util.OptionValidationException; +import org.apache.seatunnel.connectors.seatunnel.file.config.HadoopConf; +import org.apache.seatunnel.connectors.seatunnel.file.ftp.config.FtpConf; +import org.apache.seatunnel.connectors.seatunnel.file.ftp.config.FtpFileBaseOptions; +import org.apache.seatunnel.connectors.seatunnel.file.ftp.sink.FtpFileSinkFactory; +import org.apache.seatunnel.connectors.seatunnel.file.ftp.source.FtpFileSourceFactory; + +import org.apache.commons.net.ftp.FTPClient; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.InOrder; +import org.mockito.MockedConstruction; +import org.mockito.Mockito; + +import java.io.IOException; +import java.net.InetAddress; +import java.net.ServerSocket; +import java.net.Socket; +import java.net.SocketTimeoutException; +import java.net.URI; +import java.net.URISyntaxException; +import java.util.HashMap; +import java.util.Map; +import java.util.concurrent.Callable; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicBoolean; + +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.mockConstruction; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +/** + * Covers the control-connection read timeout. + * + * <p>The timeout cannot be observed from outside the client: {@code FTPClient} exposes no getter + * for the value actually in effect on an accepted socket, and a timeout set after the greeting has + * been read is invisible. The test therefore intercepts construction and asserts the exact value + * handed to the client, plus the order relative to {@code connect()}, which is what decides whether + * the greeting read is protected. + */ +class SeaTunnelFTPFileSystemTimeoutTest { + + private static final String HOST = "ftp.example.com"; + private static final int PORT = 21; + private static final int DEFAULT_CONTROL_TIMEOUT = 60_000; + + /** + * Wall-clock bound for the real-socket calls, deliberately unrelated to the timeout under test. + * + * <p>It only has to expire long after a working {@code control_timeout} would, so that a + * regression surfaces as a bounded assertion failure instead of a hung build. + */ + private static final long DEADLINE_SECONDS = 30; + + /** Handshake behaviour the fake client should exhibit, set before the call under test. */ + private static final class Server { + int greetingCode = 220; + /** Reply code the server reports for the login exchange: 230 accepted, 530 refused. */ + int loginReplyCode = 230; + + boolean loginAccepted = true; + /** + * When set, {@code connect()} throws this instead of succeeding, like a timed-out greeting. + */ + IOException connectFailure; + /** When set, {@code login()} throws this instead, like a timed-out login reply. */ + IOException loginFailure; + + void stub(FTPClient client) throws IOException { + // connect() decides whether to continue by reading getReplyCode(), and login() runs + // only after the greeting has been accepted. Stubbing one fixed reply code makes the + // two exchanges indistinguishable: a refused-login code such as 530 would be rejected + // as a bad greeting, so connect() would never reach login() and the refused-login + // branch would never be exercised. The reply therefore follows the wire order: the + // greeting code first, the login code once credentials have been sent. + // + // The flag is per client, not per server: the filesystem may construct more than one + // client (getFileStatus resolves the home directory first), and a client that has not + // sent credentials yet must still see a greeting. + AtomicBoolean loginAttempted = new AtomicBoolean(); + when(client.getReplyCode()) + .thenAnswer(reply -> loginAttempted.get() ? loginReplyCode : greetingCode); + when(client.printWorkingDirectory()).thenReturn("/"); + when(client.isConnected()).thenReturn(true); + if (connectFailure != null) { + Mockito.doThrow(connectFailure).when(client).connect(anyString(), Mockito.anyInt()); + return; + } + if (loginFailure != null) { + when(client.login(anyString(), anyString())) + .thenAnswer( + credentials -> { + loginAttempted.set(true); + throw loginFailure; + }); + return; + } + // A refused login reports the rejection through the reply code rather than by + // throwing, so the code advances to loginReplyCode once credentials have been sent. + when(client.login(anyString(), anyString())) + .thenAnswer( + credentials -> { + loginAttempted.set(true); + return loginAccepted; + }); + } + } + + @Test + void controlTimeoutDefaultsToSixtySeconds() throws Exception { + assertTimeoutApplied(DEFAULT_CONTROL_TIMEOUT, baseConf()); + } + + @Test + void controlTimeoutUsesTheConfiguredValue() throws Exception { + Configuration conf = baseConf(); + conf.setInt(SeaTunnelFTPFileSystem.FS_FTP_CONTROL_TIMEOUT, 2_000); + assertTimeoutApplied(2_000, conf); + } + + @Test + void zeroRestoresTheWaitIndefinitelyBehaviour() throws Exception { + Configuration conf = baseConf(); + conf.setInt(SeaTunnelFTPFileSystem.FS_FTP_CONTROL_TIMEOUT, 0); + assertTimeoutApplied(0, conf); + } + + private void assertTimeoutApplied(int expected, Configuration conf) throws Exception { + Server server = new Server(); + try (MockedConstruction<FTPClient> mocked = constructing(server)) { + fileSystem(conf).getFileStatus(new Path("/")); + + verify(mocked.constructed().get(0)).setDefaultTimeout(expected); + } + } + + @Test + void controlTimeoutIsAppliedBeforeConnecting() throws Exception { + Configuration conf = baseConf(); + conf.setInt(SeaTunnelFTPFileSystem.FS_FTP_CONTROL_TIMEOUT, 1_500); + + try (MockedConstruction<FTPClient> mocked = constructing(new Server())) { + fileSystem(conf).getFileStatus(new Path("/")); + FTPClient client = mocked.constructed().get(0); + + // Applying the timeout after connect() leaves the greeting unprotected, and that read + // is exactly the one that hangs when a server accepts the connection and stays silent. + InOrder order = Mockito.inOrder(client); + order.verify(client).setDefaultTimeout(1_500); + order.verify(client).connect(HOST, PORT); + } + } + + @Test + void clientIsDisconnectedWhenTheServerDoesNotAcceptTheLogin() throws Exception { + Server server = new Server(); + // The greeting must stay positive: a nonpositive greeting is rejected by connect() before + // login() runs, so a 530 greeting would test the greeting branch twice and never reach + // loginAccepted == false. + server.greetingCode = 220; + server.loginAccepted = false; + server.loginReplyCode = 530; + + try (MockedConstruction<FTPClient> mocked = constructing(server)) { + Assertions.assertThrows( + IOException.class, () -> fileSystem(baseConf()).getFileStatus(new Path("/"))); + + // The socket is open by now, so leaving it to the garbage collector pins a control + // connection on the server until the server gives up on it. + FTPClient client = mocked.constructed().get(0); + verify(client).connect(HOST, PORT); + verify(client).login(anyString(), anyString()); + verify(client).disconnect(); + // QUIT must not be attempted on a server that just refused the login. + verify(client, never()).logout(); + } + } + + @Test + void clientIsClosedWhenTheGreetingReadTimesOut() throws Exception { + Server server = new Server(); + server.connectFailure = new SocketTimeoutException("Read timed out"); + + try (MockedConstruction<FTPClient> mocked = constructing(server)) { + Assertions.assertThrows( + SocketTimeoutException.class, + () -> fileSystem(baseConf()).getFileStatus(new Path("/"))); + + FTPClient client = mocked.constructed().get(0); + // connect() throws before the caller can receive the client, so this is the only + // place that can release the socket the client allocated before reading the greeting. + verify(client).disconnect(); + // logout() would write QUIT and wait for its reply on the channel that just timed + // out, costing another full control_timeout before the failure surfaces. + verify(client, never()).logout(); + verify(client, never()).login(anyString(), anyString()); + } + } + + @Test + void clientIsClosedWhenTheLoginReplyTimesOut() throws Exception { + Server server = new Server(); + server.loginFailure = new SocketTimeoutException("Read timed out"); + + try (MockedConstruction<FTPClient> mocked = constructing(server)) { + Assertions.assertThrows( + SocketTimeoutException.class, + () -> fileSystem(baseConf()).getFileStatus(new Path("/"))); + + FTPClient client = mocked.constructed().get(0); + verify(client).disconnect(); + verify(client, never()).logout(); + } + } + + @Test + void clientIsDisconnectedWhenTheGreetingIsRejected() throws Exception { + Server server = new Server(); + server.greetingCode = 421; + + try (MockedConstruction<FTPClient> mocked = constructing(server)) { + Assertions.assertThrows( + IOException.class, () -> fileSystem(baseConf()).getFileStatus(new Path("/"))); + + FTPClient client = mocked.constructed().get(0); + verify(client).connect(HOST, PORT); + verify(client).disconnect(); + verify(client, never()).login(anyString(), anyString()); + } + } + + @Test + void successfulCallClosesTheConnectionExactlyOnce() throws Exception { + try (MockedConstruction<FTPClient> mocked = constructing(new Server())) { + fileSystem(baseConf()).getFileStatus(new Path("/")); + + // The finally block in the caller owns the release. The new failure-path disconnect + // must not add a second one, otherwise a successful call would tear the client down + // twice and the second attempt would throw on an already closed socket. + verify(mocked.constructed().get(0), Mockito.times(1)).disconnect(); + } + } + + @Test + void controlTimeoutOptionIsDeclaredForBothSourceAndSink() { + // The control channel is shared by reads and writes, so a user who can only configure one + // side would still get the unbounded wait on the other. + Assertions.assertTrue( + new FtpFileSourceFactory() + .optionRule() + .getOptionalOptions() + .contains(FtpFileBaseOptions.FTP_CONTROL_TIMEOUT)); + Assertions.assertTrue( + new FtpFileSinkFactory() + .optionRule() + .getOptionalOptions() + .contains(FtpFileBaseOptions.FTP_CONTROL_TIMEOUT)); + } + + @Test + void negativeControlTimeoutIsRejectedByBothFactories() { + // Passed through unchecked, a negative value only fails later, inside Socket.setSoTimeout, + // as an IllegalArgumentException that does not name the option the user set. + // + // Both configs are otherwise complete and valid. An incomplete config would also throw, + // and the assertion below would then pass even with the control_timeout condition removed + // -- it would merely be re-detecting a missing required option. + Map<String, Object> source = new HashMap<>(); + source.put("host", HOST); + source.put("port", PORT); + source.put("user", "user"); + source.put("password", "password"); + source.put("path", "/tmp/seatunnel"); + source.put("control_timeout", -1); + + // The sink requires a path and a format; the source deliberately omits file_format_type so + // that no format-conditional option (schema, delimiters) becomes required. Otherwise the + // validation would report a second, unrelated error and the single-error assertion below + // would no longer pin the control_timeout constraint. + Map<String, Object> sink = new HashMap<>(source); + sink.put("file_format_type", "text"); + + assertOnlyControlTimeoutIsRejected(new FtpFileSourceFactory().optionRule(), source); + assertOnlyControlTimeoutIsRejected(new FtpFileSinkFactory().optionRule(), sink); + } + + /** + * Asserts the rejection is the {@code control_timeout} value constraint and nothing else. + * + * <p>Asserting the exception type alone is not enough: any missing required option raises the + * same type, so a config regression would look identical to the behaviour under test. + */ + private static void assertOnlyControlTimeoutIsRejected( + OptionRule rule, Map<String, Object> raw) { + OptionValidationException failure = + Assertions.assertThrows( + OptionValidationException.class, + () -> ConfigValidator.of(ReadonlyConfig.fromMap(raw)).validate(rule)); + + String message = failure.getRawMessage(); + Assertions.assertTrue( + message.contains("(1 error)"), + "Expected control_timeout to be the only rejected option, got:\n" + message); + Assertions.assertTrue( + message.contains("option: control_timeout"), + "The failure should name control_timeout, got:\n" + message); + Assertions.assertTrue( + message.contains("type: value"), + "The failure should be the value constraint, not a missing option, got:\n" + + message); + Assertions.assertTrue( + message.contains("'control_timeout' >= 0"), + "The failure should quote the violated constraint, got:\n" + message); + } + + @Test + void controlTimeoutIsForwardedToTheHadoopConfiguration() { + Map<String, Object> raw = new HashMap<>(); + raw.put("host", HOST); + raw.put("port", PORT); + raw.put("user", "user"); + raw.put("password", "password"); + raw.put("control_timeout", 4_500); + + HadoopConf hadoopConf = FtpConf.buildWithConfig(ReadonlyConfig.fromMap(raw)); + + // The value travels as a string through the Hadoop configuration map; a missing key would + // silently fall back to the default whatever the user configured. + Assertions.assertEquals( + "4500", + hadoopConf.getExtraOptions().get(SeaTunnelFTPFileSystem.FS_FTP_CONTROL_TIMEOUT)); + } + + /** + * A real server that accepts the TCP connection and then stays silent, like a stalled peer. + * + * <p>The {@code FTPClient} is not intercepted here: the point is to exercise commons-net's own + * read path so the socket read is genuinely bounded and the connection is genuinely released. + * Anything that only asserts that {@code setDefaultTimeout} was called would pass even if the + * value never reached the socket. + */ + private static final class SilentFtpServer implements AutoCloseable { + private final ServerSocket serverSocket; + private final Thread acceptor; + private final CountDownLatch accepted = new CountDownLatch(1); + private final CountDownLatch peerClosed = new CountDownLatch(1); + /** Kept so close() can unblock a client that is still waiting on this connection. */ + private volatile Socket connection; + + SilentFtpServer() throws IOException { + serverSocket = new ServerSocket(0, 1, InetAddress.getLoopbackAddress()); + acceptor = new Thread(this::serve, "silent-ftp-server"); + acceptor.setDaemon(true); + } + + void start() { + acceptor.start(); + } + + int port() { + return serverSocket.getLocalPort(); + } + + void awaitAccepted() throws InterruptedException { + Assertions.assertTrue( + accepted.await(10, TimeUnit.SECONDS), + "The client never opened the control connection"); + } + + /** + * Waits for the client to close its side. A read returning -1 or throwing means the peer + * sent FIN or RST; a socket left to the garbage collector would keep this waiting. + */ + void awaitPeerClosed() throws InterruptedException { + Assertions.assertTrue( + peerClosed.await(10, TimeUnit.SECONDS), + "The control connection was not closed after the handshake failed"); + } + + private void serve() { + try (Socket socket = serverSocket.accept()) { + connection = socket; + accepted.countDown(); + // Never write the 220 greeting. Read forever so a FIN from the client is observed + // as end-of-stream here. + while (socket.getInputStream().read() != -1) { + // discard anything the client sends; this server never replies + } + } catch (IOException e) { + // A reset from the client is also an observable close. + } finally { + peerClosed.countDown(); + } + } + + @Override + public void close() throws IOException { + serverSocket.close(); + // Releasing the accepted socket lets a client still blocked in a read (the + // control_timeout = 0 case) finish instead of leaking its thread past the test. + Socket open = connection; + if (open != null) { + open.close(); + } + } + } + + @Test + void stalledGreetingFailsWithinTheConfiguredTimeoutAndClosesTheConnection() throws Exception { + try (SilentFtpServer server = new SilentFtpServer()) { + server.start(); + Configuration conf = confFor(server.port()); + // Short enough to keep the test fast, long enough not to race a slow CI machine. + conf.setInt(SeaTunnelFTPFileSystem.FS_FTP_CONTROL_TIMEOUT, 1_000); + + Throwable failure = + callWithinDeadline( + server, + () -> { + fileSystem(conf, server.port()).getFileStatus(new Path("/")); + return null; + }); + + server.awaitAccepted(); + Assertions.assertInstanceOf( + SocketTimeoutException.class, + failure, + () -> + "The stalled greeting should fail with a socket timeout, got: " + + failure); + Assertions.assertNotNull(failure.getMessage()); + // Direct close, not logout: the server never replies, so QUIT would block again and + // the connection would still be open when the exception surfaces. + server.awaitPeerClosed(); + } + } + + @Test + void zeroControlTimeoutDoesNotBoundTheStalledGreeting() throws Exception { + SilentFtpServer server = new SilentFtpServer(); + ExecutorService executor = Executors.newSingleThreadExecutor(); + boolean terminated; + try { + server.start(); + Configuration conf = confFor(server.port()); + // 0 restores the previous wait-indefinitely behaviour, so the call must still be + // blocked well past the timeout the other test proves is effective. + conf.setInt(SeaTunnelFTPFileSystem.FS_FTP_CONTROL_TIMEOUT, 0); + + Future<?> call = + executor.submit( + () -> fileSystem(conf, server.port()).getFileStatus(new Path("/"))); + server.awaitAccepted(); + // An explicit bounded observation rather than a sleep-then-isDone: a call that is + // still blocked is the positive result here, so the timeout is the assertion. + Assertions.assertThrows( + TimeoutException.class, + () -> call.get(1_500, TimeUnit.MILLISECONDS), + "control_timeout = 0 must restore the unbounded wait, but the call returned"); + } finally { + // Closing the peer is what releases a read that no socket timeout will release, so it + // has to happen before the executor is checked for termination; awaiting first would + // always time out and report a leak that is really this ordering. + server.close(); + executor.shutdownNow(); + terminated = executor.awaitTermination(10, TimeUnit.SECONDS); + } + Assertions.assertTrue( + terminated, + "The blocked control read outlived the test even after the peer was closed"); + } + + /** + * Runs the call on its own thread under a deadline that is independent of {@code + * control_timeout}. + * + * <p>The regression this test exists to catch is a control read that is no longer bounded. + * Invoking the filesystem inline would then never return, so neither the assertion nor the + * cleanup would run and the build would hang instead of reporting a failure. Returning the + * failure instead of asserting it keeps every phase of the test interruptible. + * + * <p>The peer is force-closed only when the call is still blocked, which can only happen on the + * regression path: an ordinary failure has already released its socket, so closing here + * unconditionally would make the caller's peer-closed check vacuous. + */ + private static Throwable callWithinDeadline(SilentFtpServer server, Callable<Void> call) + throws Exception { + ExecutorService executor = Executors.newSingleThreadExecutor(); + Future<Void> pending = executor.submit(call); + Throwable outcome; + boolean terminated; + try { + pending.get(DEADLINE_SECONDS, TimeUnit.SECONDS); + outcome = + new AssertionError( + "The stalled greeting did not fail the call, so the control read is" + + " apparently not bounded at all"); + } catch (ExecutionException e) { + outcome = e.getCause(); + } catch (TimeoutException e) { + outcome = + new AssertionError( + "The call was still blocked after " + + DEADLINE_SECONDS + + " s although control_timeout was 1000 ms; a job would stay" + + " RUNNING on a silent server", + e); + } finally { + if (!pending.isDone()) { + // Only reachable after a regression: closing the peer is the sole thing that + // releases a read no socket timeout is bounding. + server.close(); + } + executor.shutdownNow(); + terminated = executor.awaitTermination(10, TimeUnit.SECONDS); + } + Assertions.assertTrue( + terminated, "The control connection was not released, so its thread leaked"); + return outcome; + } + + private static MockedConstruction<FTPClient> constructing(Server server) { + return mockConstruction( + FTPClient.class, + (mock, context) -> { + try { + server.stub(mock); + } catch (IOException e) { + throw new IllegalStateException(e); + } + }); + } + + private static Configuration baseConf() { + Configuration conf = new Configuration(); + conf.set(SeaTunnelFTPFileSystem.FS_FTP_HOST, HOST); + conf.setInt(SeaTunnelFTPFileSystem.FS_FTP_HOST_PORT, PORT); + conf.set(SeaTunnelFTPFileSystem.FS_FTP_USER_PREFIX + HOST, "user"); + conf.set(SeaTunnelFTPFileSystem.FS_FTP_PASSWORD_PREFIX + HOST, "password"); + conf.set(SeaTunnelFTPFileSystem.FS_FTP_CONNECTION_MODE, "passive_local"); + return conf; + } + + /** Configuration pointing at a real local server instead of the mocked {@link #HOST}. */ + private static Configuration confFor(int port) { + Configuration conf = baseConf(); + conf.set(SeaTunnelFTPFileSystem.FS_FTP_HOST, "localhost"); + conf.setInt(SeaTunnelFTPFileSystem.FS_FTP_HOST_PORT, port); + conf.set(SeaTunnelFTPFileSystem.FS_FTP_USER_PREFIX + "localhost", "user"); + conf.set(SeaTunnelFTPFileSystem.FS_FTP_PASSWORD_PREFIX + "localhost", "password"); + return conf; + } + + private static SeaTunnelFTPFileSystem fileSystem(Configuration conf, int port) + throws IOException, URISyntaxException { + return fileSystem(conf, new URI("ftp://localhost:" + port)); + } + + private static SeaTunnelFTPFileSystem fileSystem(Configuration conf, URI uri) + throws IOException { + SeaTunnelFTPFileSystem fs = new SeaTunnelFTPFileSystem(); + fs.initialize(uri, conf); + return fs; + } + + private static SeaTunnelFTPFileSystem fileSystem(Configuration conf) + throws IOException, URISyntaxException { + return fileSystem(conf, new URI("ftp://" + HOST + ":" + PORT)); + } +} diff --git a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-ftp-e2e/src/test/java/org/apache/seatunnel/e2e/connector/file/ftp/FtpFileIT.java b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-ftp-e2e/src/test/java/org/apache/seatunnel/e2e/connector/file/ftp/FtpFileIT.java index d23fe15e50..008b50311f 100644 --- a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-ftp-e2e/src/test/java/org/apache/seatunnel/e2e/connector/file/ftp/FtpFileIT.java +++ b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-ftp-e2e/src/test/java/org/apache/seatunnel/e2e/connector/file/ftp/FtpFileIT.java @@ -198,6 +198,28 @@ public class FtpFileIT extends TestSuiteBase implements TestResource { deleteFileFromContainer(homePath); } + @TestTemplate + public void testFtpControlTimeoutIsHonouredThroughTheJobConfig(TestContainer container) + throws IOException, InterruptedException { + // The unit tests pin the value handed to FTPClient, but only a real job shows the option + // survives parsing, factory validation and connector instantiation on both sides. The + // config uses a distinct non-default timeout per side, so an option wired into only one + // connector would be visible here. + Container.ExecResult execResult = container.executeJob("/text/ftp_control_timeout.conf"); + Assertions.assertEquals(0, execResult.getExitCode(), execResult.getStderr()); + + String homePath = ftpHomeDir + "/uploads/seatunnel-control-timeout"; + Assertions.assertEquals(1, getFileListFromContainer(homePath).size()); + + // Same row count as the source file: the timeout must not have altered the transfer. + Container.ExecResult lineCount = + ftpContainer.execInContainer( + "sh", "-c", "awk 'END {print NR}' " + homePath + "/e2e.txt"); + Assertions.assertEquals("5", lineCount.getStdout().trim()); + + deleteFileFromContainer(homePath); + } + @TestTemplate public void testFtpToFtpForBinary(TestContainer container) throws IOException, InterruptedException { diff --git a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-ftp-e2e/src/test/resources/text/ftp_control_timeout.conf b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-ftp-e2e/src/test/resources/text/ftp_control_timeout.conf new file mode 100644 index 0000000000..8295c08b7f --- /dev/null +++ b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-ftp-e2e/src/test/resources/text/ftp_control_timeout.conf @@ -0,0 +1,51 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Exercises an explicit, non-default control_timeout through a packaged job, on both +# the source and the sink. The values differ per side so a shared option that only +# reached one connector would still be caught. +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + FtpFile { + host = "ftp" + port = 21 + user = seatunnel + password = pass + path = "/tmp/seatunnel/read/text/name=tyrantlucifer/hobby=coding/e2e.txt" + file_format_type = "binary" + encoding = "UTF-8" + control_timeout = 30000 + } +} + +sink { + FtpFile { + host = "ftp" + port = 21 + user = seatunnel + password = pass + tmp_path = "/upload-tmp/seatunnel" + path = "/uploads/seatunnel-control-timeout" + file_format_type = "binary" + encoding = "UTF-8" + control_timeout = 15000 + } +}
