This is an automated email from the ASF dual-hosted git repository.
jarvis pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 5aad17e1f8 [Improve][Zeta] Set the default `slot-num` value to twice
the number of CPU cores (#9601)
5aad17e1f8 is described below
commit 5aad17e1f87b5f9b579636d17225181591c40ca9
Author: Jia Fan <[email protected]>
AuthorDate: Wed Jul 30 15:37:21 2025 +0800
[Improve][Zeta] Set the default `slot-num` value to twice the number of CPU
cores (#9601)
---
.../seatunnel-engine/hybrid-cluster-deployment.md | 3 +-
.../separated-cluster-deployment.md | 3 +-
.../seatunnel-engine/hybrid-cluster-deployment.md | 3 +-
.../separated-cluster-deployment.md | 3 +-
.../common/config/YamlSeaTunnelConfigLocator.java | 2 +-
.../common/config/server/ServerConfigOptions.java | 2 +-
.../config/YamlSeaTunnelConfigParserTest.java | 37 ++++++++++++++++++++--
...custmoize-client.yaml => customize-client.yaml} | 2 +-
...tmoize-client.yaml => customize-seatunnel.yaml} | 26 +++------------
9 files changed, 50 insertions(+), 31 deletions(-)
diff --git a/docs/en/seatunnel-engine/hybrid-cluster-deployment.md
b/docs/en/seatunnel-engine/hybrid-cluster-deployment.md
index 6aad26dccb..d63219b2f0 100644
--- a/docs/en/seatunnel-engine/hybrid-cluster-deployment.md
+++ b/docs/en/seatunnel-engine/hybrid-cluster-deployment.md
@@ -54,7 +54,8 @@ seatunnel:
### 4.2 Slot Configuration
-The number of slots determines the number of task groups that the cluster node
can run in parallel. The formula for the number of slots required for a task is
N = 2 + P (the parallelism configured by the task). By default, the number of
slots in the SeaTunnel Engine is dynamic, that is, there is no limit on the
number. We recommend that the number of slots be set to twice the number of CPU
cores on the node.
+The number of slots determines the number of task groups that the cluster node
can run in parallel. The formula for the number of slots required for a task is
N = 2 + P (the parallelism configured by the task). By default, the number of
slots in the SeaTunnel Engine is dynamic, that is, there is no limit on the
number.
+We recommend that the number of slots be set to twice the number of CPU cores
on the node, it's a default value when `dynamic-slot` is set to false and not
set `slot-num`.
Configuration of dynamic slot number (default):
diff --git a/docs/en/seatunnel-engine/separated-cluster-deployment.md
b/docs/en/seatunnel-engine/separated-cluster-deployment.md
index 1244042dbb..d01043edef 100644
--- a/docs/en/seatunnel-engine/separated-cluster-deployment.md
+++ b/docs/en/seatunnel-engine/separated-cluster-deployment.md
@@ -88,7 +88,8 @@ Since in the separated cluster mode, the Worker node does not
store Imap data, t
### 4.2 Slot configuration (this parameter is not effective on the Master node)
-The number of Slots determines the number of task groups that can be run in
parallel on the cluster node. The number of Slots required by a task is
formulated as N = 2 + P (parallelism configured by the task). By default, the
number of Slots of SeaTunnel Engine is dynamic, that is, there is no limit on
the number. We recommend that the number of Slots be set to twice the number of
CPU cores of the node.
+The number of Slots determines the number of task groups that can be run in
parallel on the cluster node. The number of Slots required by a task is
formulated as N = 2 + P (parallelism configured by the task). By default, the
number of Slots of SeaTunnel Engine is dynamic, that is, there is no limit on
the number.
+We recommend that the number of slots be set to twice the number of CPU cores
on the node, it's a default value when `dynamic-slot` is set to false and not
set `slot-num`.
The configuration of dynamic slot number (default) is as follows:
diff --git a/docs/zh/seatunnel-engine/hybrid-cluster-deployment.md
b/docs/zh/seatunnel-engine/hybrid-cluster-deployment.md
index 3adb6f6022..664e6e3296 100644
--- a/docs/zh/seatunnel-engine/hybrid-cluster-deployment.md
+++ b/docs/zh/seatunnel-engine/hybrid-cluster-deployment.md
@@ -54,7 +54,8 @@ seatunnel:
### 4.2 Slot配置
-Slot数量决定了集群节点可以并行运行的任务组数量。一个任务需要的Slot的个数公式为 N = 2 + P(任务配置的并行度)。
默认情况下SeaTunnel Engine的slot个数为动态，即不限制个数。我们建议slot的个数设置为节点CPU核心数的2倍。
+Slot数量决定了集群节点可以并行运行的任务组数量。一个任务需要的Slot的个数公式为 N = 2 + P(任务配置的并行度)。
默认情况下SeaTunnel Engine的slot个数为动态，即不限制个数。
+我们建议slot的个数设置为节点CPU核心数的2倍, 这也是当 `dynamic-slot` 设置为 false 且未设置 `slot-num` 时的默认值。
动态slot个数（默认）配置如下：
diff --git a/docs/zh/seatunnel-engine/separated-cluster-deployment.md
b/docs/zh/seatunnel-engine/separated-cluster-deployment.md
index 7293e17c4e..0bbfd12a47 100644
--- a/docs/zh/seatunnel-engine/separated-cluster-deployment.md
+++ b/docs/zh/seatunnel-engine/separated-cluster-deployment.md
@@ -92,7 +92,8 @@ seatunnel:
### 4.2 Slot配置（该参数在Master节点无效）
-Slot数量决定了集群节点可以并行运行的任务组数量。一个任务需要的Slot的个数公式为 N = 2 + P(任务配置的并行度)。
默认情况下SeaTunnel Engine的slot个数为动态，即不限制个数。我们建议slot的个数设置为节点CPU核心数的2倍。
+Slot数量决定了集群节点可以并行运行的任务组数量。一个任务需要的Slot的个数公式为 N = 2 + P(任务配置的并行度)。
默认情况下SeaTunnel Engine的slot个数为动态，即不限制个数。
+我们建议slot的个数设置为节点CPU核心数的2倍, 这也是当 `dynamic-slot` 设置为 false 且未设置 `slot-num` 时的默认值。
动态slot个数（默认）配置如下：
diff --git
a/seatunnel-engine/seatunnel-engine-common/src/main/java/org/apache/seatunnel/engine/common/config/YamlSeaTunnelConfigLocator.java
b/seatunnel-engine/seatunnel-engine-common/src/main/java/org/apache/seatunnel/engine/common/config/YamlSeaTunnelConfigLocator.java
index 0b3691df10..53ef13076b 100644
---
a/seatunnel-engine/seatunnel-engine-common/src/main/java/org/apache/seatunnel/engine/common/config/YamlSeaTunnelConfigLocator.java
+++
b/seatunnel-engine/seatunnel-engine-common/src/main/java/org/apache/seatunnel/engine/common/config/YamlSeaTunnelConfigLocator.java
@@ -24,7 +24,7 @@ import com.hazelcast.internal.config.AbstractConfigLocator;
import static
com.hazelcast.internal.config.DeclarativeConfigUtil.YAML_ACCEPTED_SUFFIXES;
/** A support class for the {@link YamlSeaTunnelConfigBuilder} to locate the
yaml configuration. */
-public final class YamlSeaTunnelConfigLocator extends AbstractConfigLocator {
+public class YamlSeaTunnelConfigLocator extends AbstractConfigLocator {
public YamlSeaTunnelConfigLocator() {}
diff --git
a/seatunnel-engine/seatunnel-engine-common/src/main/java/org/apache/seatunnel/engine/common/config/server/ServerConfigOptions.java
b/seatunnel-engine/seatunnel-engine-common/src/main/java/org/apache/seatunnel/engine/common/config/server/ServerConfigOptions.java
index 9deb98f29b..9811fa0842 100644
---
a/seatunnel-engine/seatunnel-engine-common/src/main/java/org/apache/seatunnel/engine/common/config/server/ServerConfigOptions.java
+++
b/seatunnel-engine/seatunnel-engine-common/src/main/java/org/apache/seatunnel/engine/common/config/server/ServerConfigOptions.java
@@ -400,7 +400,7 @@ public class ServerConfigOptions {
public static final Option<Integer> SLOT_NUM =
Options.key("slot-num")
.intType()
- .defaultValue(2)
+
.defaultValue(Runtime.getRuntime().availableProcessors() * 2)
.withDescription(
"The number of slots. Only valid when dynamic
slot is disabled.");
diff --git
a/seatunnel-engine/seatunnel-engine-common/src/test/java/org/apache/seatunnel/engine/common/config/YamlSeaTunnelConfigParserTest.java
b/seatunnel-engine/seatunnel-engine-common/src/test/java/org/apache/seatunnel/engine/common/config/YamlSeaTunnelConfigParserTest.java
index bae761237d..89ac90cd72 100644
---
a/seatunnel-engine/seatunnel-engine-common/src/test/java/org/apache/seatunnel/engine/common/config/YamlSeaTunnelConfigParserTest.java
+++
b/seatunnel-engine/seatunnel-engine-common/src/test/java/org/apache/seatunnel/engine/common/config/YamlSeaTunnelConfigParserTest.java
@@ -25,6 +25,8 @@ import com.hazelcast.client.config.YamlClientConfigBuilder;
import java.io.IOException;
+import static
com.hazelcast.internal.config.DeclarativeConfigUtil.YAML_ACCEPTED_SUFFIXES;
+
public class YamlSeaTunnelConfigParserTest {
@Test
@@ -85,10 +87,10 @@ public class YamlSeaTunnelConfigParserTest {
@Test
public void testCustomizeClientConfig() throws IOException {
YamlClientConfigBuilder yamlClientConfigBuilder =
- new YamlClientConfigBuilder("custmoize-client.yaml");
+ new YamlClientConfigBuilder("customize-client.yaml");
ClientConfig clientConfig = yamlClientConfigBuilder.build();
- Assertions.assertEquals("custmoize", clientConfig.getClusterName());
+ Assertions.assertEquals("customize", clientConfig.getClusterName());
Assertions.assertEquals(
3000L,
clientConfig
@@ -96,4 +98,35 @@ public class YamlSeaTunnelConfigParserTest {
.getConnectionRetryConfig()
.getClusterConnectTimeoutMillis());
}
+
+ @Test
+ public void testCustomizeSeaTunnelYaml() throws IOException {
+ YamlSeaTunnelConfigLocator yamlConfigLocator =
+ new YamlSeaTunnelConfigLocator() {
+ @Override
+ protected boolean locateInWorkDir() {
+ return loadFromWorkingDirectory(
+ "customize-seatunnel", YAML_ACCEPTED_SUFFIXES);
+ }
+
+ @Override
+ protected boolean locateOnClasspath() {
+ return loadConfigurationFromClasspath(
+ "customize-seatunnel", YAML_ACCEPTED_SUFFIXES);
+ }
+ };
+ SeaTunnelConfig config;
+ if (yamlConfigLocator.locateInWorkDirOrOnClasspath()) {
+ // 2. Try loading YAML config from the working directory or from
the classpath
+ config = new
YamlSeaTunnelConfigBuilder(yamlConfigLocator).setProperties(null).build();
+ } else {
+ throw new RuntimeException("can't find yaml in resources");
+ }
+
+
Assertions.assertFalse(config.getEngineConfig().getSlotServiceConfig().isDynamicSlot());
+ // test the default slot number should be 2 * availableProcessors
+ Assertions.assertEquals(
+ Runtime.getRuntime().availableProcessors() * 2,
+ config.getEngineConfig().getSlotServiceConfig().getSlotNum());
+ }
}
diff --git
a/seatunnel-engine/seatunnel-engine-common/src/test/resources/custmoize-client.yaml
b/seatunnel-engine/seatunnel-engine-common/src/test/resources/customize-client.yaml
similarity index 97%
copy from
seatunnel-engine/seatunnel-engine-common/src/test/resources/custmoize-client.yaml
copy to
seatunnel-engine/seatunnel-engine-common/src/test/resources/customize-client.yaml
index 55dd529622..1074ada0d5 100644
---
a/seatunnel-engine/seatunnel-engine-common/src/test/resources/custmoize-client.yaml
+++
b/seatunnel-engine/seatunnel-engine-common/src/test/resources/customize-client.yaml
@@ -16,7 +16,7 @@
#
hazelcast-client:
- cluster-name: custmoize
+ cluster-name: customize
connection-strategy:
connection-retry:
cluster-connect-timeout-millis: 3000
diff --git
a/seatunnel-engine/seatunnel-engine-common/src/test/resources/custmoize-client.yaml
b/seatunnel-engine/seatunnel-engine-common/src/test/resources/customize-seatunnel.yaml
similarity index 64%
rename from
seatunnel-engine/seatunnel-engine-common/src/test/resources/custmoize-client.yaml
rename to
seatunnel-engine/seatunnel-engine-common/src/test/resources/customize-seatunnel.yaml
index 55dd529622..f8d8cc5098 100644
---
a/seatunnel-engine/seatunnel-engine-common/src/test/resources/custmoize-client.yaml
+++
b/seatunnel-engine/seatunnel-engine-common/src/test/resources/customize-seatunnel.yaml
@@ -15,25 +15,7 @@
# limitations under the License.
#
-hazelcast-client:
- cluster-name: custmoize
- connection-strategy:
- connection-retry:
- cluster-connect-timeout-millis: 3000
- network:
- cluster-members:
- - host:5801
- - host:5802
- - host:5803
- - host:5804
- - host:5805
- - host:5806
- - host:5807
- - host:5808
- - host:5809
- - host:5810
- - host:5811
- - host:5812
- - host:5813
- - host:5814
- - host:5815
+seatunnel:
+ engine:
+ slot-service:
+ dynamic-slot: false
