This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new e25c90a0167 [Fix][Connector-V2][Jdbc] Fix DATE partition column
splitting (#11396)
e25c90a0167 is described below
commit e25c90a0167e2e4303613c7f94d9323a298e8da6
Author: JacobZheng0927 <[email protected]>
AuthorDate: Fri Nov 6 16:23:06 2026 +0000
[Fix][Connector-V2][Jdbc] Fix DATE partition column splitting (#11396)
Co-authored-by: Daniel <[email protected]>
Co-authored-by: zhiweiniu <[email protected]>
---
docs/en/connectors/source/Jdbc.md | 6 +-
docs/zh/connectors/source/Jdbc.md | 6 +-
.../seatunnel/jdbc/source/FixedChunkSplitter.java | 108 ++++++++--
.../jdbc/source/FixedChunkSplitterTest.java | 230 +++++++++++++++++++++
.../seatunnel/jdbc/JdbcMysqlSplitIT.java | 87 ++++++++
5 files changed, 419 insertions(+), 18 deletions(-)
diff --git a/docs/en/connectors/source/Jdbc.md
b/docs/en/connectors/source/Jdbc.md
index 929aea532dc..6a30c60df45 100644
--- a/docs/en/connectors/source/Jdbc.md
+++ b/docs/en/connectors/source/Jdbc.md
@@ -167,8 +167,8 @@ Use `query` to select only the required columns.
| connect_timeout_ms | Int | No | 86400000
| Connection timeout in milliseconds when establishing the JDBC
connection. `0` means no timeout.
[...]
| socket_timeout_ms | Int | No | 86400000
| Socket read timeout in milliseconds after the JDBC connection is
established. `0` means no timeout.
[...]
| partition_column | String | No | -
| The column name for split data.
[...]
-| partition_upper_bound | String | No | -
| Inclusive upper value used for query-based partitioning. When omitted,
SeaTunnel queries the source for the maximum value.
[...]
-| partition_lower_bound | String | No | -
| Inclusive lower value used for query-based partitioning. When omitted,
SeaTunnel queries the source for the minimum value.
[...]
+| partition_upper_bound | String | No | -
| Inclusive upper value used for query-based partitioning. When omitted,
SeaTunnel queries the source for the maximum value. For a DATE partition
column, an ISO-8601 date string (`yyyy-MM-dd`) or a legacy epoch-day number can
be used.
[...]
+| partition_lower_bound | String | No | -
| Inclusive lower value used for query-based partitioning. When omitted,
SeaTunnel queries the source for the minimum value. For a DATE partition
column, an ISO-8601 date string (`yyyy-MM-dd`) or a legacy epoch-day number can
be used.
[...]
| partition_num | Int | No | 10
| Number of splits when top-level `query` and `partition_column` select
the fixed splitter, whether or not `table_path` is also present. Dynamic
`table_path` and `table_list` layouts use `split.size` instead.
[...]
| decimal_type_narrowing | Boolean | No | true
| Decimal type narrowing, if true, the decimal type will be narrowed to
the int or long type if without loss of precision. Only support for Oracle at
now. Please refer to `decimal_type_narrowing` below
[...]
| int_type_narrowing | Boolean | No | true
| Int type narrowing, if true, the tinyint(1) type will be narrowed to
the boolean type if without loss of precision. Support for MySQL at now. Please
refer to `int_type_narrowing` below
[...]
@@ -349,6 +349,8 @@ Only the top-level combination of `query` and
`partition_column` selects the leg
For fixed splitting of integer keys, including BIGINT and zero-scale DECIMAL,
the partitions cover the numeric range without gaps or overlaps. An uneven
division by `partition_num` preserves the remaining values, and each interval
stays within the lower and upper bounds.
+Fixed query partitioning supports columns mapped to SeaTunnel DATE with ISO
dates or whole epoch-day bounds. Support follows the dialect mapping; for
example, Oracle DATE maps to TIMESTAMP and is not covered here. TIME, TIMESTAMP
and TIMESTAMP_TZ partition columns are not supported; use a supported partition
column or omit partition_column to read the query as a single split.
+
Entries inside `table_list` continue to use dynamic splitting even when they
include `query` or partition settings. Do not expect `split.size` to affect the
top-level fixed partition mode.
### Query and primary-key caution
diff --git a/docs/zh/connectors/source/Jdbc.md
b/docs/zh/connectors/source/Jdbc.md
index 7a472060962..c5e5613a21a 100644
--- a/docs/zh/connectors/source/Jdbc.md
+++ b/docs/zh/connectors/source/Jdbc.md
@@ -163,8 +163,8 @@ driver = "org.mariadb.jdbc.Driver"
| connect_timeout_ms | Int | 否 | 86400000 | 建立
JDBC 连接时的连接超时时间,单位毫秒。`0` 表示不超时。
[...]
| socket_timeout_ms | Int | 否 | 86400000 |
JDBC 连接建立后的 socket 读取超时时间,单位毫秒。`0` 表示不超时。
[...]
| partition_column | String | 否 | - |
用于分割数据的列名。
[...]
-| partition_upper_bound | String | 否 | - |
query 分区读取使用的闭区间上界。未配置时,SeaTunnel 会从源表查询最大值。
[...]
-| partition_lower_bound | String | 否 | - |
query 分区读取使用的闭区间下界。未配置时,SeaTunnel 会从源表查询最小值。
[...]
+| partition_upper_bound | String | 否 | - |
query 分区读取使用的闭区间上界。未配置时,SeaTunnel 会从源表查询最大值。对于 DATE 分区列,支持 ISO-8601
日期字符串(`yyyy-MM-dd`)或兼容历史配置的 epoch-day 数字。
[...]
+| partition_lower_bound | String | 否 | - |
query 分区读取使用的闭区间下界。未配置时,SeaTunnel 会从源表查询最小值。对于 DATE 分区列,支持 ISO-8601
日期字符串(`yyyy-MM-dd`)或兼容历史配置的 epoch-day 数字。
[...]
| partition_num | Int | 否 | 10 | 顶层
`query` 与 `partition_column` 触发 fixed splitter 时的分片数,无论是否同时配置 `table_path`。动态
`table_path` 和 `table_list` 布局改用 `split.size`。
[...]
| decimal_type_narrowing | Boolean | 否 | true |
十进制类型缩小,如果为 true,十进制类型将缩小为 int 或 long 类型(如果没有精度损失)。目前仅支持 Oracle。请参考下面的
`decimal_type_narrowing`
[...]
| int_type_narrowing | Boolean | 否 | true | Int
类型缩小,如果为 true,tinyint(1) 类型将缩小为布尔类型(如果没有精度损失)。目前支持 MySQL。请参考下面的
`int_type_narrowing`
[...]
@@ -197,6 +197,8 @@ driver = "org.mariadb.jdbc.Driver"
| split.string_split_mode_collate | String | 否 | -
| 当 string_split_mode 设置为 `charset_based`
且表具有特殊排序规则时,指定要使用的排序规则。如果未指定,将使用数据库的默认排序规则。
|
| common-options | | 否 | - |
源插件通用参数,请参考 [源通用选项](../common-options/source-common-options.md) 详见。
[...]
+固定查询分片支持映射为 SeaTunnel DATE 类型的列,边界可使用 ISO 日期或整数 epoch-day。支持范围取决于方言映射;例如
Oracle DATE 映射为 TIMESTAMP,不在本次支持范围内。TIME、TIMESTAMP 和 TIMESTAMP_TZ
列不支持作为分片列;请选择受支持的列,或不设置 partition_column,以单个分片读取查询。
+
### 表匹配
请按照数据库方言要求填写完整表路径:
diff --git
a/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/FixedChunkSplitter.java
b/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/FixedChunkSplitter.java
index 5ad48e8554b..7be8ae09ec5 100644
---
a/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/FixedChunkSplitter.java
+++
b/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/FixedChunkSplitter.java
@@ -44,6 +44,9 @@ import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.sql.Time;
import java.sql.Timestamp;
+import java.time.DateTimeException;
+import java.time.LocalDate;
+import java.time.format.DateTimeParseException;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
@@ -199,16 +202,11 @@ public class FixedChunkSplitter extends ChunkSplitter {
private Collection<JdbcSourceSplit> getJdbcSourceSplits(
JdbcSourceTable table, String splitKeyName, SeaTunnelDataType
splitKeyType)
throws SQLException {
- BigDecimal partitionStart =
- StringUtils.isBlank(table.getPartitionStart())
- ? null
- : new BigDecimal(table.getPartitionStart());
- BigDecimal partitionEnd =
- StringUtils.isBlank(table.getPartitionEnd())
- ? null
- : new BigDecimal(table.getPartitionEnd());
+ BigDecimal partitionStart =
parsePartitionBound(table.getPartitionStart(), splitKeyType);
+ BigDecimal partitionEnd = parsePartitionBound(table.getPartitionEnd(),
splitKeyType);
if (partitionStart == null || partitionEnd == null) {
- Pair<BigDecimal, BigDecimal> range = findSplitColumnRange(table,
splitKeyName);
+ Pair<BigDecimal, BigDecimal> range =
+ findSplitColumnRange(table, splitKeyName, splitKeyType);
partitionStart = range.getLeft();
partitionEnd = range.getRight();
}
@@ -424,8 +422,8 @@ public class FixedChunkSplitter extends ChunkSplitter {
table.getQuery(),
splitKeyName,
splitKeyType,
- parameterValues[i][0],
- parameterValues[i][1]);
+ convertSplitBoundary(splitKeyType,
parameterValues[i][0]),
+ convertSplitBoundary(splitKeyType,
parameterValues[i][1]));
splits.add(split);
}
return splits;
@@ -484,6 +482,8 @@ public class FixedChunkSplitter extends ChunkSplitter {
statement.setByte(i + 1, (Byte) param);
} else if (param instanceof Short) {
statement.setShort(i + 1, (Short) param);
+ } else if (param instanceof LocalDate) {
+ statement.setDate(i + 1, Date.valueOf((LocalDate) param));
} else if (param instanceof Date) {
statement.setDate(i + 1, (Date) param);
} else if (param instanceof Time) {
@@ -521,20 +521,100 @@ public class FixedChunkSplitter extends ChunkSplitter {
return Pair.of(((String) min), ((String) max));
}
+ /** Converts temporal DATE min/max values to epoch days before numeric
range arithmetic. */
private Pair<BigDecimal, BigDecimal> findSplitColumnRange(
- JdbcSourceTable table, String columnName) throws SQLException {
+ JdbcSourceTable table, String columnName, SeaTunnelDataType
splitKeyType)
+ throws SQLException {
Pair<Object, Object> splitColumnRange = queryMinMax(table, columnName);
Object min = splitColumnRange.getLeft();
Object max = splitColumnRange.getRight();
if (min != null) {
- min = convertToBigDecimal(min);
+ min = convertToBigDecimal(min, splitKeyType);
}
if (max != null) {
- max = convertToBigDecimal(max);
+ max = convertToBigDecimal(max, splitKeyType);
}
return Pair.of(((BigDecimal) min), ((BigDecimal) max));
}
+ /**
+ * Parses partition bounds into the unit used by the numeric splitter.
+ *
+ * <p>DATE values use epoch days. The numeric fallback preserves
compatibility with existing
+ * configurations that supplied raw epoch-day values.
+ */
+ private BigDecimal parsePartitionBound(String bound, SeaTunnelDataType
splitKeyType) {
+ if (StringUtils.isBlank(bound)) {
+ return null;
+ }
+ SqlType sqlType = splitKeyType.getSqlType();
+ boolean dateSplitKey = SqlType.DATE.equals(sqlType);
+ if (SqlType.DATE.equals(sqlType)) {
+ try {
+ return BigDecimal.valueOf(LocalDate.parse(bound).toEpochDay());
+ } catch (DateTimeParseException ignored) {
+ }
+ }
+ try {
+ BigDecimal numericBound = new BigDecimal(bound);
+ if (dateSplitKey && numericBound.stripTrailingZeros().scale() > 0)
{
+ throw new JdbcConnectorException(
+ CommonErrorCodeDeprecated.ILLEGAL_ARGUMENT,
+ "DATE partition bounds must be whole numbers: " +
bound);
+ }
+ return numericBound;
+ } catch (NumberFormatException e) {
+ if (dateSplitKey) {
+ String expectedFormat = "yyyy-MM-dd or a whole epoch-day
number.";
+ throw new JdbcConnectorException(
+ CommonErrorCodeDeprecated.ILLEGAL_ARGUMENT,
+ "Invalid "
+ + sqlType
+ + " partition bound '"
+ + bound
+ + "'. Expected "
+ + expectedFormat,
+ e);
+ }
+ throw e;
+ }
+ }
+
+ /** Keeps DATE split boundaries independent of JVM timezones until JDBC
parameter binding. */
+ private Serializable convertSplitBoundary(
+ SeaTunnelDataType splitKeyType, Serializable splitBoundary) {
+ if (!(splitBoundary instanceof BigDecimal)) {
+ return splitBoundary;
+ }
+ BigDecimal boundary = (BigDecimal) splitBoundary;
+ if (SqlType.DATE.equals(splitKeyType.getSqlType())) {
+ try {
+ return LocalDate.ofEpochDay(boundary.longValueExact());
+ } catch (ArithmeticException | DateTimeException e) {
+ throw new JdbcConnectorException(
+ CommonErrorCodeDeprecated.ILLEGAL_ARGUMENT,
+ "DATE partition bound is outside the supported date
range: " + boundary,
+ e);
+ }
+ }
+ return splitBoundary;
+ }
+
+ /**
+ * Converts DATE min/max results to epoch days, including drivers
returning DATE as TIMESTAMP.
+ */
+ private BigDecimal convertToBigDecimal(Object o, SeaTunnelDataType
splitKeyType) {
+ if (SqlType.DATE.equals(splitKeyType.getSqlType())) {
+ if (o instanceof Date) {
+ return BigDecimal.valueOf(((Date)
o).toLocalDate().toEpochDay());
+ } else if (o instanceof Timestamp) {
+ return BigDecimal.valueOf(
+ ((Timestamp)
o).toLocalDateTime().toLocalDate().toEpochDay());
+ }
+ }
+ return convertToBigDecimal(o);
+ }
+
private BigDecimal convertToBigDecimal(Object o) {
if (o instanceof BigDecimal) {
return (BigDecimal) o;
diff --git
a/seatunnel-connectors-v2/connector-jdbc/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/FixedChunkSplitterTest.java
b/seatunnel-connectors-v2/connector-jdbc/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/FixedChunkSplitterTest.java
index c1b230c2c96..852f591506d 100644
---
a/seatunnel-connectors-v2/connector-jdbc/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/FixedChunkSplitterTest.java
+++
b/seatunnel-connectors-v2/connector-jdbc/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/FixedChunkSplitterTest.java
@@ -17,9 +17,16 @@
package org.apache.seatunnel.connectors.seatunnel.jdbc.source;
+import org.apache.seatunnel.shade.org.apache.commons.lang3.tuple.Pair;
+
+import org.apache.seatunnel.api.serialization.DefaultSerializer;
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.PhysicalColumn;
+import org.apache.seatunnel.api.table.catalog.TableIdentifier;
import org.apache.seatunnel.api.table.catalog.TablePath;
import org.apache.seatunnel.api.table.catalog.TableSchema;
import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.LocalTimeType;
import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
import
org.apache.seatunnel.connectors.seatunnel.jdbc.config.JdbcConnectionConfig;
@@ -28,6 +35,7 @@ import
org.apache.seatunnel.connectors.seatunnel.jdbc.exception.JdbcConnectorExc
import org.duckdb.DuckDBDriver;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.parallel.ResourceLock;
import lombok.extern.slf4j.Slf4j;
@@ -36,10 +44,13 @@ import java.lang.reflect.Method;
import java.lang.reflect.Proxy;
import java.math.BigDecimal;
import java.sql.Connection;
+import java.sql.Date;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
+import java.sql.Timestamp;
+import java.time.LocalDate;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
@@ -51,6 +62,7 @@ import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.Set;
+import java.util.TimeZone;
import java.util.stream.Collectors;
import java.util.stream.LongStream;
@@ -130,6 +142,178 @@ public class FixedChunkSplitterTest {
assertTrue(exception.getMessage().contains("does not support
range/auto"));
}
+ @Test
+ public void testDateSplitsWithIsoAndLegacyBoundsBindJdbcDates() throws
Exception {
+ for (String[] bounds :
+ new String[][] {
+ {"2024-01-01", "2024-01-03"},
+ {"19723", "19725"}
+ }) {
+ CapturingFixedChunkSplitter splitter = new
CapturingFixedChunkSplitter(mysqlConfig());
+ assertDateSplits(splitter, dateTable(bounds[0], bounds[1]));
+ }
+ }
+
+ @Test
+ public void testDateSplitsDiscoverDateAndTimestampMinMax() throws
Exception {
+ for (Object[] bounds :
+ new Object[][] {
+ {Date.valueOf("2024-01-01"), Date.valueOf("2024-01-03")},
+ {
+ Timestamp.valueOf("2024-01-01 12:34:56"),
+ Timestamp.valueOf("2024-01-03 12:34:56")
+ }
+ }) {
+ CapturingFixedChunkSplitter splitter =
+ new CapturingFixedChunkSplitter(mysqlConfig()) {
+ @Override
+ protected Pair<Object, Object> queryMinMax(
+ JdbcSourceTable table, String columnName) {
+ return Pair.of(bounds[0], bounds[1]);
+ }
+ };
+ assertDateSplits(splitter, dateTable(null, null));
+ }
+ }
+
+ @Test
+ @ResourceLock("java.util.TimeZone")
+ public void testDateSplitsPreserveDaysAcrossTimeZoneSerialization() throws
Exception {
+ TimeZone original = TimeZone.getDefault();
+ try {
+ for (String[] zones :
+ new String[][] {
+ {"Asia/Shanghai", "America/Los_Angeles"},
+ {"Etc/GMT+12", "Pacific/Kiritimati"}
+ }) {
+ TimeZone.setDefault(TimeZone.getTimeZone(zones[0]));
+ CapturingFixedChunkSplitter splitter =
+ new CapturingFixedChunkSplitter(mysqlConfig());
+ DefaultSerializer<JdbcSourceSplit> serializer = new
DefaultSerializer<>();
+ List<byte[]> serialized = new ArrayList<>();
+ for (JdbcSourceSplit split :
+ splitter.createSplits(dateTable("2024-01-01",
"2024-01-03"), dateKey())) {
+ serialized.add(serializer.serialize(split));
+ }
+ TimeZone.setDefault(TimeZone.getTimeZone(zones[1]));
+ for (int i = 0; i < serialized.size(); i++) {
+ JdbcSourceSplit restored =
serializer.deserialize(serialized.get(i));
+ splitter.generateSplitStatement(restored,
TableSchema.builder().build());
+ LocalDate expected = LocalDate.of(2024, 1, 1).plusDays(i);
+ assertEquals(expected,
splitter.dateParameters.get(1).toLocalDate());
+ assertEquals(expected,
splitter.dateParameters.get(2).toLocalDate());
+ assertTrue(splitter.bigDecimalParameters.isEmpty());
+ }
+ assertEquals(3, serialized.size());
+ }
+ } finally {
+ TimeZone.setDefault(original);
+ }
+ }
+
+ @Test
+ public void testRejectInvalidAndFractionalDateBounds() {
+ FixedChunkSplitter splitter = new FixedChunkSplitter(mysqlConfig());
+ for (String bound : new String[] {"2024-01-03 00:00:00", "20000.5"}) {
+ JdbcConnectorException exception =
+ assertThrows(
+ JdbcConnectorException.class,
+ () -> splitter.createSplits(dateTable(bound,
"2024-01-03"), dateKey()));
+ assertTrue(exception.getMessage().contains("DATE"));
+ }
+ }
+
+ @Test
+ public void testRejectOutOfRangeDateBounds() {
+ FixedChunkSplitter splitter = new FixedChunkSplitter(mysqlConfig());
+ for (String bound :
+ new String[] {String.valueOf(Long.MAX_VALUE),
"1000000000000000000000000000000"}) {
+ JdbcConnectorException error =
+ assertThrows(
+ JdbcConnectorException.class,
+ () -> splitter.createSplits(dateTable(bound,
bound), dateKey()));
+ assertTrue(error.getMessage().contains("DATE partition bound"));
+ }
+ }
+
+ @Test
+ public void testExistingJdbcDateSplitParametersRemainSupported() throws
Exception {
+ CapturingFixedChunkSplitter splitter = new
CapturingFixedChunkSplitter(mysqlConfig());
+ Date date = Date.valueOf("2024-01-01");
+ JdbcSourceSplit split =
+ new JdbcSourceSplit(
+ TablePath.of("db", "tbl"),
+ "legacy-date",
+ null,
+ "create_date",
+ LocalTimeType.LOCAL_DATE_TYPE,
+ date,
+ date);
+ splitter.generateSplitStatement(split, TableSchema.builder().build());
+ assertEquals(date, splitter.dateParameters.get(1));
+ assertEquals(date, splitter.dateParameters.get(2));
+ }
+
+ @Test
+ public void testNonDateTemporalPartitionColumnsRemainUnsupported() {
+ FixedChunkSplitter splitter = new FixedChunkSplitter(mysqlConfig());
+ for (SeaTunnelDataType<?> type :
+ new SeaTunnelDataType<?>[] {
+ LocalTimeType.LOCAL_TIME_TYPE,
+ LocalTimeType.LOCAL_DATE_TIME_TYPE,
+ LocalTimeType.OFFSET_DATE_TIME_TYPE
+ }) {
+ TableSchema schema =
+ TableSchema.builder()
+ .column(PhysicalColumn.of("created_at", type, 0L,
true, null, null))
+ .build();
+ JdbcSourceTable table =
+ JdbcSourceTable.builder()
+ .tablePath(TablePath.of("db", "tbl"))
+ .partitionColumn("created_at")
+ .catalogTable(
+ CatalogTable.of(
+ TableIdentifier.of("catalog",
"db", "tbl"),
+ schema,
+ Collections.emptyMap(),
+ Collections.emptyList(),
+ ""))
+ .build();
+ assertThrows(JdbcConnectorException.class, () ->
splitter.generateSplits(table));
+ }
+ }
+
+ private static JdbcSourceTable dateTable(String lower, String upper) {
+ return JdbcSourceTable.builder()
+ .tablePath(TablePath.of("db", "tbl"))
+ .partitionNumber(3)
+ .partitionStart(lower)
+ .partitionEnd(upper)
+ .build();
+ }
+
+ private static SeaTunnelRowType dateKey() {
+ return new SeaTunnelRowType(
+ new String[] {"create_date"},
+ new SeaTunnelDataType[] {LocalTimeType.LOCAL_DATE_TYPE});
+ }
+
+ private static void assertDateSplits(
+ CapturingFixedChunkSplitter splitter, JdbcSourceTable table)
throws Exception {
+ List<JdbcSourceSplit> splits = new
ArrayList<>(splitter.createSplits(table, dateKey()));
+ assertEquals(3, splits.size());
+ for (int i = 0; i < splits.size(); i++) {
+ Date expected = Date.valueOf(LocalDate.of(2024, 1, 1).plusDays(i));
+ JdbcSourceSplit split = splits.get(i);
+ assertEquals(expected.toLocalDate(), split.getSplitStart());
+ assertEquals(expected.toLocalDate(), split.getSplitEnd());
+ splitter.generateSplitStatement(split,
TableSchema.builder().build());
+ assertEquals(expected, splitter.dateParameters.get(1));
+ assertEquals(expected, splitter.dateParameters.get(2));
+ assertTrue(splitter.bigDecimalParameters.isEmpty());
+ }
+ }
+
@Test
public void testConvertFloat() throws Exception {
JdbcSourceConfig config =
@@ -189,12 +373,15 @@ public class FixedChunkSplitterTest {
.driverName("com.mysql.cj.jdbc.Driver")
.build())
.stringSplitStrategy(StringSplitStrategy.RANGE)
+ .enableConcurrentRead(true)
.build();
}
private static class CapturingFixedChunkSplitter extends
FixedChunkSplitter {
private String sql;
private final Map<Integer, String> stringParameters = new HashMap<>();
+ private final Map<Integer, Date> dateParameters = new HashMap<>();
+ private final Map<Integer, BigDecimal> bigDecimalParameters = new
HashMap<>();
private CapturingFixedChunkSplitter(JdbcSourceConfig config) {
super(config);
@@ -207,6 +394,10 @@ public class FixedChunkSplitterTest {
(proxy, method, args) -> {
if ("setString".equals(method.getName())) {
stringParameters.put((Integer) args[0], (String)
args[1]);
+ } else if ("setDate".equals(method.getName())) {
+ dateParameters.put((Integer) args[0], (Date)
args[1]);
+ } else if ("setBigDecimal".equals(method.getName())) {
+ bigDecimalParameters.put((Integer) args[0],
(BigDecimal) args[1]);
}
return null;
};
@@ -325,6 +516,45 @@ public class FixedChunkSplitterTest {
}
}
+ @Test
+ public void testDateSplitsUseNumericRangeFixForUnevenAndExcessPartitions()
throws Exception {
+ try (Connection connection = new DuckDBDriver().connect(DUCKDB_URL,
new Properties());
+ DuckDbFixedChunkSplitter splitter =
+ new DuckDbFixedChunkSplitter(duckDbConfig(),
connection)) {
+ try (Statement statement = connection.createStatement()) {
+ statement.execute("CREATE TABLE t (id BIGINT, create_date
DATE)");
+ }
+ try (PreparedStatement insert =
+ connection.prepareStatement("INSERT INTO t VALUES (?,
?)")) {
+ for (int id = 0; id <= 11; id++) {
+ insert.setInt(1, id);
+ insert.setDate(2, Date.valueOf(LocalDate.of(2024, 1,
1).plusDays(id - 1)));
+ insert.addBatch();
+ }
+ insert.executeBatch();
+ }
+ for (int partitions : new int[] {3, 20}) {
+ JdbcSourceTable table =
+ JdbcSourceTable.builder()
+ .tablePath(TablePath.of("main", "t"))
+ .query("SELECT id, create_date FROM t")
+ .partitionColumn("create_date")
+ .partitionStart("2024-01-01")
+ .partitionEnd("2024-01-10")
+ .partitionNumber(partitions)
+ .build();
+ Collection<JdbcSourceSplit> splits =
splitter.createSplits(table, dateKey());
+ assertEquals(Math.min(partitions, 10), splits.size());
+ List<Long> ids = readAllIds(splitter, splits);
+ Collections.sort(ids);
+ assertEquals(
+ LongStream.rangeClosed(1,
10).boxed().collect(Collectors.toList()),
+ ids,
+ "DATE ranges must retain every day exactly once and
exclude outsiders");
+ }
+ }
+ }
+
private static List<Long> readAllIds(ChunkSplitter splitter,
Collection<JdbcSourceSplit> splits)
throws SQLException {
List<Long> ids = new ArrayList<>();
diff --git
a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-7/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcMysqlSplitIT.java
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-7/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcMysqlSplitIT.java
index fe850d5f9c3..aa7d7cbea68 100644
---
a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-7/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcMysqlSplitIT.java
+++
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-7/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcMysqlSplitIT.java
@@ -355,6 +355,93 @@ public class JdbcMysqlSplitIT extends TestSuiteBase
implements TestResource {
private JdbcUrlUtil.UrlInfo mysqlUrlInfo;
+ @Test
+ public void testFixedDateSplitsReadEveryRow() throws Exception {
+ String tableName = "split_date_review";
+ TablePath tablePath = TablePath.of(MYSQL_DATABASE, tableName);
+ Map<String, Object> configMap = new HashMap<>();
+ configMap.put("url", mysqlUrlInfo.getUrlWithDatabase().get());
+ configMap.put("driver", "com.mysql.cj.jdbc.Driver");
+ configMap.put("user", MYSQL_USERNAME);
+ configMap.put("password", MYSQL_PASSWORD);
+ configMap.put("query", "SELECT * FROM " + tableName);
+ configMap.put("partition_column", "created_on");
+ try (Connection connection =
+ DriverManager.getConnection(
+ mysqlUrlInfo.getUrlWithDatabase().get(),
+ MYSQL_USERNAME,
+ MYSQL_PASSWORD);
+ Statement statement = connection.createStatement();
+ MySqlCatalog catalog =
+ new MySqlCatalog(
+ "mysql", MYSQL_USERNAME, MYSQL_PASSWORD,
mysqlUrlInfo, null)) {
+ statement.execute(
+ "CREATE TABLE " + tableName + " (id INT PRIMARY KEY,
created_on DATE)");
+ try {
+ try (PreparedStatement insert =
+ connection.prepareStatement(
+ "INSERT INTO " + tableName + " VALUES (?,
?)")) {
+ for (int id = 0; id <= 11; id++) {
+ insert.setInt(1, id);
+ insert.setDate(2, Date.valueOf(LocalDate.of(2024, 1,
1).plusDays(id - 1)));
+ insert.addBatch();
+ }
+ insert.executeBatch();
+ }
+ catalog.open();
+ CatalogTable catalogTable = catalog.getTable(tablePath);
+ for (String[] bounds :
+ new String[][] {
+ {null, null}, {"2024-01-01", "2024-01-10"},
{"19723", "19732"}
+ }) {
+ for (int partitions : new int[] {3, 20}) {
+ JdbcSourceTable sourceTable =
+ JdbcSourceTable.builder()
+ .tablePath(tablePath)
+ .catalogTable(catalogTable)
+ .query("SELECT * FROM " + tableName)
+ .partitionColumn("created_on")
+ .partitionNumber(partitions)
+ .partitionStart(bounds[0])
+ .partitionEnd(bounds[1])
+ .build();
+ try (FixedChunkSplitter splitter =
getFixedChunkSplitter(configMap)) {
+ Collection<JdbcSourceSplit> splits =
+ splitter.generateSplits(sourceTable);
+ int firstId = bounds[0] == null ? 0 : 1;
+ int lastId = bounds[1] == null ? 11 : 10;
+ Assertions.assertEquals(
+ Math.min(partitions, lastId - firstId +
1), splits.size());
+ Set<Integer> ids = new HashSet<>();
+ for (JdbcSourceSplit split : splits) {
+ try (PreparedStatement query =
+
splitter.generateSplitStatement(
+ split,
catalogTable.getTableSchema());
+ ResultSet rows = query.executeQuery())
{
+ while (rows.next()) {
+ Assertions.assertTrue(
+ ids.add(rows.getInt("id")),
"Split rows overlap");
+ Assertions.assertEquals(
+ LocalDate.of(2024, 1, 1)
+
.plusDays(rows.getInt("id") - 1),
+
rows.getDate("created_on").toLocalDate());
+ }
+ }
+ }
+ Set<Integer> expected = new HashSet<>();
+ for (int id = firstId; id <= lastId; id++) {
+ expected.add(id);
+ }
+ Assertions.assertEquals(expected, ids);
+ }
+ }
+ }
+ } finally {
+ statement.execute("DROP TABLE IF EXISTS " + tableName);
+ }
+ }
+ }
+
@Test
public void testSplit() throws Exception {
Map<String, Object> configMap = new HashMap<>();