This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 535d4237dd [Fix][Connector-V2] Create Lance BIGINT columns as int64
(#13061)
535d4237dd is described below
commit 535d4237dd4b6ae92b14a9cbd790acf8d81cc71d
Author: Sepuri Sai Krishna <[email protected]>
AuthorDate: Mon Nov 9 14:46:20 2026 +0000
[Fix][Connector-V2] Create Lance BIGINT columns as int64 (#13061)
---
docs/en/connectors/sink/Lance.md | 4 +-
.../introduction/concepts/incompatible-changes.md | 7 +++
docs/zh/connectors/sink/Lance.md | 4 +-
.../introduction/concepts/incompatible-changes.md | 7 +++
.../seatunnel/lance/catalog/LanceCatalog.java | 8 ++-
.../seatunnel/lance/data/LanceTypeMapper.java | 8 ++-
.../seatunnel/lance/utils/SchemaUtils.java | 5 +-
.../lance/namespace/LanceCatalogTest.java | 6 ++
.../lance/utils/FragmentConverterTest.java | 57 +++++++++++++++++
.../seatunnel/e2e/connector/lance/LanceIT.java | 40 ++++++++++++
.../test/resources/lance/fake_bigint_to_lance.conf | 71 ++++++++++++++++++++++
11 files changed, 209 insertions(+), 8 deletions(-)
diff --git a/docs/en/connectors/sink/Lance.md b/docs/en/connectors/sink/Lance.md
index eac7f6d701..681cc3322f 100644
--- a/docs/en/connectors/sink/Lance.md
+++ b/docs/en/connectors/sink/Lance.md
@@ -130,7 +130,7 @@ replica becomes a bottleneck. See [Sink Common
Options](../common-options/sink-c
## Data Type Mapping
-Lance uses the Apache Arrow type system. The sink creates the Lance schema
from the incoming SeaTunnel schema. The current mapping narrows every integer
SeaTunnel type (`TINYINT`, `SMALLINT`, `INT`, `BIGINT`) to Arrow `int32`, so
`BIGINT` values outside the signed 32-bit range are truncated.
+Lance uses the Apache Arrow type system. The sink creates the Lance schema
from the incoming SeaTunnel schema. `TINYINT`, `SMALLINT` and `INT` are written
as Arrow `int32`, and `BIGINT` as Arrow `int64`. When the dataset already
exists, the sink writes to its existing column types, so a `BIGINT` column that
an earlier version created as `int32` still truncates values outside the signed
32-bit range.
| SeaTunnel Data Type | Lance / Arrow Data Type |
|---------------------|-------------------------|
@@ -138,7 +138,7 @@ Lance uses the Apache Arrow type system. The sink creates
the Lance schema from
| TINYINT | int32 |
| SMALLINT | int32 |
| INT | int32 |
-| BIGINT | int32 (values outside the signed 32-bit range are
truncated) |
+| BIGINT | int64 |
| FLOAT | float32 |
| DOUBLE | float64 |
| DECIMAL | decimal128 |
diff --git a/docs/en/introduction/concepts/incompatible-changes.md
b/docs/en/introduction/concepts/incompatible-changes.md
index 61d47aa3e6..93bbd5f334 100644
--- a/docs/en/introduction/concepts/incompatible-changes.md
+++ b/docs/en/introduction/concepts/incompatible-changes.md
@@ -5,6 +5,13 @@ You need to check this document before you upgrade to related
version.
## dev
+### Lance sink: BIGINT columns are created as int64
+
+- **Behavior change: a `BIGINT` column in a dataset the Lance sink creates is
Arrow `int64`**
+ - **Description**: When the Lance sink created a dataset, it declared
`BIGINT` columns as Arrow `int32` and wrote each value with
`Number.intValue()`, so a value outside the signed 32-bit range was silently
wrapped. For example, `3000000000` was stored as `-1294967296`, and the epoch
millisecond `1739586083123` as `124328243`. The sink, and the catalog's
`createTable`, now declare `BIGINT` as `int64` and store the value unchanged.
+ - **Impact**: A dataset created by this version has an `int64` column where
an earlier version created `int32`. A dataset that already exists keeps its
column types, because the sink writes to the schema it finds, so its `int32`
column still wraps large values.
+ - **Migration Guide**: No configuration change is needed. If a consumer
expects `int32` for these columns, update it to read `int64`. To store large
values in a dataset created by an earlier version, recreate the dataset and
write the rows again from the source.
+
### Databend source: a DECIMAL column is read as DECIMAL instead of STRING
- **Behavior change: a Databend `DECIMAL(p, s)` column reaches the job as
`DECIMAL(p, s)`**
diff --git a/docs/zh/connectors/sink/Lance.md b/docs/zh/connectors/sink/Lance.md
index 7115fb3ead..83ab68cbca 100644
--- a/docs/zh/connectors/sink/Lance.md
+++ b/docs/zh/connectors/sink/Lance.md
@@ -122,7 +122,7 @@ lance.write.storage.options = {
## 数据类型映射
-Lance 使用 Apache Arrow 类型系统。sink 会根据上游 SeaTunnel 表结构创建 Lance
schema。当前映射会把所有整数类型(`TINYINT`、`SMALLINT`、`INT`、`BIGINT`)一律收窄为 Arrow
`int32`,因此超出有符号 32 位范围的 `BIGINT` 值会被截断。
+Lance 使用 Apache Arrow 类型系统。sink 会根据上游 SeaTunnel 表结构创建 Lance
schema。`TINYINT`、`SMALLINT` 和 `INT` 写为 Arrow `int32`,`BIGINT` 写为 Arrow
`int64`。如果数据集已经存在,sink 会按已有列的类型写入,因此早期版本创建为 `int32` 的 `BIGINT` 列,仍会截断超出有符号 32
位范围的值。
| SeaTunnel 数据类型 | Lance / Arrow 数据类型 |
|--------------------|------------------------|
@@ -130,7 +130,7 @@ Lance 使用 Apache Arrow 类型系统。sink 会根据上游 SeaTunnel 表结
| TINYINT | int32 |
| SMALLINT | int32 |
| INT | int32 |
-| BIGINT | int32(超出有符号 32 位范围的值会被截断) |
+| BIGINT | int64 |
| FLOAT | float32 |
| DOUBLE | float64 |
| DECIMAL | decimal128 |
diff --git a/docs/zh/introduction/concepts/incompatible-changes.md
b/docs/zh/introduction/concepts/incompatible-changes.md
index a1fd24b43e..cde3e0c4b7 100644
--- a/docs/zh/introduction/concepts/incompatible-changes.md
+++ b/docs/zh/introduction/concepts/incompatible-changes.md
@@ -4,6 +4,13 @@
## dev
+### Lance Sink:BIGINT 列创建为 int64
+
+- **行为变更:Lance Sink 创建的数据集中,`BIGINT` 列为 Arrow `int64`**
+ - **描述**:Lance Sink 创建数据集时,会把 `BIGINT` 列声明为 Arrow `int32`,并用
`Number.intValue()` 写入每个值,因此超出有符号 32 位范围的值会被静默回绕。例如 `3000000000` 被存为
`-1294967296`,毫秒时间戳 `1739586083123` 被存为 `124328243`。现在 Sink 以及 Catalog 的
`createTable` 都把 `BIGINT` 声明为 `int64`,值按原样存储。
+ - **影响**:本版本创建的数据集中该列为 `int64`,而早期版本创建的是 `int32`。已经存在的数据集保留原有的列类型,因为 Sink
会按已有的 schema 写入,所以其中的 `int32` 列仍会回绕较大的值。
+ - **迁移指南**:无需修改配置。如果下游按 `int32` 读取这些列,请改为读取
`int64`。若要在早期版本创建的数据集中存储较大的值,请重新创建数据集,并从数据源重新写入数据。
+
### Databend Source:DECIMAL 列按 DECIMAL 读取,而不再是 STRING
- **行为变更:Databend 的 `DECIMAL(p, s)` 列在作业中的类型为 `DECIMAL(p, s)`**
diff --git
a/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/catalog/LanceCatalog.java
b/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/catalog/LanceCatalog.java
index 70c10c27b1..d2a2bae633 100644
---
a/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/catalog/LanceCatalog.java
+++
b/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/catalog/LanceCatalog.java
@@ -447,7 +447,13 @@ public class LanceCatalog implements Catalog {
new com.lancedb.lance.namespace.model.JsonArrowDataType();
if (arrowType instanceof
org.apache.arrow.vector.types.pojo.ArrowType.Int) {
- jsonType.setType("int32");
+ org.apache.arrow.vector.types.pojo.ArrowType.Int intType =
+ (org.apache.arrow.vector.types.pojo.ArrowType.Int)
arrowType;
+ if (intType.getBitWidth() == 64) {
+ jsonType.setType(intType.getIsSigned() ? "int64" :
"uint64");
+ } else {
+ jsonType.setType("int32");
+ }
} else if (arrowType instanceof
org.apache.arrow.vector.types.pojo.ArrowType.Utf8) {
jsonType.setType("utf8");
} else if (arrowType instanceof
org.apache.arrow.vector.types.pojo.ArrowType.Bool) {
diff --git
a/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/data/LanceTypeMapper.java
b/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/data/LanceTypeMapper.java
index 0f824b44cf..3d20469359 100644
---
a/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/data/LanceTypeMapper.java
+++
b/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/data/LanceTypeMapper.java
@@ -49,12 +49,13 @@ public class LanceTypeMapper {
case "int8":
case "int16":
case "int32":
- case "int64":
case "uint8":
case "uint16":
case "uint32":
case "uint64":
return BasicType.INT_TYPE;
+ case "int64":
+ return BasicType.LONG_TYPE;
case "utf8":
case "largeutf8":
case "string":
@@ -91,11 +92,14 @@ public class LanceTypeMapper {
switch (type.getSqlType()) {
case TINYINT:
case SMALLINT:
- case BIGINT:
case INT:
JsonArrowDataType intType = new JsonArrowDataType();
intType.setType("int32");
return intType;
+ case BIGINT:
+ JsonArrowDataType bigintType = new JsonArrowDataType();
+ bigintType.setType("int64");
+ return bigintType;
case STRING:
JsonArrowDataType stringType = new JsonArrowDataType();
stringType.setType("utf8");
diff --git
a/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/SchemaUtils.java
b/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/SchemaUtils.java
index ffe9eee501..93a8f37ad2 100644
---
a/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/SchemaUtils.java
+++
b/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/SchemaUtils.java
@@ -69,10 +69,13 @@ public class SchemaUtils {
case TINYINT:
case SMALLINT:
case INT:
- case BIGINT:
field = Field.nullable(fieldName, new
ArrowType.Int(32, true));
fieldList.add(field);
break;
+ case BIGINT:
+ field = Field.nullable(fieldName, new
ArrowType.Int(64, true));
+ fieldList.add(field);
+ break;
case FLOAT:
case DOUBLE:
field =
diff --git
a/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/namespace/LanceCatalogTest.java
b/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/namespace/LanceCatalogTest.java
index b5d62f1806..0a2c31089d 100644
---
a/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/namespace/LanceCatalogTest.java
+++
b/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/namespace/LanceCatalogTest.java
@@ -117,6 +117,12 @@ public class LanceCatalogTest {
// The getTable() should return the same table structure as created,
including primary key
// and comment
Assertions.assertEquals(templateTable.toString(), table.toString());
+ // TableSchema.toString() leaves out the columns, so check the integer
types directly. The
+ // directory namespace returns no schema, so these come from the
dataset's own schema.
+ Assertions.assertEquals(
+ BasicType.INT_TYPE,
table.getTableSchema().getColumn("integer_col").getDataType());
+ Assertions.assertEquals(
+ BasicType.LONG_TYPE,
table.getTableSchema().getColumn("long_col").getDataType());
}
@Test
diff --git
a/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/FragmentConverterTest.java
b/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/FragmentConverterTest.java
index 3e042d6733..e2d539d592 100644
---
a/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/FragmentConverterTest.java
+++
b/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/FragmentConverterTest.java
@@ -17,6 +17,8 @@
package org.apache.seatunnel.connectors.seatunnel.lance.utils;
+import org.apache.seatunnel.api.table.catalog.PhysicalColumn;
+import org.apache.seatunnel.api.table.catalog.TableSchema;
import org.apache.seatunnel.api.table.type.ArrayType;
import org.apache.seatunnel.api.table.type.BasicType;
import org.apache.seatunnel.api.table.type.LocalTimeType;
@@ -27,6 +29,7 @@ import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
import
org.apache.seatunnel.connectors.seatunnel.lance.exception.LanceConnectorException;
import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
import org.apache.arrow.vector.TimeStampMicroTZVector;
import org.apache.arrow.vector.TimeStampMilliVector;
import org.apache.arrow.vector.VarCharVector;
@@ -290,6 +293,60 @@ class FragmentConverterTest {
}
}
+ @Test
+ void shouldKeepEveryBitOfABigint() {
+ SeaTunnelRowType rowType =
+ new SeaTunnelRowType(
+ new String[] {"c_bigint"},
+ new SeaTunnelDataType<?>[] {BasicType.LONG_TYPE});
+ // Every value is outside the int32 range. The third is an epoch in
milliseconds.
+ long[] values = {
+ 3_000_000_000L, -3_000_000_000L, 1_739_586_083_123L,
Long.MAX_VALUE, Long.MIN_VALUE
+ };
+ List<SeaTunnelRow> rows =
+ Arrays.asList(
+ new SeaTunnelRow(new Object[] {values[0]}),
+ new SeaTunnelRow(new Object[] {values[1]}),
+ new SeaTunnelRow(new Object[] {values[2]}),
+ new SeaTunnelRow(new Object[] {values[3]}),
+ new SeaTunnelRow(new Object[] {values[4]}));
+ // The schema the sink declares when it creates the dataset itself.
+ Schema schema = SchemaUtils.convertSchema(rows.get(0), rowType);
+ Assertions.assertEquals(new ArrowType.Int(64, true),
schema.getFields().get(0).getType());
+
+ try (RootAllocator allocator = new RootAllocator();
+ VectorSchemaRoot root =
+ FragmentConverter.convertToVectorSchemaRoot(
+ rows, rowType, schema, allocator)) {
+ BigIntVector vector = (BigIntVector) root.getVector("c_bigint");
+ for (int i = 0; i < values.length; i++) {
+ Assertions.assertEquals(values[i], vector.get(i));
+ }
+ }
+ }
+
+ @Test
+ void shouldDeclareABigintColumnAsInt64InTheCatalogSchema() {
+ TableSchema tableSchema =
+ TableSchema.builder()
+ .column(
+ PhysicalColumn.of(
+ "c_bigint",
+ BasicType.LONG_TYPE,
+ (Long) null,
+ true,
+ null,
+ null))
+ .build();
+ Assertions.assertEquals(
+ "int64",
+ SchemaUtils.convertJsonArrowSchema(tableSchema)
+ .getFields()
+ .get(0)
+ .getType()
+ .getType());
+ }
+
private static long epochMicros(LocalDateTime value) {
Instant instant = value.atZone(ZoneId.systemDefault()).toInstant();
return ChronoUnit.MICROS.between(Instant.EPOCH, instant);
diff --git
a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/java/org/apache/seatunnel/e2e/connector/lance/LanceIT.java
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/java/org/apache/seatunnel/e2e/connector/lance/LanceIT.java
index dddca4518d..ac2cb504c8 100644
---
a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/java/org/apache/seatunnel/e2e/connector/lance/LanceIT.java
+++
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/java/org/apache/seatunnel/e2e/connector/lance/LanceIT.java
@@ -26,6 +26,7 @@ import
org.apache.seatunnel.e2e.common.junit.TestContainerExtension;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
import org.apache.arrow.vector.IntVector;
import org.apache.arrow.vector.TimeStampVector;
import org.apache.arrow.vector.VectorSchemaRoot;
@@ -68,6 +69,7 @@ public class LanceIT extends TestSuiteBase {
private static final String DATASET_PATH = "/tmp/seatunnel_mnt/lanceTest/";
private static final String TABLE_NAME = "lance_sink_table";
private static final String TIMESTAMP_TABLE_NAME = "lance_timestamp_table";
+ private static final String BIGINT_TABLE_NAME = "lance_bigint_table";
@TestContainerExtension
protected final ContainerExtendedFactory extendedFactory =
@@ -157,6 +159,44 @@ public class LanceIT extends TestSuiteBase {
return fractions;
}
+ @TestTemplate
+ public void testBigintKeepsItsValue(TestContainer container)
+ throws IOException, InterruptedException {
+ Container.ExecResult writeResult =
container.executeJob("/lance/fake_bigint_to_lance.conf");
+ Assertions.assertEquals(0, writeResult.getExitCode(),
writeResult.getStderr());
+
+ Map<Integer, Long> expected = new HashMap<>();
+ expected.put(1, 3_000_000_000L);
+ expected.put(2, -3_000_000_000L);
+ expected.put(3, 1_739_586_083_123L);
+ expected.put(4, Long.MAX_VALUE);
+
+ given().ignoreExceptions()
+ .await()
+ .atMost(60000, TimeUnit.MILLISECONDS)
+ .untilAsserted(() -> Assertions.assertEquals(expected,
readBigints()));
+ }
+
+ private Map<Integer, Long> readBigints() throws Exception {
+ Map<Integer, Long> values = new HashMap<>();
+ try (BufferAllocator allocator = new RootAllocator();
+ Dataset dataset = Dataset.open(DATASET_PATH +
BIGINT_TABLE_NAME, allocator);
+ LanceScanner scanner = dataset.newScan();
+ ArrowReader reader = scanner.scanBatches()) {
+ while (reader.loadNextBatch()) {
+ VectorSchemaRoot root = reader.getVectorSchemaRoot();
+ IntVector ids = (IntVector) root.getVector("id");
+ BigIntVector bigints = (BigIntVector)
root.getVector("c_bigint");
+ for (int i = 0; i < root.getRowCount(); i++) {
+ // A duplicated row would otherwise overwrite the first
one unnoticed.
+ Assertions.assertNull(
+ values.put(ids.get(i), bigints.get(i)), "duplicate
id " + ids.get(i));
+ }
+ }
+ }
+ return values;
+ }
+
private long loadLanceTableCount() {
long count = 0;
try {
diff --git
a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/resources/lance/fake_bigint_to_lance.conf
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/resources/lance/fake_bigint_to_lance.conf
new file mode 100644
index 0000000000..241d56a3a8
--- /dev/null
+++
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/resources/lance/fake_bigint_to_lance.conf
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+env {
+ parallelism = 1
+ job.mode = "BATCH"
+
+ # You can set spark configuration here
+ spark.app.name = "SeaTunnel"
+ spark.executor.instances = 2
+ spark.executor.cores = 1
+ spark.executor.memory = "1g"
+ spark.master = local
+}
+
+source {
+ FakeSource {
+ schema = {
+ fields {
+ id = int
+ c_bigint = bigint
+ }
+ }
+ # Each value is outside the int32 range. The third is an epoch in
milliseconds.
+ rows = [
+ {
+ kind = INSERT
+ fields = [1, 3000000000]
+ },
+ {
+ kind = INSERT
+ fields = [2, -3000000000]
+ },
+ {
+ kind = INSERT
+ fields = [3, 1739586083123]
+ },
+ {
+ kind = INSERT
+ fields = [4, 9223372036854775807]
+ }
+ ]
+ plugin_output = "fake"
+ }
+}
+
+transform {
+}
+
+sink {
+ Lance {
+ dataset_path = "/tmp/seatunnel_mnt/lanceTest/lance_bigint_table"
+ namespace_type = "dir"
+ namespace_id = "root"
+ table = "lance_bigint_table"
+ }
+}