This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-13061-e1fe5c5ef22c71a483dffad3f34fe202774a2d61 in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit 535d4237dd4b6ae92b14a9cbd790acf8d81cc71d Author: Sepuri Sai Krishna <[email protected]> AuthorDate: Mon Nov 9 14:46:20 2026 +0000 [Fix][Connector-V2] Create Lance BIGINT columns as int64 (#13061) --- docs/en/connectors/sink/Lance.md | 4 +- .../introduction/concepts/incompatible-changes.md | 7 +++ docs/zh/connectors/sink/Lance.md | 4 +- .../introduction/concepts/incompatible-changes.md | 7 +++ .../seatunnel/lance/catalog/LanceCatalog.java | 8 ++- .../seatunnel/lance/data/LanceTypeMapper.java | 8 ++- .../seatunnel/lance/utils/SchemaUtils.java | 5 +- .../lance/namespace/LanceCatalogTest.java | 6 ++ .../lance/utils/FragmentConverterTest.java | 57 +++++++++++++++++ .../seatunnel/e2e/connector/lance/LanceIT.java | 40 ++++++++++++ .../test/resources/lance/fake_bigint_to_lance.conf | 71 ++++++++++++++++++++++ 11 files changed, 209 insertions(+), 8 deletions(-) diff --git a/docs/en/connectors/sink/Lance.md b/docs/en/connectors/sink/Lance.md index eac7f6d701..681cc3322f 100644 --- a/docs/en/connectors/sink/Lance.md +++ b/docs/en/connectors/sink/Lance.md @@ -130,7 +130,7 @@ replica becomes a bottleneck. See [Sink Common Options](../common-options/sink-c ## Data Type Mapping -Lance uses the Apache Arrow type system. The sink creates the Lance schema from the incoming SeaTunnel schema. The current mapping narrows every integer SeaTunnel type (`TINYINT`, `SMALLINT`, `INT`, `BIGINT`) to Arrow `int32`, so `BIGINT` values outside the signed 32-bit range are truncated. +Lance uses the Apache Arrow type system. The sink creates the Lance schema from the incoming SeaTunnel schema. `TINYINT`, `SMALLINT` and `INT` are written as Arrow `int32`, and `BIGINT` as Arrow `int64`. When the dataset already exists, the sink writes to its existing column types, so a `BIGINT` column that an earlier version created as `int32` still truncates values outside the signed 32-bit range. | SeaTunnel Data Type | Lance / Arrow Data Type | |---------------------|-------------------------| @@ -138,7 +138,7 @@ Lance uses the Apache Arrow type system. The sink creates the Lance schema from | TINYINT | int32 | | SMALLINT | int32 | | INT | int32 | -| BIGINT | int32 (values outside the signed 32-bit range are truncated) | +| BIGINT | int64 | | FLOAT | float32 | | DOUBLE | float64 | | DECIMAL | decimal128 | diff --git a/docs/en/introduction/concepts/incompatible-changes.md b/docs/en/introduction/concepts/incompatible-changes.md index 61d47aa3e6..93bbd5f334 100644 --- a/docs/en/introduction/concepts/incompatible-changes.md +++ b/docs/en/introduction/concepts/incompatible-changes.md @@ -5,6 +5,13 @@ You need to check this document before you upgrade to related version. ## dev +### Lance sink: BIGINT columns are created as int64 + +- **Behavior change: a `BIGINT` column in a dataset the Lance sink creates is Arrow `int64`** + - **Description**: When the Lance sink created a dataset, it declared `BIGINT` columns as Arrow `int32` and wrote each value with `Number.intValue()`, so a value outside the signed 32-bit range was silently wrapped. For example, `3000000000` was stored as `-1294967296`, and the epoch millisecond `1739586083123` as `124328243`. The sink, and the catalog's `createTable`, now declare `BIGINT` as `int64` and store the value unchanged. + - **Impact**: A dataset created by this version has an `int64` column where an earlier version created `int32`. A dataset that already exists keeps its column types, because the sink writes to the schema it finds, so its `int32` column still wraps large values. + - **Migration Guide**: No configuration change is needed. If a consumer expects `int32` for these columns, update it to read `int64`. To store large values in a dataset created by an earlier version, recreate the dataset and write the rows again from the source. + ### Databend source: a DECIMAL column is read as DECIMAL instead of STRING - **Behavior change: a Databend `DECIMAL(p, s)` column reaches the job as `DECIMAL(p, s)`** diff --git a/docs/zh/connectors/sink/Lance.md b/docs/zh/connectors/sink/Lance.md index 7115fb3ead..83ab68cbca 100644 --- a/docs/zh/connectors/sink/Lance.md +++ b/docs/zh/connectors/sink/Lance.md @@ -122,7 +122,7 @@ lance.write.storage.options = { ## 数据类型映射 -Lance 使用 Apache Arrow 类型系统。sink 会根据上游 SeaTunnel 表结构创建 Lance schema。当前映射会把所有整数类型(`TINYINT`、`SMALLINT`、`INT`、`BIGINT`)一律收窄为 Arrow `int32`,因此超出有符号 32 位范围的 `BIGINT` 值会被截断。 +Lance 使用 Apache Arrow 类型系统。sink 会根据上游 SeaTunnel 表结构创建 Lance schema。`TINYINT`、`SMALLINT` 和 `INT` 写为 Arrow `int32`,`BIGINT` 写为 Arrow `int64`。如果数据集已经存在,sink 会按已有列的类型写入,因此早期版本创建为 `int32` 的 `BIGINT` 列,仍会截断超出有符号 32 位范围的值。 | SeaTunnel 数据类型 | Lance / Arrow 数据类型 | |--------------------|------------------------| @@ -130,7 +130,7 @@ Lance 使用 Apache Arrow 类型系统。sink 会根据上游 SeaTunnel 表结 | TINYINT | int32 | | SMALLINT | int32 | | INT | int32 | -| BIGINT | int32(超出有符号 32 位范围的值会被截断) | +| BIGINT | int64 | | FLOAT | float32 | | DOUBLE | float64 | | DECIMAL | decimal128 | diff --git a/docs/zh/introduction/concepts/incompatible-changes.md b/docs/zh/introduction/concepts/incompatible-changes.md index a1fd24b43e..cde3e0c4b7 100644 --- a/docs/zh/introduction/concepts/incompatible-changes.md +++ b/docs/zh/introduction/concepts/incompatible-changes.md @@ -4,6 +4,13 @@ ## dev +### Lance Sink:BIGINT 列创建为 int64 + +- **行为变更:Lance Sink 创建的数据集中,`BIGINT` 列为 Arrow `int64`** + - **描述**:Lance Sink 创建数据集时,会把 `BIGINT` 列声明为 Arrow `int32`,并用 `Number.intValue()` 写入每个值,因此超出有符号 32 位范围的值会被静默回绕。例如 `3000000000` 被存为 `-1294967296`,毫秒时间戳 `1739586083123` 被存为 `124328243`。现在 Sink 以及 Catalog 的 `createTable` 都把 `BIGINT` 声明为 `int64`,值按原样存储。 + - **影响**:本版本创建的数据集中该列为 `int64`,而早期版本创建的是 `int32`。已经存在的数据集保留原有的列类型,因为 Sink 会按已有的 schema 写入,所以其中的 `int32` 列仍会回绕较大的值。 + - **迁移指南**:无需修改配置。如果下游按 `int32` 读取这些列,请改为读取 `int64`。若要在早期版本创建的数据集中存储较大的值,请重新创建数据集,并从数据源重新写入数据。 + ### Databend Source:DECIMAL 列按 DECIMAL 读取,而不再是 STRING - **行为变更:Databend 的 `DECIMAL(p, s)` 列在作业中的类型为 `DECIMAL(p, s)`** diff --git a/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/catalog/LanceCatalog.java b/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/catalog/LanceCatalog.java index 70c10c27b1..d2a2bae633 100644 --- a/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/catalog/LanceCatalog.java +++ b/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/catalog/LanceCatalog.java @@ -447,7 +447,13 @@ public class LanceCatalog implements Catalog { new com.lancedb.lance.namespace.model.JsonArrowDataType(); if (arrowType instanceof org.apache.arrow.vector.types.pojo.ArrowType.Int) { - jsonType.setType("int32"); + org.apache.arrow.vector.types.pojo.ArrowType.Int intType = + (org.apache.arrow.vector.types.pojo.ArrowType.Int) arrowType; + if (intType.getBitWidth() == 64) { + jsonType.setType(intType.getIsSigned() ? "int64" : "uint64"); + } else { + jsonType.setType("int32"); + } } else if (arrowType instanceof org.apache.arrow.vector.types.pojo.ArrowType.Utf8) { jsonType.setType("utf8"); } else if (arrowType instanceof org.apache.arrow.vector.types.pojo.ArrowType.Bool) { diff --git a/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/data/LanceTypeMapper.java b/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/data/LanceTypeMapper.java index 0f824b44cf..3d20469359 100644 --- a/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/data/LanceTypeMapper.java +++ b/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/data/LanceTypeMapper.java @@ -49,12 +49,13 @@ public class LanceTypeMapper { case "int8": case "int16": case "int32": - case "int64": case "uint8": case "uint16": case "uint32": case "uint64": return BasicType.INT_TYPE; + case "int64": + return BasicType.LONG_TYPE; case "utf8": case "largeutf8": case "string": @@ -91,11 +92,14 @@ public class LanceTypeMapper { switch (type.getSqlType()) { case TINYINT: case SMALLINT: - case BIGINT: case INT: JsonArrowDataType intType = new JsonArrowDataType(); intType.setType("int32"); return intType; + case BIGINT: + JsonArrowDataType bigintType = new JsonArrowDataType(); + bigintType.setType("int64"); + return bigintType; case STRING: JsonArrowDataType stringType = new JsonArrowDataType(); stringType.setType("utf8"); diff --git a/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/SchemaUtils.java b/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/SchemaUtils.java index ffe9eee501..93a8f37ad2 100644 --- a/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/SchemaUtils.java +++ b/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/SchemaUtils.java @@ -69,10 +69,13 @@ public class SchemaUtils { case TINYINT: case SMALLINT: case INT: - case BIGINT: field = Field.nullable(fieldName, new ArrowType.Int(32, true)); fieldList.add(field); break; + case BIGINT: + field = Field.nullable(fieldName, new ArrowType.Int(64, true)); + fieldList.add(field); + break; case FLOAT: case DOUBLE: field = diff --git a/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/namespace/LanceCatalogTest.java b/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/namespace/LanceCatalogTest.java index b5d62f1806..0a2c31089d 100644 --- a/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/namespace/LanceCatalogTest.java +++ b/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/namespace/LanceCatalogTest.java @@ -117,6 +117,12 @@ public class LanceCatalogTest { // The getTable() should return the same table structure as created, including primary key // and comment Assertions.assertEquals(templateTable.toString(), table.toString()); + // TableSchema.toString() leaves out the columns, so check the integer types directly. The + // directory namespace returns no schema, so these come from the dataset's own schema. + Assertions.assertEquals( + BasicType.INT_TYPE, table.getTableSchema().getColumn("integer_col").getDataType()); + Assertions.assertEquals( + BasicType.LONG_TYPE, table.getTableSchema().getColumn("long_col").getDataType()); } @Test diff --git a/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/FragmentConverterTest.java b/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/FragmentConverterTest.java index 3e042d6733..e2d539d592 100644 --- a/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/FragmentConverterTest.java +++ b/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/FragmentConverterTest.java @@ -17,6 +17,8 @@ package org.apache.seatunnel.connectors.seatunnel.lance.utils; +import org.apache.seatunnel.api.table.catalog.PhysicalColumn; +import org.apache.seatunnel.api.table.catalog.TableSchema; import org.apache.seatunnel.api.table.type.ArrayType; import org.apache.seatunnel.api.table.type.BasicType; import org.apache.seatunnel.api.table.type.LocalTimeType; @@ -27,6 +29,7 @@ import org.apache.seatunnel.api.table.type.SeaTunnelRowType; import org.apache.seatunnel.connectors.seatunnel.lance.exception.LanceConnectorException; import org.apache.arrow.memory.RootAllocator; +import org.apache.arrow.vector.BigIntVector; import org.apache.arrow.vector.TimeStampMicroTZVector; import org.apache.arrow.vector.TimeStampMilliVector; import org.apache.arrow.vector.VarCharVector; @@ -290,6 +293,60 @@ class FragmentConverterTest { } } + @Test + void shouldKeepEveryBitOfABigint() { + SeaTunnelRowType rowType = + new SeaTunnelRowType( + new String[] {"c_bigint"}, + new SeaTunnelDataType<?>[] {BasicType.LONG_TYPE}); + // Every value is outside the int32 range. The third is an epoch in milliseconds. + long[] values = { + 3_000_000_000L, -3_000_000_000L, 1_739_586_083_123L, Long.MAX_VALUE, Long.MIN_VALUE + }; + List<SeaTunnelRow> rows = + Arrays.asList( + new SeaTunnelRow(new Object[] {values[0]}), + new SeaTunnelRow(new Object[] {values[1]}), + new SeaTunnelRow(new Object[] {values[2]}), + new SeaTunnelRow(new Object[] {values[3]}), + new SeaTunnelRow(new Object[] {values[4]})); + // The schema the sink declares when it creates the dataset itself. + Schema schema = SchemaUtils.convertSchema(rows.get(0), rowType); + Assertions.assertEquals(new ArrowType.Int(64, true), schema.getFields().get(0).getType()); + + try (RootAllocator allocator = new RootAllocator(); + VectorSchemaRoot root = + FragmentConverter.convertToVectorSchemaRoot( + rows, rowType, schema, allocator)) { + BigIntVector vector = (BigIntVector) root.getVector("c_bigint"); + for (int i = 0; i < values.length; i++) { + Assertions.assertEquals(values[i], vector.get(i)); + } + } + } + + @Test + void shouldDeclareABigintColumnAsInt64InTheCatalogSchema() { + TableSchema tableSchema = + TableSchema.builder() + .column( + PhysicalColumn.of( + "c_bigint", + BasicType.LONG_TYPE, + (Long) null, + true, + null, + null)) + .build(); + Assertions.assertEquals( + "int64", + SchemaUtils.convertJsonArrowSchema(tableSchema) + .getFields() + .get(0) + .getType() + .getType()); + } + private static long epochMicros(LocalDateTime value) { Instant instant = value.atZone(ZoneId.systemDefault()).toInstant(); return ChronoUnit.MICROS.between(Instant.EPOCH, instant); diff --git a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/java/org/apache/seatunnel/e2e/connector/lance/LanceIT.java b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/java/org/apache/seatunnel/e2e/connector/lance/LanceIT.java index dddca4518d..ac2cb504c8 100644 --- a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/java/org/apache/seatunnel/e2e/connector/lance/LanceIT.java +++ b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/java/org/apache/seatunnel/e2e/connector/lance/LanceIT.java @@ -26,6 +26,7 @@ import org.apache.seatunnel.e2e.common.junit.TestContainerExtension; import org.apache.arrow.memory.BufferAllocator; import org.apache.arrow.memory.RootAllocator; +import org.apache.arrow.vector.BigIntVector; import org.apache.arrow.vector.IntVector; import org.apache.arrow.vector.TimeStampVector; import org.apache.arrow.vector.VectorSchemaRoot; @@ -68,6 +69,7 @@ public class LanceIT extends TestSuiteBase { private static final String DATASET_PATH = "/tmp/seatunnel_mnt/lanceTest/"; private static final String TABLE_NAME = "lance_sink_table"; private static final String TIMESTAMP_TABLE_NAME = "lance_timestamp_table"; + private static final String BIGINT_TABLE_NAME = "lance_bigint_table"; @TestContainerExtension protected final ContainerExtendedFactory extendedFactory = @@ -157,6 +159,44 @@ public class LanceIT extends TestSuiteBase { return fractions; } + @TestTemplate + public void testBigintKeepsItsValue(TestContainer container) + throws IOException, InterruptedException { + Container.ExecResult writeResult = container.executeJob("/lance/fake_bigint_to_lance.conf"); + Assertions.assertEquals(0, writeResult.getExitCode(), writeResult.getStderr()); + + Map<Integer, Long> expected = new HashMap<>(); + expected.put(1, 3_000_000_000L); + expected.put(2, -3_000_000_000L); + expected.put(3, 1_739_586_083_123L); + expected.put(4, Long.MAX_VALUE); + + given().ignoreExceptions() + .await() + .atMost(60000, TimeUnit.MILLISECONDS) + .untilAsserted(() -> Assertions.assertEquals(expected, readBigints())); + } + + private Map<Integer, Long> readBigints() throws Exception { + Map<Integer, Long> values = new HashMap<>(); + try (BufferAllocator allocator = new RootAllocator(); + Dataset dataset = Dataset.open(DATASET_PATH + BIGINT_TABLE_NAME, allocator); + LanceScanner scanner = dataset.newScan(); + ArrowReader reader = scanner.scanBatches()) { + while (reader.loadNextBatch()) { + VectorSchemaRoot root = reader.getVectorSchemaRoot(); + IntVector ids = (IntVector) root.getVector("id"); + BigIntVector bigints = (BigIntVector) root.getVector("c_bigint"); + for (int i = 0; i < root.getRowCount(); i++) { + // A duplicated row would otherwise overwrite the first one unnoticed. + Assertions.assertNull( + values.put(ids.get(i), bigints.get(i)), "duplicate id " + ids.get(i)); + } + } + } + return values; + } + private long loadLanceTableCount() { long count = 0; try { diff --git a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/resources/lance/fake_bigint_to_lance.conf b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/resources/lance/fake_bigint_to_lance.conf new file mode 100644 index 0000000000..241d56a3a8 --- /dev/null +++ b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/resources/lance/fake_bigint_to_lance.conf @@ -0,0 +1,71 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +env { + parallelism = 1 + job.mode = "BATCH" + + # You can set spark configuration here + spark.app.name = "SeaTunnel" + spark.executor.instances = 2 + spark.executor.cores = 1 + spark.executor.memory = "1g" + spark.master = local +} + +source { + FakeSource { + schema = { + fields { + id = int + c_bigint = bigint + } + } + # Each value is outside the int32 range. The third is an epoch in milliseconds. + rows = [ + { + kind = INSERT + fields = [1, 3000000000] + }, + { + kind = INSERT + fields = [2, -3000000000] + }, + { + kind = INSERT + fields = [3, 1739586083123] + }, + { + kind = INSERT + fields = [4, 9223372036854775807] + } + ] + plugin_output = "fake" + } +} + +transform { +} + +sink { + Lance { + dataset_path = "/tmp/seatunnel_mnt/lanceTest/lance_bigint_table" + namespace_type = "dir" + namespace_id = "root" + table = "lance_bigint_table" + } +}
