SEPURI-SAI-KRISHNA opened a new issue, #13057:
URL: https://github.com/apache/seatunnel/issues/13057
### Search before asking
- [X] I had searched in the issues and found no similar issues.
### What happened
When the Lance sink creates a dataset, `SchemaUtils.convertSchema` declares
every integer type, `BIGINT` included, as Arrow `int32`:
```java
case TINYINT:
case SMALLINT:
case INT:
case BIGINT:
field = Field.nullable(fieldName, new ArrowType.Int(32, true));
```
`IntTypeWriter` then writes a 32-bit column with `numValue.intValue()`, so a
`BIGINT` outside the signed 32-bit range is wrapped without any error. Measured
with the real `SchemaUtils.convertSchema` and
`FragmentConverter.convertToVectorSchemaRoot`:
| BIGINT written | stored |
| --- | --- |
| `42` | `42` |
| `3000000000` | `-1294967296` |
| `1739586083123` (an epoch in milliseconds) | `124328243` |
| `9223372036854775807` | `-1` |
Nothing fails. Ids and epoch milliseconds are common `BIGINT` values, and
they are stored wrong.
`LanceTypeMapper.convertJsonArrowType`, which the catalog's `createTable`
uses, maps `BIGINT` to `int32` in the same way.
The sink documentation describes this as a known caveat ("`BIGINT` values
outside the signed 32-bit range are truncated"). Arrow has a 64-bit integer
type, and the catalog already maps `int64` to `Int(64)`
(`LanceCatalog.convertJsonArrowTypeToArrowType`), so I think the narrowing
should be fixed rather than documented.
### What you expected to happen
A `BIGINT` column is created as Arrow `int64` and every value is stored
unchanged.
### SeaTunnel Version
dev, verified at commit `f7a575432`.
### SeaTunnel Config
```conf
env {
parallelism = 1
job.mode = "BATCH"
}
source {
FakeSource {
schema = {
fields {
id = int
c_bigint = bigint
}
}
rows = [
{
kind = INSERT
fields = [1, 3000000000]
},
{
kind = INSERT
fields = [2, 1739586083123]
}
]
}
}
sink {
Lance {
dataset_path = "/tmp/lance/bigint_table"
namespace_type = "dir"
namespace_id = "root"
table = "bigint_table"
}
}
```
### Running Command
```shell
./bin/seatunnel.sh --config config/fake_to_lance.conf -e local
```
### Error Exception
```log
None. Nothing in the write path raises an error; the values are wrapped as
shown above.
```
### Zeta or Flink or Spark Version
Not engine specific. The schema is built in the sink writer.
### Java or Scala Version
Not version specific. Measured on JDK 17.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
