zhengxiang378928908-code opened a new pull request, #12787: URL: https://github.com/apache/seatunnel/pull/12787
### Purpose of this pull request Fixes #12753. `AbstractDorisTypeConverter#convert` declared Doris `LARGEINT` (128-bit signed integer, up to 39 digits) as `DECIMAL(20, 0)`, so any non-Doris sink failed on values longer than 20 digits. Doris → Doris only worked because of the `sourceType = largeint` special case in `reconvert()`. This PR declares scalar `LARGEINT` and the `ARRAY<largeint>` element type as `DECIMAL(39, 0)`, and keeps the Doris → Doris round trip mapping back to `LARGEINT`. ### Does this PR introduce _any_ user-facing change? Yes. The source schema of a Doris `LARGEINT` column changes from `DECIMAL(20, 0)` to `DECIMAL(39, 0)`. Docs (en/zh) updated. ### Engine limit found: Spark cannot represent `DECIMAL(39, 0)` As asked in #12753, describing it here instead of clamping: - **Component:** `seatunnel-translation-spark-common`, `TypeConverterUtils.convert` (line 111), which builds `org.apache.spark.sql.types.DecimalType(39, 0)`. - **Error:** `org.apache.spark.sql.AnalysisException: decimal can only support precision up to 38` - **Stage:** source schema translation, before any row is read: `SourceExecuteProcessor.execute` → `DataFrameReader.load` → `BatchSourceReader.readSchema` → `MultiTableManager.getTableSchema` → `TypeConverterUtils.parcel`. - **Scope:** Spark 2.4 and Spark 3.3 only. Zeta and Flink pass. Because the failure is at schema time, **any** Doris table that has a `LARGEINT` column fails on Spark, regardless of the actual values. Before this PR, the same tables worked on Spark as long as values fit in 20 digits. - **CI evidence** (fork run https://github.com/zhengxiang378928908-code/seatunnel/actions/runs/38510021504), same result on JDK 11 and 17; every failure is a Spark container (`[6]` Spark 2.4, `[7]` Spark 3.3): - `DorisLargeintToMysqlIT.testLargeintToMysqlAutoCreate` (new) - `DorisIT.testDoris`, `DorisIT.testDorisDirectToBe`, `DorisIT.testNoSchemaDoris` (existing, passed before this PR) - `DorisMultiReadIT.testDorisMultiRead` (existing, passed before this PR) Options I see, would like your call before changing anything: 1. **Handle it in the Spark translation layer** (separate PR, merged first): when a SeaTunnel `DecimalType` has precision > 38, map it explicitly to a type Spark can carry (for example `StringType`) and convert values at the row boundary, with a log line and a docs note. This keeps existing Spark + Doris jobs working. 2. **Fail fast on Spark and document it:** throw a clear `SeaTunnelRuntimeException` naming the column and the 38-digit limit, disable the affected Doris e2e cases on Spark with a reason, and document that `LARGEINT` is unsupported on Spark. ### How was this patch tested? - Unit: `DorisTypeConvertorV1Test`, `DorisTypeConvertorV2Test`, new `DorisLargeintReadTest`. - E2E: new `DorisLargeintToMysqlIT` (Doris `LARGEINT` → MySQL `DECIMAL(39,0)`, including min/max 128-bit values); updated `DorisIT`, `DorisMultiReadIT`. Passing on Zeta and Flink; failing on Spark as described above. ### Check list * [x] No new Jar binary package. * [ ] Update the [`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
