zhengxiang378928908-code opened a new pull request, #12761:
URL: https://github.com/apache/seatunnel/pull/12761
### Purpose of this pull request
Part of #12754: the focused IPV4 / IPV6 step agreed in the issue. STRUCT,
TIMESTAMPTZ and the aggregate-state types are left for separate PRs.
Reading a Doris table that has an `IPV4` or `IPV6` column currently fails
while the catalog table is loaded:
```
ErrorCode:[COMMON-20], ErrorDescription:['doris' table 'db.t' unsupported
get catalog table with field data types '{"v4":"IPV4","v6":"IPV6"}']
```
The converter mapping is not the only gap. I ran this against a real Doris
4.0.6 instance, and each layer had to be fixed before the next one was reached:
1. **Catalog**: `AbstractDorisTypeConverter` has no case for `IPV4` / `IPV6`.
2. **Scan open**: `SchemaUtils.convertToSchema` calls
`desc.getType().name()`. The shaded `thrift-service` 1.0.0 predates `IPV4`,
`IPV6`, `VARIANT` and other newer `TPrimitiveType` values (1.0.1 has them), so
these deserialize to `null` and the reader fails with an NPE. That schema is
only used in a debug log.
3. **Values**: Doris BE sends `IPV4` through Arrow as a signed 32-bit
integer (`Int(32, true)`). For example, `192.168.0.1` arrives as `-1062731775`
and `255.255.255.255` as `-1`. Reading it into a `STRING` field then fails with
`ClassCastException: Integer cannot be cast to String`. `IPV6` arrives as UTF-8
text and needs no conversion.
Changes:
- `AbstractDorisTypeConverter`: map `IPV4` / `IPV6` to `STRING`, with column
lengths 15 and 39 (the longest text forms), keeping the source type.
- `DorisTypeConverterV2#reconvert`: keep `IPV4` / `IPV6` when the source
type says so, so Doris → Doris auto-create recreates the same column types.
This follows the existing `JSON` / `VARIANT` handling.
- `DorisValueReader`: convert `IPV4` fields from the Arrow integer to
dotted-decimal text. Columns are identified by their catalog source type, so
the shared `ArrowToSeatunnelRowReader` is untouched.
- `SchemaUtils`: tolerate a `null` thrift type instead of failing. I did not
bump `thrift-service`, since that lives in `seatunnel-shade` and #11703 was
reverted for release timing.
- Docs: added `IPV4` / `IPV6` to the source and sink type tables (en / zh).
### Does this PR introduce _any_ user-facing change?
Yes. Doris tables with `IPV4` / `IPV6` columns can now be read. The values
come out as text, for example `192.168.0.1` and `2001:db8::1`. Doris → Doris
auto-create keeps the `ipv4` / `ipv6` column types. No option, default or
existing mapping changes.
### How was this patch tested?
**Unit tests** (6 new):
- `DorisTypeConvertorV2Test`: `IPV4` / `IPV6` catalog mapping (type, length,
source type), and reconvert preserving both types.
- `DorisValueReaderIpv4Test`: integer to text for `0.0.0.0`,
`255.255.255.255`, `192.168.0.1`, `10.0.0.255` and the unsigned form; only
`IPV4` columns are converted (`IPV6`, `VARCHAR` and `INT` stay unchanged);
`NULL` stays `NULL`.
- `SchemaUtilsTest`: a `TScanColumnDesc` with an unknown (`null`) type no
longer throws.
```
./mvnw test -pl seatunnel-connectors-v2/connector-doris
```
**End to end on a real Doris 4.0.6.** I used the 3.0.0 binary with this
patched `connector-doris`. The source table holds the boundary values plus a
`NULL` row:
| id | IPV4 | IPV6 |
|---|---|---|
| 1 | 0.0.0.0 | :: |
| 2 | 255.255.255.255 | ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff |
| 3 | 192.168.0.1 | 2001:db8::1 |
| 4 | 10.0.0.255 | ::ffff:192.168.0.1 |
| 5 | NULL | NULL |
- Before: the job fails with the `COMMON-20` error above.
- After, Doris → Console: all 5 rows are printed exactly as in the table.
- After, Doris → Doris with auto-create: the target table is created with
`v4 ipv4` / `v6 ipv6`.
- Writing `IPV4` / `IPV6` text into such columns through the Doris sink
(stream load, JSON): every row matches the source with `<=>`, and
`NumberFilteredRows = 0`.
**Why there is no E2E case.** The Doris E2E image is
`doris-all-in-one-2.1.0`. On 2.1.0, `information_schema.columns` reports `IPV4`
/ `IPV6` columns as `unknown`, which I checked on a local 2.1.0 container. The
catalog therefore cannot identify them on that image. I added an `IPV4` /
`IPV6` pair to the `DorisIT` all-type round trip, then removed it again,
because it would fail for that reason rather than because of this change. If
upgrading the E2E image is acceptable, I'm happy to add the case in a follow-up.
**A related observation, not addressed here.** Because the shaded
`thrift-service` 1.0.0 also lacks `VARIANT`, reading `VARIANT` columns probably
hit the same NPE in `SchemaUtils` before this change. The `null` handling here
avoids it, but I have not verified `VARIANT` reads end to end.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
— no new dependency
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs — source and sink
type mapping tables updated (en / zh)
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR — not applicable, backward compatible
* [x] If you are contributing the connector code, please check that the
following files are updated — not applicable, no new connector
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
