zhengxiang378928908-code opened a new pull request, #12882:
URL: https://github.com/apache/seatunnel/pull/12882
### Purpose of this pull request
Fixes #12752.
`DamengCatalog` took every column's length from
`ALL_TAB_COLUMNS.DATA_LENGTH`. For `NVARCHAR(n)` and `NVARCHAR2(n)` that is a
byte count, while the columns hold n characters, so `NVARCHAR(10)` was reported
as `NVARCHAR(20)` (GB18030) or `NVARCHAR(40)` (UTF-8), and the length was then
expanded again for SeaTunnel's column length.
Following @DanielLeens's review in #12752, this PR selects the metadata
explicitly and does not divide `DATA_LENGTH`.
**What Dameng actually stores.** The definition of `SYS.ALL_TAB_COLUMNS`
(from `SYS.SYSTEXTS`) is:
```sql
DATA_LENGTH = CASE ... WHEN COL.SCALE = 7 AND UNICODE() = 1 THEN COL.LENGTH$
* 4
WHEN COL.SCALE = 7 AND UNICODE() = 0 THEN COL.LENGTH$
* 2
ELSE COL.LENGTH$ END
CHAR_LENGTH = CASE WHEN COL.TYPE$ IN
('CHAR','CHARACTER','VARCHAR','VARCHAR2','NCHAR','NVARCHAR')
THEN COL.LENGTH$ ELSE 0 END -- NVARCHAR2 is not in
the list
```
`SYS.SYSCOLUMNS.LENGTH$` is the declared length. `DATA_LENGTH` multiplies it
by the charset's bytes per character for character-length columns (`SCALE =
7`), and `CHAR_LENGTH` is 0 for `NVARCHAR2` only because the view leaves it out
of the list. So `DATA_LENGTH / 2` would be wrong on a UTF-8 database.
**Change.** `SELECT_COLUMNS_SQL` also selects `ALL_TAB_COLUMNS.CHAR_LENGTH`
and `SYS.SYSCOLUMNS.LENGTH$` (LEFT JOIN through `SYS.SYSOBJECTS` on owner,
table and column name). For `NVARCHAR` and `NVARCHAR2` only, the length is
`LENGTH$`, then `CHAR_LENGTH` if `LENGTH$` is missing, then `DATA_LENGTH` (the
current behavior, which oversizes but never truncates). All other types keep
`DATA_LENGTH`, so `CHAR`, `CHARACTER`, `VARCHAR`, `VARCHAR2` and `NCHAR` are
unchanged.
**Measurements** on DM V8 (`0x7000c`, the `laglangyue/dmdb8` image used by
`JdbcDmIT`), with four databases initialized by `dminit` with each
`LENGTH_IN_CHAR` / `CHARSET` combination:
| column | LIC=0 GB18030 | LIC=1 GB18030 | LIC=0 UTF-8 | LIC=1 UTF-8 |
|---|---|---|---|---|
| `NVARCHAR(10)` DATA_LENGTH / CHAR_LENGTH / LENGTH$ | 20 / 10 / 10 | 20 /
10 / 10 | 40 / 10 / 10 | 40 / 10 / 10 |
| `NVARCHAR2(10)` DATA_LENGTH / CHAR_LENGTH / LENGTH$ | 20 / **0** / 10 | 20
/ **0** / 10 | 40 / **0** / 10 | 40 / **0** / 10 |
| `NVARCHAR(100)`, `NVARCHAR2(100)` | 200 / 100 or 0 / 100 | same | 400 /
100 or 0 / 100 | same |
| capacity of `NVARCHAR(10)` and `NVARCHAR2(10)` | 10 × `a` ok, 11 × `a`
rejected, 10 × `达` ok | same | same | same |
| `CHAR(10)`, `VARCHAR(10)`, `VARCHAR2(10)`, `NCHAR(10)` | 10 / 10 / 10 | 10
/ 10 / 10 | 10 / 10 / 10 | 10 / 10 / 10 |
So in every mode `NVARCHAR(n)` and `NVARCHAR2(n)` hold exactly n characters,
and `LENGTH$` is n. `LENGTH_IN_CHAR` does not change their metadata. A user
that only has `SELECT` on the table (default `PUBLIC` role) can read
`SYS.SYSCOLUMNS` and gets the same values.
Not changed here: `VARCHAR(n CHAR)` has the same pattern (`DATA_LENGTH`
2n/4n, `LENGTH$` n). It keeps the current `VARCHAR2(DATA_LENGTH)` mapping,
because changing it to `VARCHAR2(n)` would make a Dameng sink table byte-length
and too small for multi-byte values. Fixing it properly means keeping the
`CHAR` length semantics in the source type, so I'd suggest a separate issue if
you think it is worth it.
### Does this PR introduce _any_ user-facing change?
Yes. For Dameng `NVARCHAR(n)` / `NVARCHAR2(n)`, the source type changes from
`NVARCHAR(2n)` / `NVARCHAR(4n)` to `NVARCHAR(n)`, and the column length used
for sink auto-create is halved (GB18030) or quartered (UTF-8). For example,
`NVARCHAR(10)` gets `VARCHAR(40)` instead of `VARCHAR(80)` on a MySQL sink. The
new columns still hold every source value, since the length is counted in
characters. Existing tables are not altered. Recorded in
`incompatible-changes.md` (en/zh).
### How was this patch tested?
- `DamengCatalogTest` (new): the measured matrix above for `NVARCHAR` /
`NVARCHAR2` (including `CHAR_LENGTH = 0`), the fallbacks when `LENGTH$` or
`CHAR_LENGTH` is missing, unchanged `CHAR` / `CHARACTER` / `VARCHAR` /
`VARCHAR2` / `NCHAR` / `VARCHAR(10 CHAR)`, and `buildColumn` end to end with a
mocked `ResultSet` (source type and column length).
- `JdbcDmIT#testNationalCharLengthFromCatalog` (new): runs
`DamengCatalog#getTable` against real Dameng:
- the shared container (the image's default database: `LENGTH_IN_CHAR=0`,
GB18030), as `SYSDBA` and as a user with only `SELECT` on the table;
- a second container whose database is initialized with `LENGTH_IN_CHAR=1`
and `CHARSET=1` (UTF-8);
- the test asserts the mode of each database, the character capacity (10
Chinese characters fit, 11 × `a` is rejected), and the source type and length
of `NVARCHAR(10/100)`, `NVARCHAR2(10/100)`, `NCHAR`, `CHAR`, `VARCHAR`,
`VARCHAR2`.
- Locally: `DamengCatalogTest`, `DmdbTypeConverterTest` and
`DamengCreateTableSqlBuilderTest` pass (73 tests), and
`JdbcDmIT#testNationalCharLengthFromCatalog` passes.
### Check list
* [x] No new Jar binary package.
* [x] `incompatible-changes.md` updated (en/zh).
* [x] Not a new connector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
