[
https://issues.apache.org/jira/browse/FLINK-39623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gustavo de Morais updated FLINK-39623:
--------------------------------------
Release Note:
#### Strict UTF-8 validation in CAST(BYTES AS STRING)
`CAST(value AS STRING)` (and the equivalent `VARCHAR`/`CHAR(n)` targets) now
validates that the input bytes are well-formed UTF-8 when the source type is
`BINARY`, `VARBINARY`, or `BYTES`. Invalid byte sequences fail the job with a
`TableRuntimeException` instead of being silently substituted with the Unicode
replacement character `U+FFFD` as before.
Migration options:
- Use `MAKE_VALID_UTF8(bytes)` to keep the lenient decode (replaces invalid
sequences with `U+FFFD`).
- Use `TRY_CAST(bytes AS STRING)` to return `NULL` on invalid input.
- Filter at ingest with `WHERE IS_VALID_UTF8(bytes)` to route well-formed
records, or `WHERE NOT IS_VALID_UTF8(bytes)` for a dead-letter sink.
- Restore the prior behavior across the job by setting
`table.exec.legacy-bytes-to-string-cast` to `true`.
> Strict UTF-8 validation in CAST(BYTES AS STRING)
> ------------------------------------------------
>
> Key: FLINK-39623
> URL: https://issues.apache.org/jira/browse/FLINK-39623
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
> Reporter: Gustavo de Morais
> Assignee: Gustavo de Morais
> Priority: Major
> Labels: pull-request-available
>
> Implements the CAST behavior change from FLIP-568.
> {{CAST(bytes AS STRING)}} (and {{{}VARCHAR{}}}/{{{}CHAR(n){}}} targets) now
> validates UTF-8 and throws on invalid input, instead of silently substituting
> {{{}U+FFFD{}}}.
> *Migration:*
> * {{MAKE_VALID_UTF8(bytes)}} for lenient decode
> * {{TRY_CAST(bytes AS STRING)}} for {{NULL}} on failure
> * {{table.exec.legacy-bytes-to-string-cast=true}} to restore old behavior
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
