raminqaf opened a new pull request, #29311:
URL: https://github.com/apache/flink/pull/29311
> **Depends on #29310 (FLINK-40828).** This PR is stacked on it. Only the
last two commits belong here: `[FLINK-40825][table] Keep a stored NaN or
infinity when casting VARIANT to FLOAT or DOUBLE` and `[FLINK-40825][table]
Support casting primitive types to VARIANT`. I will rebase once #29310 is
merged.
## What is the purpose of the change
FLIP-521 lists the types that cast to and from VARIANT. The direction
VARIANT to SQL type exists, but `SqlCastFunction` rejected every cast to
VARIANT. This PR adds `CAST` and `TRY_CAST` from primitive types to VARIANT.
Constructed types follow in FLINK-40826.
A type casts to VARIANT only if a VARIANT kind holds its value without loss.
Any other type is rejected at validation.
| Expression | Result
|
|--------------------------------------------|------------------------------------------|
| `CAST(CAST(1 AS BIGINT) AS VARIANT)` | `1`, stored as BIGINT
|
| `CAST(42 AS VARIANT)` | `42`, stored as INT
|
| `CAST('{"a": 1}' AS VARIANT)` | the string `'{"a": 1}'`, not
an object |
| `CAST(CAST(NULL AS INT) AS VARIANT)` | SQL `NULL`, not a variant
null |
| `CAST(CAST('NaN' AS DOUBLE) AS VARIANT)` | `NaN`, stored as DOUBLE
|
| `CAST(INTERVAL '2' DAY AS VARIANT)` | fails at validation
|
| `CAST(ARRAY[1, 2] AS VARIANT)` | fails at validation,
FLINK-40826 |
## Brief change log
- Casting a VARIANT to FLOAT or DOUBLE keeps a stored NaN or infinity. The
FLOAT cast rejects only a finite value that does not fit, such as `1e40`.
Before, a VARIANT holding NaN, for example from the Avro converter, could not
be read back. This is the first commit.
- `SqlCastFunction#canCastFrom` routes a VARIANT target to
`LogicalTypeCasts` instead of rejecting it, the same way it handles UUID.
- `LogicalTypeCasts` declares the supported sources explicitly: BOOLEAN,
the numeric types, CHAR, VARCHAR, BINARY, VARBINARY, DATE, TIME, TIMESTAMP,
TIMESTAMP_LTZ, and UUID. The cast is explicit only.
- New `PrimitiveToVariantCastRule`, backed by `VariantCastUtils#fromXxx`
helpers. The rule never fails.
- Docs: a "cast to VARIANT" section in `data-types.md`, the VARIANT column
of the cast matrix, and TIME and UUID in the list of VARIANT kinds.
Notes for reviewers:
- An integer keeps the width of its SQL type. `PARSE_JSON('1')` picks the
smallest kind because JSON text carries no width, but a cast keeps the declared
type. Both are valid under the Parquet Variant spec, which treats all integer
widths as one equivalence class.
- A string is wrapped as a STRING kind and never parsed. `PARSE_JSON`
stays the way to parse JSON text.
- NaN and infinity are stored. The binary format holds IEEE numbers, and
`PARSE_JSON` rejects them only because JSON has no literal for them.
`JSON_STRING` still fails on such a value. Printing and `CAST(v AS STRING)`
show `NaN`, which relies on #29310.
- Timestamps choose microsecond or nanosecond precision by value, as
`VariantBuilder` does. Following the declared precision instead is an open
question on FLINK-40825.
- The explicit-only choice is deliberate. Making the cast implicit later
is additive, but taking implicit back would break queries.
- `SqlCastFunction` is a copy of the Calcite class. The change replaces
its existing TODO to support casts to VARIANT.
- A `LogicalTypeCastsTest` row asserted that UUID does not cast to
VARIANT. It now asserts that it does.
## Verifying this change
This change added tests and can be verified as follows:
- `LogicalTypeCastsTest` covers every supported source, and rejects
INTERVAL, TIMESTAMP WITH TIME ZONE, MULTISET, ARRAY, MAP, and ROW.
- `CastRuleProviderTest` checks rule resolution, that the cast never
fails, and that VARIANT to VARIANT stays the identity.
- `CastRulesTest` checks the stored kind for every source, including the
kept integer width, `TIMESTAMP(3)`, `TIMESTAMP(6)`, and `TIMESTAMP(9)`, NaN and
infinity, and SQL NULL. It also covers NaN and infinity read back from VARIANT
to FLOAT and DOUBLE.
- `CastFunctionITCase` round-trips each type through VARIANT in SQL and
the Table API, for literals and for values computed at runtime, including NaN
and infinity. It also checks the validation errors.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): yes. The new
cast runs per record and builds one VARIANT per value.
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs, in `data-types.md` (English
and Chinese)
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
