Ramin Gharib created FLINK-40828:
------------------------------------
Summary: Make VARIANT toString() and result printing never fail
Key: FLINK-40828
URL: https://issues.apache.org/jira/browse/FLINK-40828
Project: Flink
Issue Type: Improvement
Components: API / Core, Table SQL / Planner, Table SQL / Runtime
Reporter: Ramin Gharib
Assignee: Ramin Gharib
h3. Problem
{\{Variant#toString()}} delegates to \{{toJson()}}, which throws for values
JSON cannot represent. A VARIANT can hold such values, for example NaN from
Avro data or a UDF. It can also hold a type id written by a newer version. Such
a VARIANT breaks logs, test failure messages, and printed query results.
||Case||Current behavior||
|\{{toString()}} of \{{[1, NaN]}}|throws \{{Non-finite value NaN cannot be
serialized to JSON.}}|
|Print a VARIANT column holding \{{[1, NaN]}}|throws \{{Error when casting
VARIANT to STRING.}}|
|Print a VARIANT column holding a JSON \{{null}}|prints an empty cell|
|\{{toString()}} or print a VARIANT with an unknown type id|throws
\{{UNKNOWN_PRIMITIVE_TYPE_IN_VARIANT}}, the whole value is lost|
h3. Proposal
* Add an internal \{{VariantFormatter}} interface: \{{String format(Variant)}}.
* Move the JSON rendering out of \{{BinaryVariant}} into
\{{JsonVariantFormatter}}:
** \{{STRICT}} backs \{{toJson()}}. Output is unchanged, and it still fails on
NaN, infinity, and undecodable data, so \{{toJson()}} always returns valid JSON.
** \{{LENIENT}} backs \{{toString()}}. It never fails and is meant for
debugging. NaN becomes \{{"NaN"}}. An unknown type id becomes \{{"<UNKNOWN>"}},
other undecodable data \{{"<INVALID>"}}. The rest of the value still renders.
* Print VARIANT results through \{{SqlStringVariantFormatter}}. It renders like
\{{CAST(v AS STRING)}}, the same way MAP and ARRAY print, and never fails.
||Stored||Printed today||Printed after||
|\{{{"a":1,"b":["x",null]}}}|\{{{"a":1,"b":["x",null]}}}|\{{{a=1, b=[x,
NULL]}}}|
|\{{[1, NaN]}}|fails|\{{[1, NaN]}}|
|JSON \{{null}}|empty cell|\{{NULL}}|
|bytes \{{0xC328}}|\{{"wyg="}}|\{{x'c328'}}|
|\{{[1, <type id 31>]}}|fails|\{{[1, <UNKNOWN>]}}|
h3. Compatibility
* No public API change. The new classes are \{{@Internal}}. The javadoc of
\{{Variant#toJson()}} documents the contract.
* \{{toJson()}}, \{{JSON_STRING}}, the json and raw formats, and \{{CAST(v AS
STRING)}} are unchanged.
* Printed VARIANT output changes from JSON to the CAST form, so strings are no
longer quoted.
* No state or serialization change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
