Ramin Gharib created FLINK-40305:
------------------------------------
Summary: VARIANT string and object key decoding ignores the UTF-8
charset
Key: FLINK-40305
URL: https://issues.apache.org/jira/browse/FLINK-40305
Project: Flink
Issue Type: Bug
Components: API / Core
Reporter: Ramin Gharib
Assignee: Ramin Gharib
{\{BinaryVariantUtil}} decodes variant strings and object field names without
passing a charset, so both fall back to the JVM default:
{code:java}
// getString
return new String(value, start, length);
// getMetadataKey
return new String(metadata, stringStart + offset, nextOffset - offset);
{code}
Both are written as UTF-8 by \{{BinaryVariantInternalBuilder}}, which uses
\{{str.getBytes(StandardCharsets.UTF_8)}} for values and
\{{key.getBytes(StandardCharsets.UTF_8)}} for keys, so the write and read paths
disagree.
The default charset is only guaranteed to be UTF-8 from Java 18 onwards (JEP
400). On Java 11 and 17 it is platform dependent, so a variant holding
non-ASCII text can decode incorrectly. \{{new String(byte[], ...)}} also
substitutes \{{U+FFFD}} for malformed input rather than failing.
{\{getMetadataKey}} is the more damaging of the two, since it decodes field
names. A mangled name makes \{{getField(name)}} silently return null and
corrupts \{{getFieldNames()}} and \{{toJson()}}.
This is a regression from the original port. Spark's \{{VariantUtil}}, which
this class is derived from, passes the charset in both places:
{code:java}
return new String(value, start, length, StandardCharsets.UTF_8);
return new String(metadata, stringStart + offset, nextOffset - offset,
StandardCharsets.UTF_8);
{code}
h3. Fix
Pass \{{StandardCharsets.UTF_8}} in both calls. Add a round-trip test with a
non-ASCII string value and a non-ASCII field name.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
