raminqaf commented on code in PR #29342:
URL: https://github.com/apache/flink/pull/29342#discussion_r4152584065
##########
flink-core/src/test/java/org/apache/flink/types/variant/BinaryVariantInternalBuilderTest.java:
##########
@@ -134,6 +141,37 @@ void testParseJsonWithNonAsciiStringsAndKeys() throws
IOException {
assertThat(variant.toJson()).isEqualTo(json);
}
+ @Test
+ void testParseJsonFromUtf8Bytes() throws IOException {
+ final String json = "{\"schlüssel\":\"Grüße, 世界 🚀\",\"キー\":[\"äöü\"]}";
+
+
assertThat(BinaryVariantInternalBuilder.parseJson(json.getBytes(UTF_8), false))
+ .isEqualTo(BinaryVariantInternalBuilder.parseJson(json,
false));
+ }
+
+ private static Stream<Arguments> nonUtf8JsonBytes() {
+ return Stream.of(
+ // Charset detection would read these as UTF-16 or UTF-32, or
skip the BOM.
+ Arguments.of("trailing NUL", "1\u0000".getBytes(UTF_8)),
+ Arguments.of("leading NUL", "\u00001".getBytes(UTF_8)),
+ Arguments.of("UTF-8 BOM", "\uFEFF1".getBytes(UTF_8)),
+ Arguments.of("UTF-16BE", "{\"a\":1}".getBytes(UTF_16BE)),
+ Arguments.of("UTF-16LE", "{\"a\":1}".getBytes(UTF_16LE)),
+ Arguments.of("UTF-16 with BOM", "{\"a\":1}".getBytes(UTF_16)),
+ Arguments.of("invalid start byte", new byte[] {'"', (byte)
0xFF, '"'}),
+ Arguments.of("truncated sequence", new byte[] {'"', 'a',
(byte) 0xC3, '"'}),
+ Arguments.of(
+ "surrogate code point",
+ new byte[] {'"', (byte) 0xED, (byte) 0xA0, (byte)
0x80, '"'}));
+ }
+
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("nonUtf8JsonBytes")
+ void testParseJsonRejectsBytesThatAreNotUtf8Json(final String name, final
byte[] bytes) {
+ assertThatThrownBy(() -> BinaryVariantInternalBuilder.parseJson(bytes,
false))
+ .isInstanceOf(IOException.class);
Review Comment:
No. `BinaryVariantInternalBuilder` is `@Internal`, and the public Variant
and VariantBuilder APIs have no JSON parsing, so UDFs never reach this method.
Its only callers are `PARSE_JSON`, which wraps any failure in a
`TableRuntimeException`, and `TRY_PARSE_JSON`, which returns `NULL`. The
`IOException` matches the existing `parseJson(String, boolean)` overload.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
