raminqaf commented on code in PR #29026:
URL: https://github.com/apache/flink/pull/29026#discussion_r3903605807
##########
flink-formats/flink-json/src/main/java/org/apache/flink/formats/json/JsonParserToRowDataConverters.java:
##########
@@ -328,7 +328,10 @@ private StringData convertToString(JsonParser jp) throws
IOException {
}
private BinaryVariant convertToVariant(JsonParser jp) throws IOException {
- return
BinaryVariantInternalBuilder.parseJson(jp.readValueAsTree().toString(), false);
+ // Read the whole value first so a parse error inside the variant
cannot desync the shared
+ // parser and drop sibling fields. traverse() streams it without
re-serializing to a String.
+ // Duplicate keys keep the last value, matching how a Jackson tree
would collapse them.
+ return
BinaryVariantInternalBuilder.parseJson(jp.readValueAsTree().traverse(), true);
Review Comment:
I have improved this. One way of skipping readValueAsTree() is to parse the
JSON till the end.
```java
private BinaryVariant convertToVariant(JsonParser jp) throws IOException {
// Stream the variant straight from the shared parser, with no
intermediate tree. If a parse
// error stops us mid-value, advance the parser to the end of the
value so the enclosing row
// parser stays in sync and sibling fields survive.
// Duplicate keys keep the last value, matching how a Jackson tree
would collapse them.
final JsonToken start = jp.currentToken();
final int baseDepth = jp.getParsingContext().getNestingDepth();
try {
return BinaryVariantInternalBuilder.parseJson(jp, true);
} catch (Throwable t) {
skipToEndOfValue(jp, start, baseDepth);
throw t;
}
}
```
So if there is an invalid value in the Variant, we force progress the
JsonParser to the end of it and let the rest as is.
Example `ROW<v VARIANT, other INT>`:
```json
{
"v" : {
"a" : {
"b" : [ "Infinity", 2 ]
},
"c" : 9
},
"other" : 5
}
```
`v` is the bad Variant. We can still fetch 5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
