fmorillo7694 commented on PR #236: URL: https://github.com/apache/flink-connector-aws/pull/236#issuecomment-6013084326
Self-caught follow-up in baf3247, JSON format only. The previous commit fixed two silent-wrong-data cases in the protobuf module (null in a NOT NULL column written as the type default, DECIMAL overflow read back as null). Checking the json and avro siblings for the same shapes showed the JSON module had both, in a slightly different form: the schema we register promises something the delegated Flink JSON writer and reader do not keep. 1. NOT NULL on write. JsonSchemaConverter registers a NOT NULL column as a required, non-nullable property, but Flink's JsonRowDataSerializationSchema writes null for any null field and the GSR encode path intentionally skips validation, so the record went out contradicting its own schema. The serializer now rejects it naming the column path (top-level, nested ROW field, ARRAY element, MAP value), the same walk the schema converter does. 2. DECIMAL range on read. flink-json ends its decimal conversion in DecimalData.fromBigDecimal, which returns null when the integer part exceeds the reader's precision, so a producer with a wider schema (DECIMAL(12,2) written, DECIMAL(5,2) read) lost data without an error. The deserializer now parses to a JsonNode first, checks every DECIMAL value against the declared type (nested containers included) and fails with value, path and type. Narrower scale still rounds HALF_UP like CAST. Error wrapping is unchanged (IOException "Failed to deserialize JSON" with the cause), and flink-shaded-jackson is declared provided exactly as flink-json itself declares it. Avro is not affected: Flink's Avro writer already throws on null for a non-union field, and decimal precision and scale are part of the Avro schema, so a mismatch surfaces at schema resolution rather than as a null. GsrJsonSchemaContractTest covers both (12 tests); the NOT NULL and DECIMAL tests fail against the pre-fix code. Limitations updated on the json page (en and zh). JSON module 61 tests, spotless and checkstyle clean; fork CI running on baf3247. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
