kumarpritam863 opened a new pull request, #28848:
URL: https://github.com/apache/flink/pull/28848
## What is the purpose of the change
FLINK-39053 taught `RowDataToAvroConverters` to map a `CHAR`/`VARCHAR`
column onto an Avro `ENUM` field:
```java
if (schema.getType() == Schema.Type.ENUM) {
return new GenericData.EnumSymbol(schema, object.toString());
}
```
`GenericData.EnumSymbol` does not validate its argument. A string that is
not one of the declared symbols therefore survives the conversion and only
fails much later, inside `GenericDatumWriter`:
```
org.apache.avro.AvroTypeException: Not an enum: ...
```
wrapped in `Failed to serialize row.` That message names neither the
column, nor the offending value, nor the symbols that would have been accepted
— hard to act on for a value that occurs once in a stream.
Separately, the enum `"default"` attribute (Avro 1.9+) is ignored. Avro's
own schema resolution maps an unknown symbol onto the declared default; a
schema author who sets one is explicitly asking for that, but the job fails
instead.
FLINK-39053 is not in a release yet, so this can land before 2.4.0 ships
and no released behaviour changes.
## Brief change log
- `RowDataToAvroConverters` now validates the value before constructing
the `EnumSymbol`:
- a declared symbol converts exactly as before;
- an unknown symbol falls back to `Schema#getEnumDefault()` when the
enum declares one;
- otherwise the conversion fails with an `IllegalArgumentException`
naming the value, the enum and the allowed symbols.
- Validation uses `Schema#hasEnumSymbol`, an O(1) lookup, so there is no
measurable per-record cost.
## Verifying this change
This change added tests and can be verified as follows:
- Added
`AvroRowDataSchemaProvidedSerDeSchemaTest#testSerializeUnknownEnumSymbolFallsBackToEnumDefault`,
over both Avro encodings: an enum declaring `"default": "INACTIVE"` maps an
unknown value onto it and round-trips. Without the change
this fails with `Failed to serialize row.`
- Strengthened
`AvroRowDataSchemaProvidedSerDeSchemaTest#testSerializeWithInvalidEnumSymbol`
to assert the message now names the offending value, the enum's full name and
the allowed symbols, rather than only that some `RuntimeException` was
thrown.
- Ran `mvn clean verify` for `flink-formats/flink-avro` and
`flink-formats/flink-avro-confluent-registry`: 366 and 27 tests respectively,
no failures, 0 checkstyle violations, spotless and ArchUnit clean, japicmp
reports no incompatibility.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: **no**
- The serializers: **yes** — `RowDataToAvroConverters`, on the
`CHAR`/`VARCHAR` to Avro `ENUM` path only.
- The runtime per-record code paths (performance sensitive): **yes** —
one extra O(1) symbol lookup per enum-valued field, on a path that only exists
for Avro `ENUM` targets.
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: **no**
- The S3 file system connector: **no**
## Documentation
- Does this pull request introduce a new feature? **no**
- If yes, how is the feature documented? **not applicable** —
FLINK-39053's behaviour is not documented, and this only changes how an invalid
value is reported and how a declared enum default is honoured.
## Note for reviewers
One judgement call worth confirming: falling back to the enum's declared
`default` means an unexpected value is silently mapped instead of failing the
job. That mirrors what Avro's own schema resolution does when reading, and a
schema author
opts into it by declaring `default` at all — but if the project would
rather always fail loudly, the fallback can be dropped and the validation plus
the improved message kept.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Anthropic Claude Opus 5)
---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
