Dominik Dębowczyk created FLINK-41063:
-----------------------------------------
Summary: DataStream large-record sorting unnecessarily falls back
to Kryo for payload serialization
Key: FLINK-41063
URL: https://issues.apache.org/jira/browse/FLINK-41063
Project: Flink
Issue Type: Bug
Components: Runtime / Task
Affects Versions: 2.2.1, 2.3.0
Environment: Observed on Flink 2.2.1; regression tests reproduce the
issue and verify the fix on 2.3.0.
Reporter: Dominik Dębowczyk
With {{taskmanager.runtime.large-record-handler}} enabled, both
{{FixedLengthByteKeyComparator}} and {{VariableLengthByteKeyComparator}}
extract the entire record as a sort key, although comparison only uses the
byte-array key and timestamp.
{{LargeRecordHandler}} then infers a serializer for the complete record instead
of reusing the supplied payload serializer. This can unexpectedly select Kryo
or fail when generic types are disabled.
We encountered this with the Apache Beam Flink Runner: {{{}Kryo initialization
failed with a classloader conflict during spilling{}}}, surfaced as “The sorter
is closed already.” JFR revealed the underlying exception.
The proposed fix is to extract only the byte-array key and timestamp and return
matching flat comparators. This preserves ordering, avoids payload serializer
inference, and eliminates redundant payload serialization in the key spill file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
